Fine-Grained Resource Provisioning and Task Scheduling for Heterogeneous Applications in Distributed Green Clouds

Haitao Yuan, Member, IEEE, MengChu Zhou, Fellow, IEEE, Qing Liu, and Abdullah Abusorrah, Senior Member, IEEE

Abstract—An increasing number of enterprises have adopted cloud computing to manage their important business applications in distributed green cloud (DGC) systems for low response time and high cost-effectiveness in recent years. Task scheduling and resource allocation in DGCs have gained more attention in both academia and industry as they are costly to manage because of high energy consumption. Many factors in DGCs, e.g., prices of power grid, and the amount of green energy express strong spatial variations. The dramatic increase of arriving tasks brings a big challenge to minimize the energy cost of a DGC provider in a market where above factors all possess spatial variations. This work adopts a G/G/1 queuing system to analyze the performance of servers in DGCs. Based on it, a single-objective constrained optimization problem is formulated and solved by a proposed simulated-annealing-based bees algorithm (SBA) to find SBA can minimize the energy cost of a DGC provider by optimally allocating tasks of heterogeneous applications among multiple DGCs, and specifying the running speed of each server and the number of powered-on servers in each GC while strictly meeting response time limits of tasks of all applications. Realistic databased experimental results prove that SBA achieves lower energy cost than several benchmark scheduling methods do.

I. Introduction

A great deal of attention to providing cloud computing applications is attracted in both academia and industry[1]. Cloud computing has greatly changed the way information technology infrastructure is provided to satisfy various business needs [2]. It allows enterprises to dynamically scale down or up resources according to their actual needs by enabling on-demand infrastructure provisioning [3]. It also realizes significant improvement in mission or business proficiencies without enlarging resource needs. In addition, by supporting a pay-as-you-go service model, it removes initial capital, maintenance and software licensing cost. The trend towards it provides a new paradigm of storage and computing, and has led to the proliferation of data centers [4]. Many famous companies, e.g., Microsoft,Google, Amazon, and Apple have selected this model to provide services more efficiently and quickly to users [5].

One major concern about cloud computing is enormous energy consumption. In two or three years, about 95% of urban data centers would experience total or partial outages that incur annual cost of roughly 2 million US$ per infrastructure [6]. Among them, 28% of these outages would be caused by exceeding the maximum previous grid capacity.Besides the economic concern, the carbon footprint and heat produced by their cooling systems are significantly increasing and they are expected to exceed the airline industry emissions by 2020. According to [7], they consumed about 2.2% of total US. electricity consumption, and originated more than 43 million tons of CO2annually. It is predicted that they would consume 140 billion kilowatt-hours annually until 2020. It is shown that the cost for producing all the electricity required by them is more than $7 billion a year. Each large-scale green cloud (GC) usually needs as much energy as 25 000 households on average. With the ever-growing growth in energy consumed by them, the energy optimization has become a major concern in their server provisioning and cooling systems.

Resource over-provisioning is a major cause of power inefficiency in data centers because if resources are allocated for the peak need, they are under-utilized in most of the time.For instance, it is reported that the average server utilization is only between 10%?30% percent for them whose considerable capacity is wasted. The main component to the energy consumed by them is infrastructure including servers and other equipment. The power is dominated by the power consumed by enterprise servers, accounting up to 60% of their total energy consumption [8]. Therefore, many have proposed resource allocation and task scheduling techniques to increase their energy efficiency.

Current data center providers face an important challenge to minimize their energy cost by intelligently scheduling users’arriving tasks and provisioning their resources including computing, storage, networking, cooling and power distribution facilities. It is pointed out that computing servers need about 28% of the energy consumed by data center providers [9]. There are two types of ways to decrease the energy cost: turning off computing servers or decreasing tasks’ performance. However, the reduction of their energy consumption often deteriorates tasks’ execution performance.The reason is that applications and their data are increasing so quickly that the running of higher-performance servers requires more energy to efficiently execute users’ tasks.

However, naively reducing their energy cost may deteriorate their quality of service (QoS). QoS requirements from applications, e.g., social networking, online gaming, and mobile applications, have to be strictly met by elaborately scheduling tasks and specifying running speeds of powered-on servers. Users’ QoS needs are usually specified in service level agreements (SLAs) [10]. Any QoS violations can significantly bring the penalty to a distributed green cloud(DGC) provider because users have strict performance requirements of their applications, and then increase its total cost. Therefore, their providers also need some approaches to guarantee that their cost is not significantly increased due to SLA violations or low QoS. The unprecedented growth of tasks of applications requires new optimization algorithms to deal with both growth in energy cost. Users’ arriving tasks are delivered to back-end data centers that are typically deployed in multiple geographical sites for performance and cost concerns. Many factors, e.g., the prices of power grid, amount of green energy, maximum number of servers, running speed limits of each server, and the maximum energy given to each data center show spatial variations [11].

Consequently, it is challenging to minimize the energy cost of a DGC provider in a market where above factors all possess spatial variations. To solve it, this work adopts a G/G /1 queuing system, which is the most general model, to analyze the performance of each powered-on server. The execution time and interarrival time of each task have arbitrary probability distributions. Based on it, the energy cost is minimized by investigating such spatial variations while strictly meeting response time limits of tasks of all applications. Such spatial variations are integrated into a single-objective constrained optimization problem, which is further solved by our proposed simulated-annealing-based bees algorithm (SBA) to find a real-time close-to-optimal solution. It properly consumes the power grid, wind and solar energy by optimally allocating tasks of heterogeneous applications among multiple GCs, and specifying the running speed of each server and the number of powered-on servers in each GC. Realistic data, e.g., prices of power grid, green energy data, tasks in Google cluster, are used to evaluate SBA. The experiments demonstrate that SBA achieves lower energy cost than its several benchmark scheduling peers can do without QoS compromise.

The related studies are discussed in Section II. Section III introduces the framework of multiple GCs, and formulates a constrained optimization problem for spatial task scheduling.Section IV describes the proposed SBA to solve it. Real-life data-driven experiments are conducted to evaluate it in Section V. Finally, Section VI concludes this work.

II. Related Work

A. Energy Management

Recently, more and more studies are conducted to realize efficient energy management in DGCs [12]–[15]. Yu et al.[12] investigate an energy management problem for multiple microgrids of data centers. They aim to achieve the long-term operational cost minimization by considering uncertainties in prices of power grid, renewable energy, and arriving workloads. They design a real-time and distributed algorithm to solve their proposed problem. Fang et al. [13] propose a two-time-scale approach to minimize the energy consumed by high-performance-computing DGCs through dynamic processor frequency scaling, cooling supplement and task assignment. Then, the energy minimization problem is solved in a two-time-scale manner. The processor frequency and task assignment are optimized in a steady one, and the cooling supplement is optimized in a dynamic thermal environment.Rong et al. [14] propose a comprehensive set of mechanisms to minimize the environmental impact and maximize the efficiency of data centers by considering cost reduction,energy consumption, and environment protection. They also show the future energy-saving trends for data centers.Vasudevan et al. [15] formulate the assignment of applications to virtual machines as a problem of profile-driven optimization under different constraints, and solve it by a genetic algorithm. They improve a penalty-based genetic algorithm by applying the longest cloudlet, the fastest processor and a procedure for repairing infeasible solutions.Finally, they develop a scalable method for application assignment to realize the trade-off between resource utilization and energy efficiency.

Different from the above studies, this work aims to minimize the energy cost of a data center provider by investigating the spatial variations of several factors while strictly meeting response time limits of tasks of all applications. Then, it proposes SBA to smartly consume power grid, wind and solar energy by optimally allocating tasks of heterogeneous applications among multiple data centers, and specifying the running speed of each server and the number of powered-on servers in each GC.

B. Resource Optimization Methods

In recent years, resource optimization methods have attracted a growing amount of attention [16]–[19]. Lyu et al.[16] develop a semidistributed and heuristic offloading decision algorithm to jointly optimize the offloading,computation and communication resources for system utility maximization. The formulated problem is reduced to a submodular maximization one, and further decomposed into two subproblems. The first one is tackled with convex and quasiconvex optimization, and the second one is tackled with submodular set function optimization. Li et al. [17] address the problem of virtual network function placement by investigating service function chain requests of users. It is formulated as an integer linear programming problem for minimizing the number of physical machines, and solved by a two-stage heuristic method, which includes a correlationbased greedy algorithm and an adjustment one for requests of virtual network functions. Li et al. [18] jointly optimize resource optimization and congestion control to realize the energy efficiency-guaranteed trade-off between delay performance and throughput in heterogeneous cloud radio access networks. Their formulated stochastic optimization problem is transformed into three subproblems solved in parallel. Qiu et al. [19] design a correlated modeling method by using a Bayesian method, Laplace-Stieltjes transform, and semi-Markov models to analyze reliability performance and energy correlations for cloud applications. A recursive method is presented by using a check-pointing fault recovery mechanism, and can jointly optimize energy consumption and service time for running a cloud application.

Different from these studies, this work formulates an energy minimization problem as a single-objective constrained optimization one, and solves it with a newly proposed SBA.SBA combines the Metropolis acceptance criterion of SA into BA, and performs the SA-based selection and the disruptive selection to ensure its high convergence speed and obtained solution accuracy.

C. Application Behavior Analysis

The application behavior analysis in DGCs has been investigated [20]–[23]. Bi et al. [20] compute task arriving rates according to internal and external workload for multiple resource-intensive applications. They develop a probabilistic queueing model to cope with non-steady states in a smart controller. Then, computing and storage resource consumption in a virtualized cloud data center is minimized. Kafhali and Salah [21] propose a stochastic model according to queuing theory to analyze the performance. Data center platforms are modeled as an open queuing system for their QoS analysis.Then, the number of needed instances of virtual machines(VMs) is estimated and used to meet specified QoS requirements. Satpathy et al. [22] propose a queueing model to schedule and manage a set of VM requests. This model makes it easy to realize, analyze and validate complex systems like cloud. Its structure is designed as a single queue single service facility by using an M / M/1 queue. VM requests are executed with a first-come-first-serve (FCFS) manner and forwarded to data centers for placement. Then, a multiobjective VM placement method is designed to decrease the resource and power consumption at data centers. Ponraj [23]adds tasks into a priority-based probability queuing model,and schedules them into a suitable VM. Specifically, an M/G /1 queuing model is adopted to derive the waiting time of tasks. Then, a VM placement algorithm is proposed to reduce completion time and processing cost by considering computation resources, I/O data, QoS metrics and VM status.

Unlike these methods, this work adopts the most general model, i.e., the G/G /1 queue, to analyze the performance of each powered-on server. In our model, both execution time and interarrival time of each task follow arbitrary probability distributions. Based on the model, SBA is proposed to specify the running speed of each server and number of powered-on servers in each data center at different locations.

III. Problem Formulation

This section formulates a constrained optimization problem for spatial task scheduling. Fig.1 shows the illustrative framework of multiple GCs. Let C denote the number of GCs.Users’ arriving tasks are sent through electronic devices, e.g.,smart phones, laptops, computers and servers to C GCs, and they are scheduled by task scheduler based on a FCFS manner. SBA is periodically executed in task scheduler to minimize the energy cost by intelligently scheduling tasks of each application among multiple centers, and optimally determining the running speed of each server and number of powered-on servers in each GC. For clarity, main notations in this work are summarized in Table I.

Fig.1. Illustrative framework of multiple GCs.

A. Task Response Time Model

Currently, there are many classical algorithms, e.g.,dynamic programming [30], Lagrange multiplier [31], Branch and bound [32], Bucket elimination [33], to solve it.Nevertheless, they usually need the first-order or second-order derivatives of the objective functions. They are effective to solve some constrained optimization problems with certain required mathematical structures [34]. Yet, their optimization processes are difficult and their final solutions are often unsatisfied. To tackle such shortcomings, many studies design meta-heuristic algorithms that obtain near-optimal solutions to the constrained optimization problem in reasonable execution time. They have several advantages including easy implementation, robustness, handling complex nonlinearities and discontinuities of objective functions. In the category of intelligent optimization tools, swarm-based optimization algorithms (SOAs) are search ones that can efficiently locate relatively good solutions [35]. SOAs are inspired by methods in nature to provide an effective search towards an optimal solution. SOAs differ from direct search algorithms, e.g., hill climbing, because SOAs adopt a population of solutions for each iteration instead of one single solution. These solutions are updated iteration by iteration, output when some termination conditions are met. For example, particle swarm optimization (PSO) and its state-of-the-art variants have been widely used [36], [37].

Among SOAs, bees algorithm (BA) is an optimization one inspired by the foraging behavior of natural honey bees. It is commonly applied due to its easy implementation and quick convergence [38]. In BA, a colony of honey bees extend themselves over long distances in different directions. Flower patches with more pollen or nectar that is collected with less effort should attract more bees, whereas those with less pollen or nectar attract fewer ones. Scout bees randomly search from one patch to another, and evaluate different patches. Then,their pollen or nectar is deposited and they perform a waggle dance in a dance floor. The information including direction of flower patches, distance from their hive and fitness is communicated through their waggle dance by exchanging the angle information about between sun and patches and duration, and frequency of the dance. Follower bees follow a dancer bee to quickly and efficiently collect food. The same patch is advertised through the dance for many times when going to the hive if it is good enough as a food source, and more bees are attracted to that patch. The flower patches with more nectar or pollen are visited by more bees. Thus, patches may be visited by more bees, or abandoned depending on the fitness. BA has been applied in many areas, e.g., real-time production scheduling [39] and intelligent transportation systems [40].

BA is very efficient in obtaining high-quality solutions to constrained optimization problems. However, there are a number of tunable parameters that need to be figured out. In addition, it is often easy to trap into a local optimum in its search process and causes premature convergence. Thus, the quality of its final solutions is unsatisfied if it is used to solve complicated optimization problems with large solution spaces.Simulated annealing (SA) is able to escape from a locally optimal solution by conditionally enabling some moves to worsen solutions by using its criterion of Metropolis acceptance [41]. It is demonstrated that SA can theoretically find a global optimum with high probability, and, therefore, it is able to obtain high-accuracy solutions to different types of discrete and continuous optimization problems [42].Nevertheless, its convergence process is relatively slow.Therefore, this work designs a hybrid algorithm named simulated-annealing-based bees algorithm (SBA) to solve the unconstrained optimization problem by integrating the Metropolis acceptance criterion of SA into BA. Specifically,this work performs SA-based selection with to update each elite or non-elite bee. The other novelty is SA-based selection and the use of disruptive selection to increase its convergence speed and solution accuracy.that the higher and lower-quality individuals are more preferable. This means that disruptive selection aims to increase the diversity of individuals in the population by retaining diverse individuals. Then, the population of h scout bees is sorted with disruptive selection in (31).

Algorithm 1 SBA 1: Initialize a population of scout bees g ←1 h 2: tg ←?t 3: 4: while do 5: Evaluate the fitness of the population h g ≤G 6: Sort the population of scout bees with disruptive selection in (31)??7: Select the best scout bee ( )t 8: Select sites for the neighborhood search ?n 9: Determine the neighborhood size, l ←1e 10: for to do χ 11: Recruit bees for elite site χ l 12: Select the best one among recruited bees 13: Perform SA-based selection with (30) to update elite bee 14: end for l ←1t?e l 15: for to do δ 16: Recruit bees for non-elite site δ l 17: Select the best one among recruited bees 18: Perform SA-based selection with (30) to update non-elite bee 19: end for 20: Update the fittest bee from each selected site h?t l 21: Assign remaining bees to random search h 22: Produce a new population of scout bees 23: Reduce the neighborhood radius g ←g+1 24: 25: 26: end while tg ←tg?1?ε 27: Output ??

Line 20 updates the fittest bee from each selected site to form the next population. More bees are recruited to follow the elite e sites to search in their neighborhood including more promising solutions. Therefore, the differential recruitment is an important and key operation of BA. Line 21 assigns h?t remaining bees to random search. Line 22 produces a new population of h scout bees. Then, the new population has two parts including a representative from each selected patch, and other scout bees that randomly searched. Line 23 reduces the neighborhood radius. Line 25 reduces tgby ε, which denotes the temperature colling rate. Finally, Line 27 outputs the best scout bee ( ??), which is further transformed into decision

V. Performance Evaluation

This work evaluates the proposed SBA with real-life data.SBA is implemented and coded with MATLAB 2017, and it is executed in a server with a 32-GB DDR4 memory and an Intel Xeon E5-2699AV4 CPU at 2.4 GHz.

A. Parameter Setting

This work adopts realistic task arriving rates of three applications in Google cluster trace11 https://github.com/google/cluster-datato evaluate the proposed SBA. Fig.2 shows task arriving rates. Besides, as shown in Fig.3, this work also adopts real-life prices of power grid collected from three different places22 http://www.energyonline.com/Data/for three GCs. In addition, the length of each time slot is 5 min, i.e., L=300 s.

Fig.2. Task arriving rates of three applications.

Here, this work considers three applications deployed in three GCs, i.e., C=3 and N=3. Following [11], [28], Table II shows the setting of parameters related to energy suppliers including power grid, wind energy and solar energy. In addition, this work adopts real-life data about wind speed33 http://www.nrel.gov/midc/nwtc_m2/and solar irradiance4http://www.nrel.gov/midc/srrl_bms/for 24 hours. Figs. 4 and 5 show wind speed and solar irradiance in three GCs.

It is worth noting that SBA is sensitive to its parameter

Fig.3. Prices of power grid in three GCs.

TABLE II Parameter Setting of Wind and Solar Energy

Fig.4. Solar irradiance of three GCs.

Fig.5. Wind speed of three GCs.

B. Experimental Results

Figs. 6–8 show the arriving rates of tasks of three applications allocated to three GCs, respectively. It is clearly observed that the number of tasks of each application allocated to GC 1 is the highest and that allocated to GC 3 is the lowest. Figs. 9–11 show the number of powered-on servers for three applications, respectively. It is clearly observed that they all do not exceed their corresponding limits. Besides, it is also observed that the number of powered-on servers in GC 1 for each application is the highest and that in GC 3 is the lowest. This is because the price of power grid of GC 1 is the lowest and that of GC 3 is the highest. Therefore, the largest number of tasks are scheduled to GC 1 with the largest number of powered-on servers among three GCs.

Fig.12 illustrates the amount of power grid energy consumed by three GCs. As shown in Fig.3, prices of power grid of three GCs vary from each other. SBA aims to minimize the energy cost of the DGC provider by smartly scheduling tasks of heterogeneous applications among multiple GCs while satisfying delay-bound constraints of all tasks of each application. It is shown that the amount of power grid energy consumed by GC 1 is the highest while that in GC 3 is the lowest. The result is consistent with prices of power grid in three GCs, i.e., the price of power grid in GC 1 is the lowest while that in GC 3 is the highest.

C. Comparison Results

To demonstrate the performance of SBA, this work compares it with two typical meta-heuristic optimization algorithms, i.e., BA [47] and genetic learning particle swarm optimization (GL-PSO) [48]. Each algorithm is repeated for 30 times independently to generate the statistical results. The reasons of selecting them as SBA’s peers are:

1) BA [47]: As a swarm-based optimization algorithm, BA is very efficient in finding high-quality solutions. However, it needs to figure out a number of tunable parameters, and it easily traps into a local optimum.

2) GL-PSO [48]: GL-PSO performs crossover, mutation,and selection on particles’ historical information to construct well diversified and highly qualified exemplars that guide particles’ search processes. GL-PSO enhances both the search efficiency, robustness, scalability and the global search ability of PSO.

The key parameter setting of BA is the same as that of SBA.In addition, the key parameter setting of GL-PSO is given as follows. The number of iterations is 1000. The population size is 100. The intertia weight is 0.7298. The accelerate

TABLE III Parameter Setting of Three GCS

Fig.6. Arriving rates of type 1 tasks allocated to three GCs.

Fig.7. Arriving rates of type 2 tasks allocated to three GCs.

Fig.8. Arriving rates of type 3 tasks allocated to three GCs.

coefficients of the locally best and the globally best individuals are both set to 2. The exemplar learning coefficient is set to 1.49618. The maximum velocity is 10.The probability of mutation is 0.1. The comparison among SBA, BA and GL-PSO can demonstrate the accuracy and the convergence speed of the final solution of SBA. In addition, it

Fig.9. Number of powered-on servers in three GCs for type 1 application.

Fig.10. Number of powered-on servers in three GCs for type 2 application.

Fig.11. Number of powered-on servers in three GCs for type 3 application.

is worth noting that BA and GL-PSO are all sensitive to their parameter setting. Consequently, similar to SBA, many experiments are performed to specify the optimal parameter setting of both BA and GL-PSO according to the grid search method [45] and similar setting of parameters in previous studies [47], [48]. In addition, BA and GL-PSO terminate their search processes if they do not find better solutions in successive 10 iterations.

Fig.12. Amount of power grid energy consumed by three GCs.

Fig.13 shows the energy cost comparison of SBA, BA and GL-PSO. It is shown that compared with BA and GL-PSO,the energy cost of SBA is decreased by 59.07% and 92.83%on average, respectively. Fig.14 shows the execution time comparison of SBA, BA and GL-PSO. It is observed that compared with BA and GL-PSO, the execution time of SBA is decreased by 26.31% and 46.15% on average, respectively.SBA’s average execution time of all time slots is 65.94 s, and it is 26.16% smaller than that of BA, 89.30 s, and 49.27%smaller than that of GL-PSO, 129.97 s. Figs. 13 and 14 demonstrate that SBA obtains a more accurate solution in less convergence time than BA and GL-PSO.

Fig.13. Energy cost comparison of SBA, BA and GL-PSO.

Fig.14. Execution time comparison of SBA, BA and GL-PSO.

Fig.15 illustrates the energy cost comparison of each iteration of SBA, BA and GL-PSO in time slot 1. Here, each iteration of SBA means Lines 5–25 in Algorithm 1. The meaning of iterations of BA and GL-PSO is similar to that of SBA. BA and GL-PSO need 951 and 996 iterations to converge to their final solutions, and their final energy cost are 76 165.77 and 218 339.94 and 218 339.94, respectively.SBA only needs 201 iterations to converge to its final solution, and its energy cost is 14 832.58$. Consequently,SBA significantly reduces the energy cost of the DGC provider in much fewer iterations than BA and GL-PSO. Fig.16 demonstrates that the integration of the Metropolis acceptance criterion of SA in SBA improves the solution diversity, and the global search accuracy of BA.

Fig.15. Energy cost of each iteration in time slot 1.

To prove the effectiveness of SBA, this work compares it with several typical task scheduling methods [49]–[52] with respect to energy cost and throughput.

1) M1: Similar to cheap price of power grid-first scheduling in [49], it schedules tasks to GCs by following the order of their prices of power grid.

2) M2: Similar to green energy-first scheduling in [50], it schedules tasks to GCs by following the order of their amount of wind and solar energy.

3) M3 [51]: It schedules tasks among distributed GCs by leveraging geographic and temporal variations of energy prices.

4) M4 [52]: It cost-effectively schedules tasks among GCs by exploiting spatial diversity of electricity prices.

Fig.16 shows the throughput comparison of SBA, M1, M2,M3 and M4, respectively. It is shown that the throughput of SBA is greater than those of M1, M2, M3 and M4 for each application in each time slot, respectively. For example, for application 1, SBA’s throughput is greater than those of M1,M2, M3 and M4 by 25.99%, 25.37%, 10.30% and 7.74% on average, respectively. The reason is that the maximum number of servers, running speed limits of each server, and maximum energy in each GC are all limited in each time slot. In addition, SBA intelligently schedules tasks of each application among GCs, and optimally sets the running speed of each server and the number of powered-on servers in each GC.Therefore, some tasks of users are refused and not scheduled to GCs when using M1, M2, M3 and M4.

Fig.16. Throughput comparison of SBA, M1, M2, M3 and M4.

Fig.16 illustrates the energy cost of SBA, M1, M2, M3 and M4, respectively. To ensure the actual performance of tasks,the penalty is required in SLAs for each rejected task [53]after the negotiation between a DGC provider and users. The penalty of each rejected task is usually greater than the largest energy cost corresponding to the execution of each task of the same application among GCs in each time slot. Thus, this motivates a DGC provider to strictly guarantee delay constraints of tasks of all applications. In Fig.16, the energy cost in each time slot is calculated by summing up the energy cost of tasks executed in GCs, and the penalty of rejected tasks in each time slot. It is shown in Fig.17 that compared with M1, M2, M3 and M4, the energy cost of SBA is decreased by 50.11%, 51.55%, 29.15%, and 25.27% on average, respectively. This is because SBA intelligently schedules tasks among GCs by jointly investigating spatial variations in prices of power grid, and the amount of green energy in GCs.

Fig.17. Energy cost comparison of SBA, M1, M2, M3 and M4.

VI. Conclusion

Cloud computing allows enterprises to achieve many benefits by reducing administrative, capital and operational cost. Yet it suffers from the high energy consumption problem that negates its advantages. Many large-scale enterprises adopt distributed green cloud (DGC) systems to provide application services to users through intelligent task scheduling. However,existing studies [54]–[57] fail to minimize the energy cost of the DGC provider by providing fine-grained resource provisioning and scheduling for tasks of heterogeneous applications. In addition, many factors, e.g., prices of power grid, and the amount of green energy in GCs show their significant spatial variations. Therefore, it is a big challenge to minimize the energy cost of a DGC provider. This work uses a G/G /1 queuing model to analyze the performance of servers,and further formulates a constrained optimization problem. It is solved by a newly proposed Simulated-annealing-based bees algorithm to find a close-to-optimal solution. Then, the energy cost minimization is achieved for a DGC provider by optimally allocating tasks of heterogeneous applications among multiple GCs, and specifying the running speed of each server and the number of powered-on servers in each GC. Real-life data-driven experiments demonstrate that the proposed algorithm is proposed to achieve can decrease energy cost and ensure the highest throughput in comparison with its several up-to-date scheduling methods. How to set optimally its user-defined parameters requires more work by using some recent approaches in [58]–[59] and additional comparisons with other intelligent scheduling methods[60]–[72] should be conducted. How to extend its application to large-scale cloud/edge/fog computing environment[73]–[75] remains open.

Appendix

According to (33), we have

It is worth noting that (34) is equivalent to (8). Then, (8) is derived.