Future Generation Computer Systems

Contents lists available at

Future Generation Computer Systems

journal homepage:

Energy-credit scheduler: An energy-aware virtual machine scheduler for cloud systems

Nakku Kim a, Jungwook Cho b, Euiseong Seo b,∗

a School of ECE, Ulsan National Institute of Science and Technology, Ulsan, 689-798, Republic of Korea

b College of ICE, Sungkyunkwan University, Suwon, 440-746, Republic of Korea

a r t i c l e i n f o

a b s t r a c t

Article history:

Received 25 October 2011 Received in revised form 3 April 2012

Accepted 17 May 2012

Available online 28 May 2012

Keywords: Virtualization Scheduling

Energy-aware computing Cloud computing Resource accounting

Virtualization facilitates the provision of flexible resources and improves energy efficiency through the consolidation of virtualized servers into a smaller number of physical servers. As an increasingly essential component of the emerging cloud computing model, virtualized environments bill their users based on processor time or the number of virtual machine instances. However, accounting based only on the depreciation of server hardware is not sufficient because the cooling and energy costs for data centers will exceed the purchase costs for hardware. This paper suggests a model for estimating the energy consumption of each virtual machine without dedicated measurement hardware. Our model estimates the energy consumption of a virtual machine based on in-processor events generated by the virtual machine. Based on this estimation model, we also propose a virtual machine scheduling algorithm that can provide computing resources according to the energy budget of each virtual machine. The suggested schemes are implemented in the Xen virtualization system, and an evaluation shows that the suggested schemes estimate and provide energy consumption with errors of less than 5% of the total energy consumption.

© 2012 Elsevier B.V. All rights reserved.

The energy costs of data centers are rapidly increasing. By 2012, these costs will be double the 2007 costs and will exceed the purchase costs for server hardware [1]. In addition to energy costs, cooling costs for data centers continue to increase because the ever-increasing power consumption by servers leads to increased heat dissipation. Thus, the cloud computing model is expected to become the next-generation computing infrastructure owing to its energy efficiency.

In a cloud system, virtualization is an essential tool for providing resources flexibly to each user and isolating security and stability issues from other users [2]. Because users’ services and applications run on separate virtual machines (VMs), virtualization helps cloud platforms to accurately monitor and control the amount of resources being provided to users. Virtualization also facilitates the guarantee of service-level agreements (SLAs) between users and cloud platform providers by resolving QoS crosstalk between services and applications. Moreover, separation of the execution context and data between VMs is supported by

∗ Corresponding author. Tel.: +82 31 299 4970; fax: +82 31 299 4921.

E-mail addresses: nkkim@unist.ac.kr (N. Kim), jmanbal@skku.edu (J. Cho), euiseong@gmail.com, euiseong@skku.edu (E. Seo).

processors and hypervisors so that users use their VMs as if they own their dedicated server hardware. Owing to these benefits, many commercial cloud systems including Amazon EC2 employ virtualization so that users can freely configure their virtual servers from kernel to application layers.

Currently, cloud services bill their users for the amount of computing resources provided [3]. In general, the amount of processor time and number of VMs allocated are common criteria for accounting [2]. Such billing systems are based on the common belief that the costs for purchasing and maintaining server hardware are proportional to the operation time. However, if significant factors that affect the management cost of servers are not considered, these billing schemes may have room for improvement. The energy consumption of a server system closely relates to the processor working time. However, the processor time does not accurately reflect the system energy consumption [4].

As stated earlier, energy and cooling costs will exceed hardware purchasing costs. Therefore, to fairly delegate energy costs to users, a billing system must infer the energy consumption for each user. However, measurement of the energy consumption for each VM ruing on a server with multiple VMs is a technically challenging problem. The use of embedded power meters in server systems is increasingly popular. However, because measurement of power consumption for the whole system is a totally different problem from measurement of that for only one VM, the problem remains even for embedded power meters.

0167-739X/$ – see front matter © 2012 Elsevier B.V. All rights reserved.

Existing research has approached the measurement of per-VM

energy consumption by accounting for and analyzing the resources used by each VM. However, most models for estimating energy consumption assume that the power consumption of each VM is simply proportional to the processor time allocated or the number of network packets processed.

In this paper, we reveal that processor time is not a satisfac- tory criterion for accurate estimation of power consumption by a VM and that some in-processor events affect processor power con- sumption more significantly than others. Based on observations made in diverse experiments, we suggest a model for estimating energy consumption that calculates the amount consumed by a VM via monitoring of processor performance counters.

Based on the proposed estimation scheme, we also propose an energy-aware VM scheduler that can limit the energy consumption of virtual machines to their energy budget. Conventional VM schedulers only consider the processor time when it comes to scheduling decisions. Different from its traditional counterparts, the energy-credit scheduler uses the energy consumption rates of VMs for scheduling. It schedules VMs so that their energy consumption rates remains below user-defined values.

Both the suggested estimation model and scheduler are imple- mented in the Xen virtual machine monitor. The implementation also provides external interfaces so that they can be easily adopted for real-world systems. We evaluate our prototype using the SPEC CPU2006 benchmark suite.

The remainder of the paper is organized as follows. The background and related works are introduced in Section 2. We suggest and evaluate a model for estimating the energy consumption of VMs in Section 3. Based on this model, we propose an energy-aware scheduler and evaluate it in Section 4. We conclude our research in Section 5.

2.1. Motivation

The amount of energy consumed by a server varies greatly depending on the workload it runs. Many studies on predicting or controlling server power consumption are based upon the assumption that the power consumption of a server is determined mainly by its processors, and that the power consumption of a processor is determined primarily by processor utilization [5,4,6]. However, energy consumption estimation based on processor utilization or execution time is generally not accurate enough for billing of energy costs.

Server vendors have recently begun to integrate power measurement hardware to monitor server power consumption on the fly [7]. By employing the integrated power measurement hardware, the energy consumption of a server can be measured accurately. However, an integrated power meter measures the power consumption of the whole server system, while the energy- based billing systems of cloud services require measurement of the energy consumption for each VM.

Despite previous research to leverage dedicated measurement devices, determination of the energy consumption of a VM is still a difficult problem for the following reasons.

First, the sampling intervals of integrated power meters generally fall between 1 and 0.1 s [8], whereas the time unit for scheduling of VMs is very short, ranging from a few hundred microseconds to a few tens of milliseconds. The time scale difference between measurement and scheduling makes it difficult to identify the energy consumption of a specific VM when there are multiple VMs competing for a processor.

Second, VMs in a multicore processor system share and compete with other VMs for system resources. Naturally, the

throughput of a VM and its energy consumption may vary

according to the characteristics and activities of other VMs that run concurrently [9]. Therefore, energy consumption measurement for a VM caot rely on blind arithmetic that only considers processor and resource usage by the VM. Thus, to identify the energy consumption of each VM, which continually and rapidly changes, energy consumption estimates must consider both the activities of the VM and the effects from other VMs on resource utilization.

Besides the VM-level energy measurement schemes, energy- aware resource provisioning schemes are also essential for energy- cost-based billing systems so that users can limit the energy consumption of their VMs to stay within their energy budgets.

Most conventional VM schedulers aim to provide processor time to VMs fairly and proportionally according to their priorities because processor time is one of the major billing criteria of conventional computing systems. In other words, processor time is the major scheduling basis in conventional schedulers [10,11]. Therefore, a novel energy-aware scheduler and its premise must be designed from scratch. At the start of our research, we conceptualized a system for our aim as follows.

To utilize an energy-aware scheduling scheme, each VM should explicitly provide its energy budget to the scheduler. The energy budget of a VM is the amount of energy it is allowed to use during a fiscal time interval. If a VM uses up its energy budget in a fiscal interval, then that VM will be suspended until the current fiscal interval finishes and the next interval begins. The VM fiscal interval is determined by the owner of the VM according to its purpose and characteristics, and may range from a few tens of milliseconds to a few minutes. Thus, the fiscal interval of a VM may differ from that of other VMs.

The fiscal interval of a VM should be carefully chosen because it supposedly affects throughput and responsiveness of the VM. When the interval is too short, excessively frequent context switches will occur. If the interval is too long, the suspension time for VMs without any energy budget left will be too long and will negatively affect the responsiveness of the services provided by the VMs.

For example, as shown in Fig. 1, when a web service VM and a data backup service VM are run concurrently on a physical server, it is beneficial for the web service VM to have a short-term plan such as 300 mWh/0.5 s, while a long-term plan such as 18000 mWh/1 min would be more suitable for the backup service VM.

In this paper, we realize this conceptual model by suggesting a model for estimating per-VM energy consumption and an energy- aware VM scheduler.

2.2. Related work

There has been intensive research on power-capping resource management schemes, which limit the power consumption of a server or a set of servers to predefined thresholds [6,12]. Additional research has introduced techniques to distribute workloads and manage computing resources so that workloads are evenly distributed over a set of servers in terms of power requirements [13]. The goal of these efforts is to reduce the costs of building and maintaining a power distribution infrastructure, as well as energy costs, by reducing the peak power consumption of single nodes or server groups [4,14,13,12].

Predicting the power and energy consumption of a server that runs a specific workload is straightforward, especially when the workload is stable in terms of resource usage. However, predicting the power and energy consumption of a server on which multiple VMs are consolidated is difficult because the VMs interfere with each other. A statistical method has been proposed for predicting the power consumption of a server with

Fig. 1. Consolidation of heterogeneous VMs in a cloud node.

consolidated VMs [14]. Based on this approach, a scheme called Soft Fuse has been suggested [4]. Similar to fuses in electric circuits, Soft Fuse limits the power consumption of a server by throttling the processor performance when it consumes too much power. These approaches, however, assume that the target system has only one processor with a single context and exclude interference and resource contention in their models. Therefore, they are not applicable to modern multicore systems.

A power distribution infrastructure must be designed to provide sufficient power. Thus, the peak power consumption rather than the average power consumption, which directly relates to energy consumption, is the more important factor that affects both costs and system availability [13]. Thus, the energy consumption of servers has attracted less interest than their peak power consumption.

An energy efficient VM-to-physical server mapping method has been suggested [15]. This mapping scheme observes changes in energy efficiency and performance for different consolidation cases and identifies the most energy-efficient consolidation mapping. Although this mapping scheme improves the overall energy efficiency of cloud systems, it does not provide any means to control the energy consumption of each VM.

ECOsystem [5], a prototype energy-centric operating system, considers energy as a first-class operating system resource. It al- locates energy to tasks according to their priorities, and sched- ules tasks within an energy budget to guarantee battery lifetime. In their research, the authors assumed that the processor consumes a fixed amount of power. This assumption is partly correct for mobile embedded systems, because the power consumption of processors in mobile embedded systems is significantly less than that in server systems and is therefore less critical to the power consumption of the overall system. This approach is for conventional single-core embedded systems and does not tackle the multicore issue either. Joule meter [16] is a software approach for measuring energy consumption of VMs in a consolidated server environment. Joule meter estimates the amount energy that each VM consumes by monitoring its resource usage dynamically. It employs two energy consumption estimation models. The first is a simple model that calculates energy consumption by multiplying processor time by the average power consumption of the processor, which is similar to the ECOsystem approach. Because of this oversimplification, the model shows poor accuracy. The second model, the refined model, uses an integrated power measurement device to collect power consumption patterns for the entire system. When a new VM is created, the system observes changes in power consumption patterns over the first 200 s. This observation yields the power consumption characteristics of the new VM after statistical processing such as linear regression. By using actual characteristics

of the power consumption pattern, this model estimates VM power consumption more accurately. However, this approach requires an integrated power measurement device. Moreover, if the workload of a VM fluctuates severely, this approach may perform poorly, because the model is based on the assumption that the behavior of the VM remains the same as in the first 200 s.

An accurate processor power consumption model was sug- gested [17]. This model estimates the power consumption of a processor by monitoring its internal activities (memory accesses, integer or floating point arithmetic operations, etc.). However, the processor considered is a single-core embedded processor and the model does not consider multicore effects. It has been demon- strated that other power estimation models based on observation of internal activities are accurate [18,19]. Based on the same ap- proach, some system simulators estimate the power and energy requirements of applications [20,21].

To optimize the energy efficiency of consolidated virtual servers while maintaining SLAs, a VM scheduler that manages both the clock frequency and scheduling time of VMs has been suggested [22]. This approach further improves the energy efficiency of consolidated virtual servers. A follow-up scheduling scheme that considers the energy budget of the physical server and its SLAs has been suggested by the same research group [23]. The scheduler measures power consumption using an external power meter and determines the resources to provide to each VM using multiple feedback loops so that each VM keeps its SLA while the server conforms to the energy budget. This scheduler introduces energy-driven resource provisioning at the VM monitor level. However, in the suggested scheme, the energy budget is given only for the entire physical server, and not for each VM.

The power consumption of a processor can be divided into leakage power and dynamic power [24]. Leakage power steadily drains away while electricity is supplied to the processor, whereas dynamic power is consumed only when the processor is executing instructions. The amount of leakage power is always the same while the system is activated. Therefore, it caot be the target of accounting nor provisioning. In this paper, we consider dynamic power only, which is the manageable part by the VM scheduler. Hereafter, ‘‘power’’ denotes dynamic power, unless otherwise stated.

3.1. Observation

A number of factors affect the power consumption of a processor. Power is consumed by components when executing

Fig. 2. Power consumption as a function of the number of retired micro-operations per 300 ms under CPU-intensive workloads.

instructions. Thus, it is expected that energy consumption, which is the product of power and time, is proportional to the number of instructions carried out. However, the same number of instructions can lead to different energy consumption, depending on the type of instructions. For example, memory access operations such as loading and storing use a different set of execution units compared to arithmetic operations.

We conducted some experiments to observe that to what de- gree diverse in-processor events affect processor power consump- tion. Most modern processors provide programmable performance counters. Each performance counter is a special register that can be programmed to count the occurrences of a certain type of activity in a processor, such as cache misses, branch prediction failures, and floating point instructions. For example, the Intel Nehalem archi- tecture processors provide four programmable performance coun- ters [25].

We identified the relationship between each in-processor event count and processor power consumption in a series of experiments, and selected some meaningful relationships to use in our estimation model.

The minimum scheduling unit of the Xen hypervisor is

10 ms [26]. Consequently, when CPU-intensive workloads are being executed, the Xen hypervisor can read the performance counters once every 10 ms. However, because the Xen hypervisor allows VMs to run for up to 30 ms without intervention, we use a 30-ms interval, which consists of three consecutive time ticks, as the unit time interval for estimation.

A 30-ms interval is too short for measurement of the power consumption of a system. Thus, to set up the estimation model, we measured power consumption changes depending on the performance counter values every 300 ms. Fig. 2 shows the average power consumption during 300-ms time intervals according to the number of micro-operations. The system ran between one and four VMs simultaneously for execution of the SPEC CPU2006 benchmark suite. The system is equipped with a quad-core Intel i5 750 processor, so up to four VMs were able to run concurrently.

Micro-operations are a more fine-grained unit of execution in a processor than an instruction. Therefore, we believe that the number of retired micro-operations is likely to reflect activities within a processor more accurately than the number of retired instructions.

Even within the same workload, the number of retired micro- operations within a time interval varies greatly over time, and this tendency results in fluctuations in power consumption patterns. In general, the average power consumption within a time interval, which can be directly interpreted as the energy consumption of the

Fig. 3. Power consumption as a function of the number of retired micro-operations per 300 ms under CPU- and memory-intensive workloads.

Table 1

Average number of last-level cache (LLC) misses for each workload during a 30-ms execution interval.

Workload Avg. LLC misses Workload Avg. LLC misses

povray	1,476	gamess	1,615
tonto	2,042	hmmer	23,585
gromacs	27,251	namd	30,948
dealII	526,830	–	–

interval, seems to be linearly proportional to the number of micro- operations.

The workloads used in our experiments shown in Fig. 2 have infrequent memory accesses in common. The results in an additional workload of frequent memory accesses are shown in Fig. 3.

The dealII benchmark has heavy LLC misses, as shown in Table 1. When LLC misses are frequent, micro-operations per second tend to be lower than for infrequent LLC misses. However, in comparison to workloads with infrequent memory accesses, more energy is consumed to execute the same number of micro- operations. To access memory, processors have to stall for some time, but memory and bus systems continuously consume power, even during the stall cycles. Consequently, in spite of the same number of micro-operations, memory-intensive workloads tend to consume more energy than non-memory-intensive workloads.

The number of working cores in a multicore processor strongly affects processor energy consumption. Although each core is a separate and independent execution unit, the cores share many components, such as on-chip caches, buses and memory controllers. Therefore, the dynamic power of a processor can be formulated as:

Pprocessor = Pshared + ΣPcore, (1)

where Pshared is the power consumption of the shared components and Pcore is the power consumption of each core that is executing instructions. Consequently, a processor should require less energy when instructions are executed over multiple cores than when the same number of instructions are executed on a single core or a smaller number of cores. The experimental results in Fig. 4 verify our assumption.

Fig. 4 shows the relationship between power consumption and the number of retired micro-operations when between one and four cores are simultaneously used. The graph points are grouped around four small regions according to the number of active cores. The differences in power consumption between one and two, two and three, or three and four cores are less than the power

Fig. 4. Power consumption as a function of the number of active cores and micro- operations during execution of gromacs.

Fig. 6. Deadlocks may occur when cores call IPIs at every context switch.

event counters that are the most closely related to processor power consumption.

Usually, for the sake of performance, the virtual processor for a VM is committed to a processor core. For VM1, . . . , VM, our aim is to estimate Evmk consumed by VMk within a time interval based on the event counts by the processor cores that run VMk.

The overall energy consumption of the entire system from its dynamic power during a fixed time interval is defined as:

Esystem = ued70 Evm

i=1

i . (2)

Fig. 5. Power consumption as a function of the number of micro-operations during simultaneous execution of diverse workloads.

consumption of a single active core. As expected, this is because the shared components consume little additional power when more cores are put into operation.

This trend is also evident when diverse heterogeneous work- loads are concurrently executed together. Fig. 5 shows the power consumption while multiple VMs run different benchmarks. Use of more cores induces more micro-operation retirements in the same time interval. In addition, use of more cores results in better en- ergy efficiency, as evidenced by the less steep increase in power consumption compared to that for a single core.

Contrary to popular belief, energy consumption does not dif- fer significantly between floating point operations and integer operations at the micro-operation level in spite of the huge dif- ference in complexity at the instruction level. In our experiments, the coefficient for correlation between the proportion of floating point instructions and energy consumption within a unit time is approximately 0.087, which indicates a loose relation. This is be- cause floating point instructions tend to be translated into more micro-operations than non-floating point instructions. Thus, our model does not categorize the instruction type.

3.2. Profiling and estimation

Although many events affect its power consumption, a processor can generally keep track of three events at the same time [25]. Therefore, we set up our estimation model using a few

Because the performance counter values are recorded at every time context switch, it is possible to keep track of the number of events that occurred during the time interval. If the power consumption is determined by the events, Evmi at time t will have the tendency

Evmi,t ∝ C1Ni + C2Mi − C3St , (3)

where Ni is the number retired instructions during the time interval, Mi is the number of memory accesses, and St is the number of active cores at time t. St depends on time t because the number of active cores changes over time. By substituting Evmi in Eq. (2) for Eq. (3), we obtain:

Esystem,t = ued70(C1Ni + C2Mi − C3St ). (4)

i=1

We obtain the coefficients C1, C2 and C3 by conducting multi- variable linear regression over data sets sampled under diverse circumstances. The coefficients are then used to estimate the energy consumption of a VM interval by substituting Ni, Mi and St in Eq. (3) for the measured performance counter values.

The performance counters are read at every context switch to keep track of their changes. To read the performance counters for other cores, a core issues inter-processor interrupts (IPIs) to the other cores so that they jump to the code that reads their performance counters. However, if every core is allowed to invoke IPIs at any time, deadlock may occur when two or multiple cores invoke IPIs simultaneously, as shown in Fig. 6.

To prevent this catastrophic situation, our approach uses a tweak in collecting a time series of performance counter values. Each core records its current performance counter values with the time stamp in an array data structure, ec_priv, at every context switch. At the end of every predefined time interval, usually 300 ms, the master core aggregates the ec_priv data for all cores and reconstructs the time series for utilization of cores, as illustrated

Table 3

Benchmark applications used for experiments (*benchmarks used to obtain the coefficients).

CPU-intensive gamess*, calculix*, namd*, tonto*, gromacs* h264ref, hmmer, bzip2, perlbench, wrf

Memory-intensive gcc, xalancbmk, sphinx3, milc, soplex, dealII* Non-resource GemsFDTD, bwaves, zeusmp

saturating

Fig. 7. Detection of the number of active cores in fine time granularity using per- core array variables.

Table 2

Specifications of the system for evaluation.

Processor Number of cores Clock frequency 4 2.66 GHz L2 cache L3 cache 256 KB per core 8 MB per processor

ISA x86_64 TDP 95 W

System (approximate values)

Peak power 332 W Idle power 200 W

Fig. 8. Estimation errors for power consumption of the profile group for different variables.

in Fig. 7. As well as the core activity history table, each event count between two consecutive time points, T1–T2 in Fig. 7, is recorded. Coefficient determination through linear regression requires the energy consumption Esystem,t for every time interval t. Once the coefficients are obtained, external measurement is no longer necessary to estimate the energy consumption. In contrast to linear regression, the estimation process obtains Esystem,t by replacing NI ,

Mi and St in Eq. (4).

3.3. Evaluation

We implemented our estimation model in Xen 4.0 for evaluation. The hardware configuration and its characteristics are shown in Table 2.

Our evaluation compares aggregate values of the estimated energy consumption for all VMs in a system with the energy consumption measured for the entire system because there is no practical method for measuring the energy consumption of a VM. Consequently, the errors between the two values are not the estimation errors for a VM, but for all VMs in the system. To measure the power consumption, we used a digital multimeter (DMM; National Instrument PXI 4070) and PXI data acquisition interfaces. The DMM was set to sample the power input to the target server every 300 ms.

Each VM was configured to equip a single virtual processor and to run some benchmark programs chosen from the SPEC CPU2006 benchmark suite. Because there are four physical cores in the system, up to four VMs can run concurrently without sharing cores.

The benchmark set for our evaluation is listed in Table 3. The programs are categorized into three groups. The CPU-intensive group consists of programs that have relatively infrequent memory accesses or LLC misses. They have high micro-operations per second in common. The programs in the memory-intensive group frequently access memory and have low LLC hit rates. All programs

Fig. 9. Estimation errors for power consumption of the CPU-intensive group for different variables.

in both the CPU- and memory-intensive groups have resource- saturating workloads. In other words, the processor remains busy and does not idle when the workloads are being executed. The workloads of the last group, the non-resource-saturating group, have significant disk accesses. The processor frequently idles while the workloads wait for the disk access to finish.

If our assumption that the internal activities of processors directly consume energy is correct, selection of workloads to obtain the coefficients in our model should hardly affect the estimation accuracy. We randomly chose six benchmark programs to sample data for linear regression to obtain the coefficients, as shown in Table 3.

We measured the accuracy of our estimation model when the VM workloads were homogeneous. In this experiment, each VM was configured to execute the same benchmark program repeatedly. The number of concurrently ruing VMs was varied from one to four, depending on the time.

Figs. 8–10 show the average estimation errors for the bench- mark program groups according to the number of parameters used

Fig. 10. Estimation errors for power consumption of the memory-intensive group for different variables.

Table 4

Constitution of the mixed workloads.

Set Benchmarks

Set 1 gamess, GemsFDTD, calculix, gcc

Set 2 hmmer, bwaves, bzip2, xalancbmk

Set 3 gromacs, tonto, sphinx3

Set 4 h264ref, perlbench, milc

Set 5 dealII, wrf

Set 6 soplex, zeusmp

Fig. 11. Normalized errors for mixed workloads.

for estimation (number of retired instructions, number of active cores and memory access activities). The errors are normalized to the total energy consumption, which included both leakage and dynamic power.

Fig. 8 shows the estimation errors when the model was applied to workloads used to obtain the model coefficients. Overestimation and underestimation are reflected by positive and negative error values, respectively. Because the model coefficients were determined from these workloads, the high estimation accuracy is a natural outcome. However, when only the micro-operation count is considered, the estimation error increases to as much as 8%; most of the errors come from overestimation.

We used only one VM utilizing a single core for the estimation model based on just the number of micro-operations to exclude the effects of resource sharing among multiple cores. When we applied the coefficients to cases for which multiple cores were in use, the enhanced energy efficiency arising from the shared processor components led to overestimation of the energy consumption. Thus, the estimation accuracy improves dramatically when the model includes the number of active cores.

The results shown in Fig. 9 are the errors when the model was applied to the workloads not used to obtain the model coefficients. The estimation accuracy of this experiment is similar to that shown in Fig. 8. The memory access count hardly affects the accuracy because the workloads shown in Figs. 8 and 9 access memory infrequently.

When a model does not consider the memory access count, it tends to underestimate the workload energy consumption. The model considering only micro-operation counts shows better accuracy for memory-intensive workloads than for CPU-intensive workloads. This is because the underestimation tendency for memory-intensive workloads offsets the overestimation tendency of the micro-operation-only model. The estimation model that includes memory access counts shows errors of less than 2%.

We also evaluated the suggested scheme for cases in which multiple VMs with heterogeneous workloads were run simulta- neously. We randomly created six mixed workloads, as listed in Table 4.

Fig. 12. Estimated and measured energy consumption per 300-ms interval during execution of diverse workloads.

The estimation errors for the mixed workload sets are shown in Fig. 11. For all the workload sets, the error rate is less than 3% and ranges from 0.5 to 7.0 Joule/s. Dividing the error range by the number of VMs yields the per-VM error, which ranges from 0.5 to

3.5 Joule/s.

Fig. 12 shows time series for measured and estimated energy consumption while ruing a series of workloads randomly chosen from Table 3. In most cases, our model estimates energy consumption with low errors despite the existence of drastic changes in energy consumption. The average error for this experiment was 0.96 Joule/s.

4.1. Energy-credit scheduler

The Credit Scheduler, the default VM scheduler of Xen, is a variant of fair-share schedulers that distributes processor time to VMs according to their credit values. We propose a modified Credit Scheduler, which we call the energy-credit scheduler (ECS), that schedules VMs according to their energy budgets instead of processor time credits.

In the ECS, similar to the Credit Scheduler, each VM is assigned its own energy credit for a fiscal interval. Because the time slice for VM scheduling is as long as 300 ms, the fiscal interval must be longer than 600 ms, which is twice the length of a time slice. The energy credit unit for a VM is joules per fiscal interval.

In the conventional Credit Scheduler, the processor time allocated to a VM is determined by the proportion of the credit of that VM to the sum of all VM credit values. Therefore, the credit

Fig. 13. Flow chart for implementation of the power-aware scheduling scheme.

value for a VM is interpreted as the guaranteed minimum processor time dedicated to that VM. For the sake of throughput of the whole system, general schedulers including the Credit Scheduler are work-conserving schedulers that keep the system busy by redistributing credit values when there are ruable tasks and none of them have available credit values.

On the contrary, the energy credit of the ECS acts as the top bound for the scheduling time because its purpose is to limit the energy consumption rate of each VM to its energy budget. When all ruable VMs use up their energy credits, the system idles until the new fiscal interval of any ruable VM begins. In other words, the ECS is a non-work-conserving scheduler.

Fig. 13 shows a flow chart of the scheduling algorithm and the corresponding scheduler function names in our prototype implementation. The algorithm is executed every 300 ms, which is T1–T2 in Fig. 7. This time interval is different to the energy fiscal intervals of VMs, which may differ from each other.

At the end of every time interval, do_burn_energy subtracts the estimated energy consumed by a VM during the interval from its remaining energy credit. If a VM has no remaining credit, the scheduler takes the VM out of the scheduler queue until it obtains credit again when the next fiscal interval begins.

The energy consumed by a VM during a fiscal interval is obtained using the suggested estimation model. The energy consumption estimation model in the ECS utilizes all three parameters.

VM energy credits are distributed on first execution of the scheduling algorithm after a new VM fiscal interval begins. Therefore, the time granularity for energy credit provision is 300 ms. When a VM excluded from the scheduler queue earns an energy credit, the scheduler puts this VM back in the queue.

4.2. Management interface

For practical application of the suggested scheme, we designed and implemented external interfaces that help administrators to identify and control the energy consumption rate of each VM.

The Xen hypervisor has a special VM, domain 0, that is dedicated to monitoring and control of other VMs in the system. Domain 0 is also responsible for handling I/O requests from the other VMs by executing system calls to the hardware on behalf of the requesting VMs. As shown in Fig. 14, domain 0 issues commands to the hypervisor or collects data from the hypervisor through hypercalls.

Table 6

Capping errors for various workloads and capping configurations.

	Configuration (J/s)	Measured energy	Error rate
	VM1 VM2 VM3 VM4	(J/s)	%

leslie3d astar – – 247.81 1.13

20 25

perlbench sjeng – – 248.30 1.33

20 25

wrf gobmk sjeng – 263.53 1.36

15 20 25

h264ref calculix perlbench – 265.53 1.36

15 20 25

calculix gobmk sjeng astar 273.93 1.46

10 15 20 25

h264ref wrf perlbench leslie3d 274.19 1.55

10 15 20 25

Fig. 14. Diagram of interfaces between the Xen hypervisor, the Linux kernel and user-level processes.

Table 5

sysfs files for monitoring and managing VM energy consumption rates.

interval Length of the fiscal interval

consumed Estimated energy consumption during the last interval

designated Designated energy consumption rate for an interval

A virtual file system called sysfs [27] is used as the communication chael between the hypervisor and domain 0 and between domain 0 and the system administrators. Administrators access the internal data of the hypervisor by reading files in the sysfs file system, and pass parameters or data to the hypervisor by writing data in the files.

Each VM has its own energy-credit interface directory in the energy_capping directory in sysfs of domain 0. The directory contains files that provide interfaces for monitoring and control of the energy consumption rate of the corresponding VM. Three files are provided for each VM, as listed in Table 5. Because these interface files are separate from the interface files for the other VMs, a VM can have its own credit value as well as its own fiscal interval.

When administrators try to write to or read from these files, the kernel of domain 0 is notified. Reacting to this notification, as shown in Fig. 14, the kernel receives the required information from the hypervisor or transmits the given parameters to the hypervisor through hypercalls.

The interface directory of a VM is created by xend, a system

management daemon working in domain 0. In addition, the interface directory is removed by xend when the VM shuts down or crashes.

4.3. Evaluation

The ECS was implemented in the Xen hypervisor and was evaluated using the environment described in Section 3.3.

To evaluate the accuracy of energy provisioning, we used the workload configurations listed in Table 6. Between one and four VMs were run simultaneously, each executing a different workload with a different energy budget. Although a VM is free to choose its own fiscal interval, all VMs in our experiment were assigned a fiscal interval of 1 s for ease of analysis.

As for evaluation of the estimation model, we compared aggregate values for the designated energy consumption of all VMs with the energy consumption values measured for the whole server. Table 6 lists the measured energy consumption and energy provisioning errors. According to the experimental results, the error rate was less than 2% of the total energy consumption.

Fig. 15. Time series for the designated and actual energy consumption rate when the energy budget of four VMs dynamically changes. The fiscal interval was set to 1 s.

The errors are mostly positive values, which means that measured energy consumption exceeds the energy budget. This is because our approach is reactive and not proactive. Our scheduler suspends a VM only after it uses up its energy budget. Therefore, a VM tends to consume slightly more energy than its budget. If the energy budget must be strictly guaranteed, the algorithm should be improved to a proactive form.

Fig. 15 shows time series for changes in the designated energy budget and the energy consumption rate measured. Four VMs ran concurrently while their energy budget changed dynamically.

The energy consumption rate rapidly followed changes in the energy budget. However, on a short time scale such as 5 s, the difference between designated and actual energy consumption varied significantly. This is because the constitution of processor activities that affect the energy consumption rate changes according to the workload phase. In spite of the short-term inaccuracy, our scheme provides energy according to the energy budget in the long run.

The overhead of the proposed scheme is negligible because the complexity of the scheduling algorithm remains the same as that of the original Credit scheduler and profiling routines consist of a few assembly instructions. In our experiments, we could not identify any significant changes in execution time or throughput.

The cloud computing model is gaining momentum because of improvements in energy efficiency and reductions in infrastruc- ture costs. Conventional billing systems for cloud computing are