Ex Post Evaluation of Large Electricity Consumer Policy Measures


 All European Union (EU) member states till 2020 had an obligatory target for energy efficiency. One of instruments for countries to achieve national energy efficiency goals is obligation for large companies (LC) and large electricity consumers (LEC) to implement certified energy management system or perform energy audit. In this study the Latvian case study of obligation for LC and LEC where examined. The analysis was carried out using a theory-based policy analysis method combined with evaluation criteria from the EU legislative assessment guidelines The Better Regulation Agenda – efficiency, effectiveness, relevance, coherence, added value, validity, complementarity, coordination, equality, sustainability and acceptability. To evaluate energy efficiency policy measures, it is also important to understand energy efficiency measures that will realistically meet the set company and national targets. AHP and TOPSIS analyzes were performed to evaluate these measures not only from energy efficiency but also from environmental, climate, engineering-technical, economic, and social aspects. The results allow us to assess the fate of existing policies and to draw conclusions on the improvements needed to meet energy efficiency and climate goals in the future.


INTRODUCTION
One of the recent IPCC special [1] reports on how to stabilize the temperature increase at 1.5 °C by the end of the century shows that end use energy efficiency plays a key role in achieving the overall energy and climate goals. The European Union (EU) Energy Efficiency Directive 2018/2002 (EED) has established an energy efficiency target for 2030 of at least 32.5 % (compared to projections of the expected energy consumption in 2030) [2]. However, the new EU policy initiative The European Green Deal is even more ambitious target for 2050 for Europe to become climate neutral [3]. This will increase energy efficiency target for 2030 even more.
To achieve member states national end-use energy saving targets, the energy efficiency policy measures are implemented. As required by the Energy efficiency directive (2012/27/EU) [4] member states can implement different policy measures including measures for large companies and large electricity consumers that can achieve end energy savings. The introduction of an obligation on large electricity consumers is not only beneficial in terms of the savings to be achieved, but it is these companies that are the first to benefit from energy efficiency measures to improve competitiveness [5]. Energy efficiency measures in households also depend more on consumer behavior and habits than on economic potential [6] in turn, large consumers, for whom energy accounts for a significant share of costs, are more willing to invest in new technologies and behavioral change [7].
In the first planning period of EED one of the measures implemented in Latvia is energy audits or energy management systems for large companies and large electricity consumers. Similar policies have been introduced in other EU Member States, but the results are mixed, mainly due to the balanced carrot-and-stick principle [8].
Of course, audits and International Organization for Standardization (ISO) published Energy management system standard ISO 50001 implementation systems are an obligation from state, but the real energy efficiency savings are provided by the energy efficiency measures taken. That is why it is important to evaluate both the measures taken now and to anticipate what measures could be introduced in the future. Analysis of the effectiveness of the measures taken can also help in the development of future legislation [9]. In Fig. 1 the main factor for developing and implementing energy efficiency measures is shown. The main aspect is to present clear objectives and precise powers of the policy implementer, and involvement of stakeholders, such as companies, utilities, and society [10]. Also, the abilities to balance and combine flexibility and continuity is as much important as an ability to adapt and merge different previously existing policies for consistent measures.
The aim of this paper is to analyze the effectiveness and compliance of existing policy options with the Better Regulation Agenda (BRA) and to make recommendations for future policy options, a TOPSIS analysis of the most energy efficient solutions for large consumers was carried out. So far, no analysis of this type of policy instrument has been performed in the Latvian case study.

METHODOLOGY
Several analytical methods were used in this study. One of them Ex post evaluation of energy audits/energy management systems of large companies and large electricity consumers is carried out by applying theory-based policy analysis method in combination with the criteria from the BRA guidelines is used to study policy implementation process and progress [11], [12]. Selected methodology includes not only an assessment of what has happened but also looks for answers to why something has or has not happened and, if possible, how much has changed. This method has been used in the past to analyze energy efficiency policy instruments, for example in the household segment in the construction and building energy efficiency segment, however, the segment of large companies and large electricity consumers has not been analyzed before [13], [14].

Combined Ex Post Evaluation of Energy Efficiency Policy Measures
The study uses a theory-based policy analysis method [15]. This method is intended to systematically assess all phases of the policy implementation process, success, and failure factors, as well as end-effects such as target achievement, impact of energy savings and cost Clear objectives and powers of the policy implementer Ability to balance and combine flexibility and continuity

Involvement of stakeholders
Ability to adapt and merge different existing policies or to develop consistent sets of policy measures effectiveness, and factors of success and failure. The general principle of this approach is that a possible theory is developed on the various stages of the intervention logic of a policy measure to achieve its objective of increasing energy efficiency. This approach has several advantages compared to other ex post evaluation methods, since it is − Evaluating the whole process of policy implementation and not focusing solely on final impacts; − Developing indicators for each phase of the implementation process. This helps assess progress and failures as widely as possible; − Helping to find out not only whether policies are successful or not, but also why they are successful or unsuccessful and how they can be improved. The theory-based policy analysis method is an iterative process for designing, evaluating, and transforming a policy measure, based on lessons learned during the initial implementation period. In practice, this means that a theoretical policy assessment creates a credible theory on how the policy measure is expected to improve energy efficiency and at what point. The basic idea is to divide the whole process of policy implementation to gain insight into where the wrong assumption was made in the process of policy development and implementation and where improvements in impact and cost effectiveness are of key an explanation was included in the publication importance. The theory-based policy analysis method includes six steps shown in Fig. 2. The theory-based policy analysis method is used in the following order: − Step 1 -policy measure is described. This includes a description of the objectives, a period when the policy measure was active, target groups, policy implementers, available budget, available information on the impact of the originally expected energy savings and the cost-effectiveness of the measure; − Step 2 -a policy theory is created. The policy theory includes all assumptions about how a policy measure should achieve the desired effect. The policy theory can be clear or indirect. Ideally, a clear theory is available. This means that policy makers have clearly described how, in their view, the policy instrument will work before it is introduced. It is that they have made clear which member has to act and that they have announced the expected outcome of each action. Often, the theory is largely indirect, and such a description is lacking. In this case, the evaluator draws up a theory. The development of a policy theory in practice involves documenting all indirect and direct assumptions in the policy implementation process and mapping the relationship between causes and consequences, including relations with other policy instruments; − Step 3 -policy theory is transformed into specific and desirable quantitative indicators. This means that an indicator is established for each intended causal and effect relations to measure whether the cause-to-effect relation has occurred and to measure whether the changes occurred (or part of) under the influence of the policy measure. This phase also includes the development of the necessary equations for the calculation of impact and cost-effectiveness; − Step 4 -the relationship between causes and effects and the indicators are reflected visually in the chart; Step 1 Description of a policy measure Step 2 Design of policy theory Step 3 Change policy theory to indicators Step 4 Visualization of policy theory Step 5 Policy theory check and adjustment Step 6 Analysis and conclusions − Step 5 -review and, if necessary, adjustment of policy theory. In step 2, the policy theory was developed using available (official) documents or experience with similar policy measures. In the fifth step, the policy theory is tested through interviews with policy makers, implementers and other actors involved in the implementation and supervision of a policy instrument; − Step 6 -includes: a) summary and analysis of available information to establish indicators; b) conclusions that are drawn on the impact and cost-effectiveness of the energy savings of the policy instrument using equations and indicators; c) analysis of progress and failures attributable to the measures analysed; d) recommendations that are made to improve the efficiency of energy savings and costs. The theory-based policy analysis method is linked to the criteria offered by 'The Better Regulation Agenda'. This is done by including the following criteria on the indicators shown in Fig. 3.
The theory-based analysis criteria: − Effectiveness: The evaluation should include analysis on progress towards achieving the objectives. It should be based on evidence on why, whether and how these changes are related to a policy measure. The answer to this question should be broader than just showing whether the measure is on the right track. The analysis should aim to identify the factors driving or delaying progress and how they are linked (or not) to a policy measure. The analysis should also try to determine whether any expected or unexpected effects have occurred; measure. The answer to this question must provide evidence of actual costs and benefits, with a clear indication of what can be attributed to a policy measure, and which cannot. The efficiency analysis is a key contribution to policy making by helping both policy makers and stakeholders to draw conclusions on whether the costs of a policy measure are proportionate to the benefits. Where appropriate, the conclusions of the assessment should clearly identify areas where it is possible to reduce inefficiency (particularly unnecessary administrative costs) and to simplify the policy measure; − Relevance: the assessment should look at the objectives of the policy measure under assessment and identify how well they (still) meet (current) needs and challenges.
The answer to this question should identify whether there is a discrepancy between the objectives of the policy measure and the (current) needs or problems. This is the key information that will help policy makers decide whether to continue, change or stop intervening. Relevance analysis is very important because if a policy measure does not help to address current needs or problems, it does not matter how efficient, effective, or coordinated it is; − Coherence: the evaluation should look at how well the policy measure works: internally and with other policy measures. The answer to this question should provide evidence of where and how policy measures work well together (for example, in order to achieve common objectives or complementary actions) or identify areas where tensions exist (e.g. targets that can potentially be contradictory or approaches that create inefficiency). Even minor changes to the planning or implementation of one intervention can lead to improvements or discrepancies in other ongoing activities.
The assessment of coherence involves how good or different measures work together; − Value added: the assessment should consider arguments on the value of a policy measure, which is in addition to the value that could be created by policy measures initiated at regional or national level by both public authorities and the private sector. For spending programs, the added value of a policy measure can be created by a variety of factors, such as benefits in coordination, improved legal certainty, greater efficiency, or complementarity. The analysis of the added value of a policy measure is often limited to qualitative, given the identified difficulties in identifying the hypothetical situation; − Validity -to what extent is the policy measure does or does not satisfy the needs of stakeholders? How much is the difference between the satisfactions of the various stakeholders? − Equality -how fair are the effects shared between different groups of society, e.g., genders, regions, social groups, etc.? − Sustainability -how much is the likelihood that the effect of the policy action will continue after the end of the measure? − Acceptability -to what extent can it be a changed in the perception of a policy measure in the audience and in general in society? − In addition, the theory-based policy analysis method is complemented by several important aspects which play a key role in the process of developing and implementing energy efficiency policy instruments: 1. The existence of clear objectives and the powers of the policy implementing body -organization or program with clear powers, responsibility and adequate resources is the most important prerequisite for success; 2. The ability to balance and consolidate flexibility and continuity -the continuity of the program is important and can be achieved either by strong political support to civil servants or by setting up independent agencies that implement policies. Several studies have found that a key success factor is continuity, which is characterized by stable and predictable conditions. Equally important is flexibility, characterized by the ability to adapt to changing conditions and the ability to reduce potential failure factors in the implementation process. Technical and market changes, organizational changes and changes in other policy areas can motivate adaptation. A number of studies have recognized the importance of the implementation agency's ability to quickly reduce specific barriers, such as the lack of information, tools and skills, for which guidelines, procedures, analytical tools, educational programs, etc. should be developed; 3. Involvement of stakeholders: involvement of stakeholders in the design and implementation of policy measures is recognized as an important success factor. It serves the dual purpose of increasing acceptance and at the same time improving efficiency by ensuring that the needs and expectations of stakeholders are considered throughout the process. The stakeholders are both target group members and various other actors that may be affected by a policy measure, such as energy auditors, energy consultants, and equipment suppliers. A common feature is that stakeholders assess simple and clear rules as well as a short deadline for examining documents when decisions are to be taken by the implementing agency. The involvement of stakeholders contributes to flexibility. The implementation agency, which has contacts with stakeholders but also has the power to adapt and improve the policy measure, is more likely to succeed; 4. The ability to adapt and consolidate different existing policies or to develop consistent sets of policy measures. The policy analysis uses different data sources: − Information provided by the Ministry of Economics from the energy efficiency monitoring system, where unverified and unprocessed data from large companies and large electricity consumers and municipalities are available; − Interviews with involved parties; − Publicly available information. During the policy implementation process, this policy analysis approach can help to identify problems and barriers at an early stage and allow the policy instrument to develop and adapt to changing technical, organizational, economic, and other conditions. An efficient and efficient energy efficiency policy focused on low-cost savings will play a key role in achieving the energy and climate policy goals beyond 2020. Therefore, the assessment should not only include the identification of savings as set out in the EU Directive, but also include approaches that encourage continuous policy analysis, learning and implementation.

Analytic Hierarchy Process and Multiple Criteria Decision Analysis Using TOPSIS Method
Multi-criteria decision analysis (MCDA) is widely used to solve various decision problems through alternative evaluation. MCDA methods can be used in various fields and main condition is to define a problem, alternatives and criteria [16].
The analytic hierarchy process (AHP) is one of the MCDA methods that originally presented by Saaty in 1977 [17]. AHP is used to determine the relative importance or weight of criteria, according to which alternatives can be ranked according to qualitative and quantitative criteria [18]. In this method, the criteria are compared in pairs and values (1 to 9) are assigned according to the importance of the criterion [19].
TOPSIS method was presented by Hwang and Yoon in 1981 and it is one of the most suitable method to use for finding solution for problem [20]. This method defines ideal and non-ideal solutions and distance between these solutions and sort alternative by closeness to ideal solution [21]. TOPSIS method is simple, easy to calculate solution and is with understandable logic that represents human choice [22].
Other multi-criteria analysis methods were identified, such as VIKOR, COPRAS and MULTIMOORA were analyzed however, this method was chosen because it allows ranking solutions as well as the available input data were sufficient for reliable and qualitative analysis.

Information on Energy Efficiency Policy
Energy Efficiency Law aims to ensure the availability of energy audits and regular, mandatory energy audits in large companies. Energy Efficiency Law has entered into force on 29.03.2016 and is still in force. The target audience of policy measure are large companies and large energy consumers.
Large companies (merchants employing more than 249 employees or accounting year turnover exceeding EUR 50 million and annual balance sheet of 43 MEUR. Large enterprises must be listed each year by December 1 by the Central Statistical Bureau, using data from the last approved reporting period of the company. An entity is included in the list of large companies if it meets the criteria of the large company in two consecutive reporting periods. Large electricity customers (merchants whose annual electricity consumption exceeds 500 MWh. The energy produced by the company, which is transferred to other users, has to not be included in the total energy consumption). The system operator has a duty to provide the Ministry of Economics each year with the annual energy final consumption data of companies conforming to the status of the large electricity consumer. The procedures by which the system operator must provide the annual energy final consumption data of companies conforming to the status of a major electricity consumer to the responsible Ministry are determined within the framework of the energy efficiency monitoring system.
There was lack of funding from the state budget to achieve this policy measure, however since 2019 there was support from government financial institution ALTUM to implement energy audits [23].
The Energy Efficiency Policy Alternative Action Plan to achieve the energy end-use savings target for 2014-2020 provides that energy audits in large companies will assess the energy consumption of companies and identify measures to improve energy efficiency and will bring the accumulated energy savings to 753.6 GWh by 2020. On the other hand, the introduction of energy management for large electricity consumers will deliver the accumulated energy savings of 54 GWh by 2020.
The implementation and maintenance of the energy plans, energy management or environmental management system are funded by large companies and large energy consumers. They also finance energy efficiency measures and their implementation.
Energy Efficiency Law provides that the existence of a certified energy management system is a favorable qualifying criterion, which is taken into account when assessing applications for large electricity consumer projects, if the project in question is fully or partly implemented by means of aid such as payments from the state or municipal budget, state or municipal guarantees, the subsidy of interest rates on loans financial assistance, as well as other financial assistance, which is granted or provided from the state, municipalities or European Union budget resources and foreign financial assistance funds.
The Energy Efficiency Law requires large companies to carry out energy audits on a regular basis. The first energy audit must be carried out within one year after the inclusion of the company in the list referred to in the second paragraph of this Section. The energy audit must be carried out every four years. This requirement does not apply to large companies if they introduce and certify an energy management system or introduce and certify an environmental management system and provide a continuous process for evaluating energy consumption to control and reduce energy consumption, covering at least 90 % of the total energy consumption of the large company and ensuring compliance with the energy consumption assessment process described in the law.
Large electricity consumer has a duty to introduce and maintain a certified energy management system in conformity with the standard, covering at least 90 % of the total final energy consumption of the large electricity consumer and ensuring compliance with the 29 procedures for evaluating energy consumption with the conditions of this Law. This requirement does not apply to the large electricity consumer if it has introduced an environmental management system by the date of coming into force of the Law and has supplemented it and certified it within six months from the date of coming into force of the Law in order to control and reduce energy consumption, covering at least 90 % of the total energy final consumption and ensuring compliance with the energy consumption assessment process with the conditions of this Law. The procedures for energy management and environmental management standards, which are applicable to the fulfilment of the requirements of this Law, in which the environmental management system has to be supplemented and the approval of such an environmental management system, which have to ensure the continuous process of evaluating energy consumption, and the addition has to be determined by the Cabinet of Ministers. These requirements do not apply to the large electricity consumer, if it regularly provides an energy audit. The energy audit must be carried out every four years.
Large company has to implement at least three energy efficiency measures proposed in the energy audit or in the framework of the certified energy management system referred to in paragraph six of this Section or in the framework of the certified environmental management system, with the highest energy savings or economic returns assessed.
Large electricity consumer must introduce at least three energy efficiency measures recommended in the framework of the energy management system or the environmental management system, with the highest energy savings or economic returns assessed.
If the above-mentioned requirements are not met, undertakings must pay a fee of a rate of seven per cent from the costs of electricity consumed in the year. These costs must be calculated by multiplying the megawatt hours consumed in the year concerned by the average electricity price published by Eurostat in Latvia in the previous year (euro/MWh). Revenues from the energy efficiency charge are transferred to the national Energy Efficiency fund.
Large companies and large electricity consumers must report to responsible authority about implementation of an energy audit or the implementation of a certified energy management system or a certified environmental management system, the proposed energy efficiency improvement measures, as well as report annually on the implemented energy efficiency improvement measures and the energy savings resulting therefrom. The legislation determines the reporting procedures. Reporting principle consists of LC or LEC submitting energy audit or inform about implemented energy management system. Then till 1 November companies report about achieved energy savings and Ministry of Economics collects all information and publishes it by 1 March of next year. The data lag is minus one year.

Results for Ex Post Evaluation
Estimates show that until now the administrative costs of implementing this policy measure have not exceeded the planned costs. Although the objectives of the policy measure are in line with society's current needs and challenges, and policy measure are adapted to technological, scientific, environmental, and social changes, the challenges associated with its implementation process reduce its implementation rate, which in turn prevents the achievement of national targets.
The added value of this policy measure is information on energy efficiency, which compulsively goes to companies and remains in long-term memory, although often not used in the short term.
The lack of a systemic approach on the part of the legislator and the Responsible Authority (MoE) leads to an entirely different effect than planned when designing this policy measure. On one hand, the legislator is obliged to introduce mandatory energy audits / energy management systems and measures and provides for penalties for non-compliance ("stick") and makes the implementation process chaotic by suddenly changing the rules, but on the other hand does not provide for supportive measures, including not publishing companybinding information that would boost their motivation, does not make interpretative communication with the audience, delays in creating explanatory materials, no targeting measures, no feedback, etc. ("carrot").
This policy measure increases public awareness of energy efficiency, since the employees of companies who have confronted it, also disseminate this experience outside their workplaces. There are only a few companies that share their positive experiences in mass media and social media. This may be due to a lack of financial resources for these measures and could be supported more by the state.

TOPSIS Analysis of Suitable Energy Efficiency Measures
To further assess the effectiveness and potential improvement of this policy instrument, a TOPSIS analysis of potentially most suitable energy efficiency measures was also carried out.
With AHP method was compared environmental, climate, engineering-technical, economic and social aspect. Experts evaluated these aspects and in Table 1 is AHP method normalize matrix. In TOPSIS method is used criteria weight from AHP method and on Table 2 are also best value for criteria and in this case for all criteria best value for alternatives is maximum (max). In this case biggest criteria weight is 0.37 and it is for environmental aspect and this criterion will have the biggest impact to evaluation process for energy efficiency measures which is as alternatives.  Using input data about each alternative and criteria is made matrix and after calculation with criteria weight is normalized weighted decision matrix Table 4 and after that for each criterion is defined ideal or best and non-ideal or worst solution.   Fig. 4 shows results from TOPSIS and best alternative is with higher value or highest closeness to ideal or best solution. In research for best energy efficiency measure highest results is for alternatives -use of renewable energy sources with 0.72 and audit with 0.71 value. The reason why the use of renewable energy sources is the most appropriate measure in the LC and LEC segment can be explained by the fact that the greatest weight is given to the environmental and climate criteria. Experts explained this by the sharp rise in the price of CO2 allowances, which suggests that it is the climate criteria that will determine the total costs of different measures. It is also planned to place even greater emphasis on the use of the Emissions Trading Instrument for sustainable development in the new programming period. In addition, the installation of renewable energy sources is already paying off in the foreseeable future, so forecasts of rising CO2 prices suggest that companies that have already installed systems will be able to be more flexible and more competitive than those relying on the business as usual scenario.
An energy audit was recognized as the second most appropriate measure, based on expert advice, as it has relatively low costs and the savings in many cases far outweigh the investments. Experts also mention that if initially the audit was mainly performed as a mandatory obligation, now companies are already assessing the potential benefits of highquality consumption measurements and analysis. Also, the fact that the level of competence of energy auditors has significantly increased can be mentioned as a more positive attitude of companies. Experts say that the use of energy monitoring solutions will also be very promising, as energy efficiency audits are a snapshot of the current situation, but to really make their operations efficient in the long run and under different operating conditions, it is important to constantly monitor resource consumption and optimize it.
Next, among the most promising measures, the replacement of windows is mentioned, which can be considered as a relatively easy-to-implement solution with the expected savings potential. Also, the modernization of the heating system allows to significantly reduce the company's costs, not only for energy resources, but often also to reduce labor costs and time by replacing conventional heating solutions with automated boilers.
Value for other measures where relatively lower, but however, their potential depends very much on the specifics of the companies and the distribution of energy consumption, as this analysis is more appropriate for the medium-sized segment of large consumers than for a specific sector.

CONCLUSIONS AND POLICY IMPLICATIONS
− The AHP method was used to assess the materiality of the criteria, and the environmental and climate criteria were the most important criteria for selecting the measure. Experts considered this to be the most important consideration for the ambitious climate goals for the coming decades; − The best alternative measures based on TOPSIS analysis where usage of renewable energy sources. Main reasons for that are reduction of environmental impact and acceptable payback period, which allows to achieve long-term improvement of the company's sustainability indicators and ensures an increase in competitiveness at fluctuating energy and emission prices; − The other measures with greatest potential based on expert evaluation and analysis where energy audit in company and energy monitoring system, which allow for a relatively inexpensive understanding of the necessary improvements, as well as continuous monitoring of their implementation and follow-up; − Ex post evaluation method shows shortcomings in the design and communication of the policy instrument with the responsible party can be identified. This analysis made it clear that, although the intentions of this instrument were mutually beneficial for both the state -to meet the targets and the companies -to reduce costs, inadequate communication and deficiencies led to lower savings and resistance from the industry; − The TOPSIS analysis makes it possible to identify the most promising energy efficiency measures specifically for large companies and consumers, which can help to develop concrete policy instruments to promote these measures and achieve savings.