Building a sustainable organizational energy evaluation system in the Asia Pacific

1 Introduction

In the face of climate change and with the vision for the future of humanity represented by the Sustainability Development Goals (see below),sustainability has become associated with the major and rapid transformation of global systems and the resilience of transformed systems to adapt over time.In fact,sustainable development has been a key issue and concern in the Asia Pacific for many years.Specifically,the Asia Pacific Economic Cooperation (APEC),a regional economic forum established in 1989,has been focusing on the sustainable development and protection of the environment since the early 1990s,as noted in its Economic Vision Statement of 1993:“Our environment is improved as we protect the quality of our air,water and green spaces and manage our energy resources and renewable resources to ensure sustainable growth and provide a more secure future for our people ” [1].APEC has reaffirmed their vision statement over the succeeding years,and specific initiatives and work plans have been implemented on sustainable development.Furthermore,APEC has been conducting an annual review of activities of sustainable development in APEC as well as coordinating and providing guidance to APEC programs and projects.

The major and rapid transformation of global systems has implications for both the design of transformational energy initiatives and their evaluation.Despite the activities noted above in the Asia Pacific,there is an urgent need to evaluate the scale,scope,nature,and sufficiency of transformation toward planetary sustainability.Specifically,as noted below,a framework (model) is needed to determine the most effective energy evaluation capacity building (ECB) strategies for achieving various individual and group level outcomes,most effective strategies for certain types of participants,and design of different strategies to maximize their impact.In addition,because leadership is critical for building a sustainable energy evaluation system,a new energy evaluation organization is identified in this study,which is expected to lead us into this new era.

2 Sustainability and evaluation

Sustainability is a central concern arising due to the crisis of climate change.As a result,increasing attention has been focused on the challenges faced in evaluating sustainability,particularly because sustainability is considered as a more general concept that is used in many domains [2-4].Furthermore,sustainable development is more likely to be effectively achieved by building sustainable organizational evaluation systems [5].

The need to investigate evaluation systems,their effectiveness,and their contribution toward sustainable development has become more pertinent in recent years with the establishment of increasing numbers of organizational evaluation systems.For example,internationally,the Sustainable Development Goals (SDGs; see below) require country-led evaluation as part of the follow-up and review processes [6].In addition,a multidisciplinary model of an ECB is needed to provide the framework for designing and implementing these comprehensive evaluations.

Unfortunately,the evaluation experience of energy efficiency and renewable energy programs and policies is limited in many countries in the Asia Pacific,for at least four reasons [7].First,government initiatives (outside of investment banks,such as the World Bank and the Asia Development Bank) often do not include evaluation as a principal activity,and instead focus their efforts on the design and implementation of energy programs.Second,the funding of data collection and evaluation of programs and policies is often not available or is of a low priority.Third,expertise (trained evaluators) is often limited; while international companies may offer such expertise,experienced country-based evaluators are lacking.Finally,the evaluation data are often lacking or not standardized (the latter is particularly important for comparisons among countries or cities).

In response to reducing the carbon emissions for improving the local and regional air quality and to address climate change,economies are in the process of developing and implementing new energy policies and programs in all sectors; as a result,there are several opportunities to make evaluation more prominent.First,evaluation can be introduced early in the planning process as the design and implementation of energy programs is initiated.Second,expertise in the public and private sectors can be increased for data gathering and evaluation.Third,there are opportunities to assess the evaluation of private sectors’ investments toward energy policy goals.Fourth,the body of knowledge on the effects of energy policies and programs can be increased and standardized.Fifth,the lessons learned from evaluation can be used to improve the energy policy,as well as the program design and implementation.

As discussed below,the above activities must be designed and implemented strategically,using a multidisciplinary model of the ECB that can integrate the different functions.Furthermore,as sustainability becomes more than an aspirational goal,the ECB also needs to be sustainable.

In summary,it is of utmost importance to increase the evaluation in the planning process,increase expertise in the private and public sectors for ECBs,increase the body of knowledge on the effects of energy policies and programs,and improve the energy policy,as well as the program design and implementation.

3 Evaluation capacity building

Given the amount of research on what makes for a good evaluation,there has been surprisingly little research on what makes an evaluation system sustainable [5].In one recent review of publications on evaluation systems [5],the authors defined the components of a successful evaluation system as (1) a use orientation (i.e.,the products and processes are useful),(2) good quality evidence (through using common standards on methodology and practice),and (3) networks/networking (focusing on improving cooperation and collaboration).This aligns well with the conceptual framework proposed by Preskil and Boyle [8],wherein sustainability is treated as a higher-level result of an evaluation system that is supported by evaluation use,quality evidence,and networks.

Preskil and Boyle’s multidisciplinary model of the ECB (Fig.1) provides the necessary framework for determining the most effective ECB strategies for achieving various individual and group level outcomes,most effective strategies for certain types of participants,and design of ECB strategies to maximize their impact.The model is not a cookbook or a manual on how to conduct the program and policy evaluations.Instead,the model provides a visual and theoretical framework that lays the groundwork for developing a sustainable evaluation community-for example,in the Asia Pacific.

Fig.1 Multidisciplinary model of evaluation capacity building

Evaluators can use this model to explore which ECB strategies are most effective for achieving various individuals and groups,and how these strategies can be designed to maximize their impact.As depicted in the left circle,evaluators can focus on technical assistance,coaching,internships,written materials,technology development,meetings,training,and conducting evaluations.Evaluators need to continually evaluate these strategies to determine the most effective one - not only after design and implementation but also during design and implementation.Additionally,evaluators need to identify the motivations,assumptions,and expectations of the targeted individuals and groups during the design of the ECB strategies,such that the strategies respond to the needs of the individuals and organizations.It would be irresponsible and unprofessional for the evaluators to not take into account these needs.

As depicted in the space between the two circles,leadership is critical in promoting ECB [5][9].An evaluation system that is successful and in use requires that there is demand for evaluations emanating from leaders in organizations; additionally,structures need to be put in place so that the evaluation use is institutionalized rather than episodic [5][9].Furthermore,evaluators need to recognize the local culture,systems,and structures in place,so that they can help shape the demand for evaluation,and communicate the ultimate goal of sustainable practice [5][9].

Incentives (which are built into policies,procedures,and norms) and established organizational practices are required to maintain the current demands and elicit new demands for evaluation,as well as make the evaluation less scary,more routine,and with defined channels that feed into the accountability mechanisms [5][10-12].This mix of incentives is described in terms of carrots,sticks,and sermons to sustain evaluation:(1) Carrots to encourage the organization to undertake evaluation,such as budget allocations linked to evaluation use (particularly,the use of evaluation evidence).(2) Sticks,such as withdrawal of funding,to penalize persons for not undertaking or using the evaluation.(3) Sermons to extol the importance of evaluation (such as senior leaders talking about how they use evaluations).Incentivizing new demands is important as it will help the evaluation survive changes in champions,leadership,and political administration,whether ministers,Chief Executive Officers,or senior civil service leadership.

The development of these strategies takes time and effort.Hopefully,as depicted in the right circle,the lessons learned from these strategies can be transferred to a process of continuous learning about evaluation.This is where evaluators can play a critical role,for example,by designing evaluation practices through the development of a strategic plan for evaluation,evaluation policies and procedures,and evaluation frameworks and processes.Evaluators must also work with others to ensure that sufficient resources are dedicated to the evaluation.Evaluation policies and procedures that favor the evaluation,which are built into organizational norms,provide consistency and transparency in expectations and reinforce a good accountability practice.The accountability system is critical for ensuring the sustainability of the evaluation framework.

Processes and products must be of good quality.Standards of appropriate methodological and process rigor have been published over the years in the evaluation field [13-16].Different organizational evaluation systems often define their own typologies of evaluation,interpretations of quality standards,methods,and quality review mechanisms depending on their own institutional demand for evaluation.In addition to methodological quality,there is also process quality,such as the value placed on transparency,capacity development,management,and accountability.A quality process also promotes use,both while it is happening and through its reporting.Process quality helps to build the legitimacy of an evaluation system and the ongoing incorporation of the lessons learnt.

The existence of networks and a networked practice is equally important to support a sustainable organizational evaluation system [5].Evaluation systems that build coordinating,cooperating,and collaborative networks are more likely to be sustainable and contribute toward sustainable development.Networks help address complex problems,such as sustainability and climate change [17].Furthermore,networks (including cooperation,coordination,and collaboration) can bring together normally autonomous individuals and organizations to develop partnerships and capabilities for evaluative thinking,setting common goals on evaluation quality and practice,and helping increase the demand for evaluation.A network of practice can further expand the individual and group capabilities for evaluative thinking:critical thinking applied in the context of evaluation,which involves identifying assumptions,posing thoughtful questions,pursuing deeper understanding through reflection and perspective taking,and informing decisions in preparation for action [18].Finally,a network function enables the evolution of the evaluation system to connect to different types of demand for evidence,yielding feedback on what is and what is not working,and in doing so,develop champions [5].

Throughout the whole process of developing ECB strategies,evaluators need to ask the following questions:

1.What types of capacities are needed?

2.Whose capacities need to be developed at the national and local levels?

3.What are the baseline skills in evaluation with a focus on energy policies and programs?

4.Where are these skills most needed?

5.What types of data are routinely collected at the local and national levels currently,so that they can be leveraged with new data?

6.How should policymakers use evaluation results in decision analysis,and are they using the results?

7.How will stakeholders use evaluation?

8.When would be the best time for evaluators to present their findings to stakeholders?

Once the ECB strategies are implemented,they need to be evaluated.One strategy for identifying the success of these strategies is to design and measure a set of indicators for evaluation capacity and evaluation practice.As an example,evaluators need to ask the following questions regarding the evaluation capacity:

1.Is there readily available access to information (methods and findings) on evaluation?

2.Are there collective learning opportunities?

3.Has an evaluation framework been developed?

4.Is there a repository of evaluations that is publicly available?

5.Are there opportunities for training in evaluation?

6.Are there policies and procedures supportive of evaluation?

7.Are there financial resources for supporting evaluation?

8.Are supervisor(s) engaged in and use evaluation?

9.Are supervisor(s) supportive of evaluation?

10.Are staff supportive of evaluation?

11.Are staff knowledgeable about evaluation and do they have the requisite skills for evaluation?

Similarly,evaluators need to ask the following questions regarding the evaluation practice:

1.Is evaluation actually being conducted,and to what extent?

2.Is evaluation (methodology,findings,lessons learned,etc.) being shared?

3.Are individuals and organizations learning from evaluation?

4.How is evaluation being used?

5.What are the motivations for performing evaluation?

6.Who is mainly responsible for evaluation?

7.How frequent is evaluation being conducted?

8.What is the level of embeddedness of evaluation in the organization - is evaluation firmly established as a critical component?

4 Evaluation capacity building in Asia Pacific

As mentioned earlier,leadership is critical for building a sustainable energy evaluation system [5].In responding to this need,a new non-profit organization (Energy Evaluation Asia Pacific (EEAP)) was established in 2018 to take a leadership role in expanding the practice of,and capacity for objective evaluation in the energy efficiency,renewable energy,and energy-related (such as water and transportation) program and policy arena.EEAP has focused its early efforts on ECB strategies (left side of Fig.1):EEAP created a website (www.energy-evaluation.org) containing key evaluation resources (e.g.,conference papers,journal articles and notices regarding upcoming conferences,workshops,webinars,etc.).EEAP also holds conferences,workshops,and webinars that bring evaluation experts,academics,policymakers,program managers,and others together to foster the development of self-sustaining evaluation communities in countries in the Asia Pacific region.The materials from these forums are made publicly available on EEAP’s website.

EEAP realizes that one organization cannot conduct all the work that is needed,particularly for conducting activities that are needed for developing a sustainable evaluation practice (right side of Fig.1).To that end,EEAP has recruited 36 Evaluation Ambassadors from 24 countries in this region for communication and networking purposes to develop an evaluation community.For example,EEAP is already working with the Ambassadors to disseminate information regarding (and from) conferences,workshops,and webinars to the Ambassadors’ networks via social media (such as,Facebook,LinkedIn,Twitter,Instagram,and WeChat).Similarly,EEAP is working with nonenergy evaluation organizations in this region as well as with the following international groups to build a stronger community and network to help conduct and achieve some of the activities mentioned in the ECB model:Asia Clean Energy Forum (organized by the Asia Development Bank),Asia Pacific Economic Cooperation’s Energy Working Group,Asia Pacific Energy Research Centre,Independent Evaluation Group of the World Bank,and the International Energy Agency.In summary,by working with the evaluation communities in Asia Pacific,EEAP hopes to provide information regarding the evaluation and collective learning opportunities,developed evaluation frameworks,and training on impact and process evaluation.

5 Key topics for evaluation in Asia Pacific

As noted earlier,evaluation plays a critical role in informing the design and implementation of energy programs and policies in Asia Pacific.Many topics are expected to be of central interest for evaluators in the coming years; in particular,the evaluation of SDGs and evaluation of gender in energy programs and policies are highlighted below.

5.1 Evaluation of sustainable development goals

Seventeen SDGs were approved by the UN General Assembly in 2015 and underpin the 2030 Agenda for Sustainable Development that officially came into force in January 2016 [6].The 17 SDGs identify 169 targets and 230 indicators [15].An integrated perspective is needed that pays attention to the interlinkages of different levels across sectors and societal actors [15][19].Hence,evaluators must track,document,connect,and support sustainable transformation at all levels of change.

The SDGs are interlinked,and each one is intended to rest on three distinct pillars to varying degrees:economic,social,and environmental sustainability [6][19].The 17 goals are based on five universal and interlinked principles:people,planet,prosperity,peach,and partnership.They define a path to end poverty,ensure prosperity,and protect the planet and its inhabitants.

In this paper,the key goal is SDG7:“ensure access to affordable,reliable,sustainable,and modern energy for all.” As a key constituent of SDG7,energy efficiency and renewable energy contribute directly or indirectly to achieving all the other SDGs,many of which are interconnected across the three dimensions of environmental sustainability,human development,and sustainable growth.

The SDGs are important for many reasons; in particular,they represent a major international development commitment for at least the next 15 years.Thus,it is essential to ensure that the SDG efforts work and that the money is well spent.The evaluation process is country driven and participatory:SDGs are “owned” by the countries (not by the UN and donor agencies).National governments are the key agencies responsible for the implementation of the SDGs in each country.Unfortunately,thus far,there is only an initial awareness of sustainability,urgency and level of challenge,or opportunities,and,as a result,evaluation is currently missing in action on sustainability [20].

The evaluation of SDGs is challenging.More often than not,evaluations are framed such that their scope focuses narrowly on an intervention and its direct outputs and outcomes.In contrast,the SDG “initiative” is huge and complex with multiple actors,objectives,and programs at all levels.Thus,evaluations must focus on the broader context within which the intervention (program,project,etc.) operates and draw the boundaries of the system - rather than evaluating the individual interventions in isolation from their context.In fact,some people do not believe that it is possible to evaluate all the SDGs together due to complexity.As a result,most agencies focus their evaluations on a particular sector (a “silo”),and different types of organizations will need to break down the silos and work together on an integrated evaluation of the SDGs (e.g.,see [21]).

Additionally,there are unique opportunities for SDG evaluation and sustainable cities.This activity will bring together many different types of organizations and talents,in addition to lending a voice to vulnerable groups and grassroot organizations,particularly interested in vulnerability,exclusion,and gender analysis.These groups will ask new questions that are also broader and more long-term (sustainability) than traditional evaluation.As mentioned earlier,evaluators will need to think beyond silos,instead connecting and interrelating interventions,and breaking down silos,examining integration,alignment,and coherence across sectoral specializations and SDGs.Addressing these challenges will provide an opportunity for evaluation to learn,refresh,invigorate,and engage in transforming the evaluation of today to a knowledge and practice that is fit for the deepening challenges that are being faced [20].

Finally,evaluators will need to focus more on process evaluation rather than on impact evaluation,at least initially [22].For example,having some understanding of the process of SDG program development will lead evaluators to measure different things at different stages in this development:first assessing the implementation,and then the short-term,proximal SDG outcomes,and once the program is functioning,assessing the longer-term,distal SDG outcomes [23].

5.2 Evaluation of gender

It is rare for an energy program or policy to benefit all individuals uniformly.Accordingly,it is most important to examine the differential impact of programs and policies,particularly on the vulnerable and those excluded.Thus,when conducting the evaluation of energy programs,two key planning questions relating to equity need to be asked:First,how can interventions be planned to address inequities? Second,are those who are intended to benefit from the intervention the ones who actually benefit? Hence,the evaluation of a specific program will move beyond the focus on outputs (e.g.,how many households were served? How many kilowatt hours were saved or produced?) toward a better understanding of who was served and where the program recipients belonged in the continuum of need? As a result,the evaluation focuses on equity (not effectiveness) and moves from average impacts to the distribution of impacts.Unfortunately,the field of gender and energy is still emerging,and most of the focus has been on energy supply rather than on energy demand.Consequently,there is a tendency for gender to remain invisible,unacknowledged,or marginalized.In fact,energy policies and programs are often regarded as gender neutral,and,as a result,they may inadvertently be discriminatory to women and their needs [24].The evaluation of energy efficiency programs and policies is not exempt from this problem.

Energy efficiency is not gender neutral.Women and men can contribute and benefit in different manners from these interventions.This is particularly evident in three key sectors:household energy management,food processing,and agriculture [25].For example,in household energy management,women bear the brunt of relying on inefficient energy sources and technologies:they often use biomass,charcoal,or agricultural waste for cooking and heating,or kerosene for lighting.This reliance results in health,safety,and empowerment (land ownership) issues.Furthermore,women play a key role in household energy use by making and influencing decisions regarding:(1) the use of appliances,particularly during peak and non-peak times of the day,(2) purchases of goods and services (inefficient products versus energy efficiency products),(3) education and shape of children’s future energy consumption habits (as a role model),and (4) indoor thermal comfort (through the selection of thermostat settings and opening and closing of windows) [25].

Fortunately,many organizations are becoming more interested in the topic of gender and energy:for example,the Organisation for Economic Cooperation and Development,World Bank,Asia Development Bank,International Union for Conservation of Nature’s Global Gender Office,United Nations Environment Program,United Nations Framework Convention on Climate Change,US Agency for International Development,Asia Pacific Economic Cooperation,Global Gender and Climate Alliance,United Nations Women,Women’s Environment and Development Organization,and the ENERGIA Gender and Energy Research Programme,The Hague.

Evaluators can work with these organizations and others to highlight the evaluation of gender in energy programs and policies by ensuring,for example,that the Terms of Reference for a project evaluation team requires gender analysis:the project team must have gender expertise,be gender balanced,and engage with relevant project and other partners working on gender.Evaluators can also partner with organizations with gender expertise for reviewing the evaluation process or reports.In the actual evaluations,the evaluation team can evaluate changes in women’s empowerment,work productivity,income,health,education,and food security.They can also examine the barriers to participation in project activities,as well as unexpected impacts on women.Evaluation frameworks will need to be developed that examine participation in policy and program design and implementation by gender and that also examine impacts of such programs and adoption of technologies by gender.

In summary,gendered aspects of energy and energy efficiency are understudied,and there is a need for more rigorous empirical research,especially to achieve ambitious energy savings and emission reduction goals,as well as sustainable cities.If gender impacts are not evaluated,they are unlikely to be given any attention.

6 Conclusions

A multidisciplinary ECB model is critical for providing the framework to determine the most effective ECB strategies for achieving various individual and group level outcomes,most effective strategies for certain types of participants,and design of different ECB strategies to maximize their impact.Evaluation capacity indicators and evaluation practice indicators will be designed for measuring the effectiveness of ECB in Asia Pacific.As a new organization,EEAP is expected to play a major leading role in these ECB efforts to facilitate the development of an evaluation community in the Asia Pacific.While initially focusing on ECB strategies,EEAP plans to work with its Evaluation Ambassadors and key organizations in the Asia Pacific to develop and disseminate sustainable evaluation practices.Thus,by evaluating what works and what does not work,EEAP and this evaluation community will play an important and integral role in the development of sustainable energy programs,policies,and systems.

It is important to note that there are numerous uncertainties and very little data on the actual implementation of an ECB in Asia Pacific.For example,will EEAP itself be sustainable so that it can continue to play a leadership role in developing a sustainable evaluation community in this region? Will sufficient resources be allocated to the evaluation of energy programs - not only by international finance organizations but also by countrybased evaluation organizations? With respect to the evaluation of SDGs and gender,will the silos be broken so that integration,alignment,and coherence can occur across sectoral specializations? EEAP plans to work with its partners in helping to answer these questions.In particular,EEAP plans to work with APEC in conducting evaluation workshops to train staff in APEC member economies,so that evaluation practices will be embedded in APEC-funded projects.

Further research is needed to address these limitations in what can be done.More research,using mixed methods,is needed to assess what is actually being done on each of the different components in the ECM model in Asia Pacific.Because the region is large,and many of the countries within the region are large,future research must use carefully selected case studies in differing geographical regions to collect the contextual information that would inform quantitative surveys and statistical/econometric analyses for quantifying the impacts of energy programs and policies [26].Hopefully,EEAP will be able to work with leaders in each country for developing a strategic plan for evaluation,evaluation policies and procedures,evaluation frameworks and processes,and resources dedicated to evaluation.

Acknowledgements

I would like to thank EEAP’s Thought Leaders (Organizing Committee) and Evaluation Ambassadors,as well as the individuals and organizations that have helped support the development of an energy evaluation community in the Asia Pacific.Additionally,I would also like to thank the reviewers of the earlier versions of this paper for their help in making this paper more useful and relevant.