Source: UNEP Project Formulation, Approval, Monitoring, and Evaluation Manual, December 2000,PCMU pp. 13-89 - 13-99
Introduction
Indicators represent a major tool in a result-oriented environmental programme management. Indicators are signs that show changes in certain conditions resulting from the intervention of an environmental programme/project. They provide evidence of the progress of programme/project activities in the attainment of environmental objectives.
Indicators are dynamic. The UNEP evaluation process continuously develops various indicators associated with the evaluation criteria of appropriateness and relevance, effectiveness and efficiency, sustainability and impact. Some of the indicators, especially those related to effectiveness and efficiency, are quantified and specified in the course of programme/project design. Those indicators related to anticipated changes in identified environmental issues or policies are incorporated, in broader terms, in the project documents, as well. But all programmes and projects must develop performance indicators which can be used by the evaluation to measure outcomes of programmes and projects.
UNEP does not only realize the need for precise indicators for evaluation purposes but also recognizes the difficulties involved in choosing and setting the measurement of indicators associated with environmental programmes/projects in general. Therefore, UNEP employs both quantitative and qualitative indicators depending on the nature of the programmes/projects under evaluation and the type of evaluation criterion.
In this guideline, UNEP presents general indicators relevant to the evaluation of environmental programme/project. They are not by all means exhaustive. They are meant to serve as general guide and framework in the evaluation process. Indicators specific to programmes/projects incorporated in the programme/project documents serve as the minimum tools for evaluation.
Evaluators are expected to use these indicator guidelines to complement the specific evaluation Terms of Reference prepared by the substantive offices in cooperation with the Evaluation and Oversight Unit.
Appropriateness and Relevance
Appropriateness and relevance is one of the key criteria used by UNEP to evaluate programmes, projects, and policies. At the higher level, appropriateness and relevance assess whether programmes/projects support the mandate and mission of UNEP. Evaluation further determines whether the objectives, approaches, strategies and priorities of sub-programmes and projects hierarchically contribute to the respective programmes and to the mission and mandate of UNEP. The presence of hierarchical linkages ensures the relevance and appropriateness of programmes and projects to UNEP's mandate and mission.
Evaluation assesses the following aspects to determine the appropriateness and relevance of programmes and projects:
Whether the project provides a budget, action plan, work programme and time schedule coherent with the project's main objectives, scope, target beneficiaries and expected outputs;
Whether the programme/project objectives integrate into and/or complement the wider UNEP mandate, mission and strategy;
Whether the programme/project conforms with the priorities and focus approved by the Governing Council in the Programme of Work;
Whether the target beneficiaries are in conformity with the Programme of Work approved by the pertinent Governing Council decisions and the requirements of the regional, sub-regional and national beneficiaries;
Whether the programme/project is relevant to specific regional or national action objectives;
the types of linkages the programme/project created and participated in, related activities and environmental problems and issues addressed by related conventions, protocols and agreements;
Whether the approach and strategy adopted is appropriate in realizing the objectives of the project or sub-programme. (Examples of approaches: provision of opportunity for participating countries to set their own priorities and activity work plans; the use of local knowledge and skills in carrying out the project activities; ensuring ownership and capacity building by allowing participation of diverse stakeholders; development of sub-regional and national structures for reviewing, formulating, harmonizing and implementing environmental policies);
Whether the mission and approach adopted by the programme/project is appropriate in light of changing contexts, such as: experience accumulated institutional or developments of emerging environmental issues or shifting priorities. The result of such assessment should lead to the formulation of a new mission that could better utilize the experiences accumulated for better results;
Whether or not assumptions made during the programme/project design were realistic. Such assumptions as expected extent of material support and level of commitment of participating or partnership organizations, often impact on programme/project performance (timely completion and quality of outputs). Assumptions, including, relevant minimum knowledge, capacity of local institutions, institutional infrastructure and government procedures, are necessary elements that impact on timely launch and implementation of projects;
Whether the tools used to estimate the assumptions at the design stage provided a firm basis for necessary adjustments within the programme/project whenever it is required.
Source of Information:
Mandate and Mission of UNEP, relevant UNEP Governing Council Decisions, Programmes of Work, Programme/Project Documents, relevant environmental conventions and protocols, Regional Arrangements on the identified environmental issue, national strategic plan on the environment.
Effectiveness and Efficiency
Effectiveness is a parameter UNEP uses to measure the extent to which a sub-programme or project is successful in achieving intended results and the impact of the results. It measures how effectively sub-programme activities respond to identified environmental problems and what contribution these make to a better managed environment.
Effectiveness is determined by assessing the management and financial systems, which affected sub-programme and project delivery, comprising: operational coordination and monitoring mechanisms, administrative procedures, financial system and institutional arrangements with national executing and cooperating agencies UNEP employed to support activities and projects. The degree and level of collaboration UNEP mobilized for environmental action is also an important measure of effectiveness.
Efficiency indicators address the issue of whether the resources allocated for an activity/project were effectively utilized to achieve planned outputs/results.
Relevant indicators may include:
Extent to which expected outputs and results were achieved as per planned budget and time frame
Evaluation determines whether the outputs envisaged in the project document have been produced in time in required standard of quality and within the stipulated time. The outputs and results are usually quantified in the project document and are usually the outcomes of activities, such as: capacity building, awareness raising, policy development and assessment, development of environmental database and dissemination of information. The realization of the short-term objectives of the programme/project can only be ascertained if the planned outputs are delivered in the prescribed time and quality. At this level, the concern is measuring achievement of results. The measurement of the impact of these results on the environment is addressed through impact assessment.
Extent of collaboration with other partners
The effective execution of UNEP's catalytic role depends on how UNEP raises global/regional/national environmental awareness, develop policy response consensus, and influence governments, UN agencies, NGOs, and other international and inter-governmental organizations for environmental action at the global, regional and national levels.
The scope of collaboration could cover advocacy and awareness raising, creation and sustainability of networking, development of projects, co-financing, joint implementation and capacity building.
Extent of the efficacy of structure and management that supported the implementation of the programme/project
This indicator looks into the impact of coordination mechanisms within UNEP, between UNEP and the national executing agency and collaborating agencies and partners through out the programme/project cycle. The management structure at the local/national/or regional levels (as the case may be) directly involved in the implementation of the programme/project is crucial to the delivery of outputs. Such a structure should facilitate reporting and monitoring of progress, timely revision of progress and efficient decision making.
Flexible and effective MOU between the collaborating agencies that take into account the synergies and their respective internal regulations that could have effect on effectiveness of collaboration is crucial.
Evaluation also looks at whether the financial allocation was optimal to carry out the project and the efficiency of financial transfers. In a multi-laterally funded project delays of transfer from one donor can constrain progress of implementation.
Effectiveness of financial utilization
Financial utilization seeks to determine how well the allocated funds were used to execute the programme/project. Important questions to raise include:
Did the programme/project deliver the outputs and results prescribed at the budgeted cost?
Did the programme/project exceed its budget?
Was the registered variance a result of:
the addition of new elements/activities while the programme/project was in progress?
original under/overestimation of what could be achieved.
Cancelled/interrupted/postponed activities due to unforeseen circumstances?
Failure to meet financial obligations by co-financiers (organizations or governments).
Poor management - financial and coordination?
Poor project design?
Source of Information:
Programme/project document, six-monthly project progress reports, self-evaluation fact sheets, structured interview and discussions with programme managers and executing/collaborating agency staff, questionnaires, and examining actual outputs and results.
Sustainability Indicators
Sustainability of programmes/projects refers to a future situation where the continuity of a project activity or use of project output by a beneficiary community, government or organization, is assured independent of the project input. The real value of UNEP's investment is largely determined by the extent to which the project or activity will continue to attain the intended environmental objectives beyond the project life.
Consequently, in UNEP, the concept of sustainability is a core consideration in project design and implementation. UNEP pursues this objective by incorporating an activity or activities that implement sustainability enabling measures/activities in programme/project implementation. Such activities are incorporated in the programme/project documents as a design requirement.
There are two types of sustainability indicators for environmental programmes/projects. The first type examines outcomes retrospectively, long after programme interventions have been completed. These indicators examine whether programmes/projects outputs/objectives have been sustained, and are often used in major impact evaluations. The second type examines aspects of ongoing programmes and activities that can be used to predict future sustainability. These indicators are used for monitoring and evaluation purposes. The sustainability indicators that follow primarily focus on the second type. Although some of these indicators can be applied retrospectively, most assume the purpose is to examine aspects of current programmes and activities that can be used to predict future sustainability.
Hence, while the actual sustainability of programme/project results and outputs can only be conclusively determined by post-project-completion follow-up activities; divisional programme evaluations, in-depth project evaluations, and self-evaluation fact sheets can determine the extent to which identified sustainability enabling measure/activities incorporated in programme/project documents have been implemented.
Moreover, evaluation should also look into capacity indicators whose presence enhances sustainability of objectives and outputs of programmes/projects. Such indicators may include: enabling environment, institutional capacity and financial sustainability. The attainment of outcomes and the impact of UNEP programme/project catalysis which leads governments and industry to change policies and ways of doing business are indications of sustainability.
(i) Progress of Sustainability Measures incorporated in a programme/project
Depending on the environmental activities the programme/project is engaged, evaluation should determine the progress of the implementation of the sustainability enabling measures incorporated in the project document. Such enabling measures can be conceived to be associated with broad environmental activities, including: capacity building; policy development; awareness raising; and environmental assessment, database development and information dissemination. Moreover evaluation can also look into the programme/project contributions in the development of an enabling environment, institutional capacity and financial sustainability. The latter aspects are usually beyond the project remit but there can be cases where programmes/projects can assist in enhancing such sustainability factors
(ii) Enabling Environment
An enabling environment engenders sound policy, ensures coordination mechanism among related ministries in a national government and among governments of a particular region to effectively promote environmental policy consensus and action at the regional and national levels, and guarantees community participation and empowerment.
The elements of an enabling environment relate not only to institutions and systems but also to the awareness and capacity of the community to demand and advocate for certain levels of environmental standards, whose awareness will foster greater involvement in a better managed environment. Relevant indicators may include:
Presence of a policy-making body/system that conducts the necessary analysis of identified environmental issue to generate policy options, and to make recommendations.
The presence of such a system is likely to review complex environmental
issues more carefully, and thereby ensure a higher degree of sustainability
for the programme/project results.
Presence of a national policy that supports or undermines the particular environmental issue-objective of the UNEP project .
Evaluation should examine whether or not such government policy or regional agreement/arrangement: contains elements that undercut the programme/project objectives, contradicts another policy, delineates key elements of the programme without apparent conflict, specifies key implementation needs (resources etc.), contains feedback process (e.g. an evaluation plan) that leads to continuous evolution of policy. It is important to review related policies and assess whether they help a programme and contribute to its long-term sustainability.
The degree of support provided for the programme/project objectives.
The support can be measured as: active opposition, weak support, neutral, strong support (rhetorical or positive) or demonstrate support by action. The foregoing can be attributed to sources: government officials, politicians, and public opinion leaders, political parties, planning bureaucrats, religious organizations, private sector leaders, NGOs, media and others.
The existence of a sector-wide environmental national plan of action or regional environmental programme.
Sector-wide programmes address cross-sectoral relationships critical to
sustainability of environmental programme/project objectives. It also facilitates joint donor planning and programming, which is increasingly becoming more important at lower levels of the environmental programme. As governments exert control over priorities and donors increase their coordination, sustainability of environmental programmes should improve. Sustainability of project outputs and objectives may be assured when UNEP projects are designed and implemented within the framework of already adopted national or regional strategies.
Presence of a legal and regulatory framework or regional agreement relating to the objectives of the UNEP programme/project.
The sustainability of programme/project outputs and objectives can further
be assured if there exists legal or regulatory statutes that support the
continued need of addressing an identified environmental issue.
Presence of policy dialogue and formulation involving NGOs, community leaders, and representatives of the private sector and special interest groups in the development or implementation of identified environmental programme or project at the national and regional levels.
The more NGOs and communities are involved in environmental policy
dialogue, the greater the potential for affected groups to support the
implementation of the programme/project, use and protect the outputs and
results.
Institutional Capacity
In this context, indicators of sustainability refer to the presence of institutional systems that are likely to be used and sustained even when changes occur in institutional staffing. The sustainable institutional capacity indicators are defined by: Planning and Management, Human Resources and Monitoring and Evaluation system.
Planning and Management Indicators
Presence of a Strategic Plan that includes: a) a mission statement for the institution; b) strategies for the near term (e.g. years), including budgets and priorities; c) a vision for the institution for the long term (beyond years); d) a human resource plan defining staffing and training needs; and e) evidence of participation in planning from a broad range of personnel within institution .
Institutions that perform effective strategic planning have a greater chance of understanding and responding to changes in conditions that affect the organization, and are more able to apply effectively available resources to client needs and to generate demand for services. Strategic planning should include inputs from a range of institutional personnel because relevant planning information will be found throughout the institution, and because there should be ownership of the plan throughout the institution . A human resource plan should be included in the strategic plan to ensure that personnel needs (in terms of required skills) are considered in the institution's planning. Strategic planning that includes human resource needs will help the institution avoid the possibility of management failure due to institutional loss of memory when personnel leave.
Presence of a system for preparing yearly operational plans for the institution including operational goals, personnel requirements, and budgets.
A strategic plan provides an institution with a description of the overall direction for its efforts and the strategies to be employed in achieving its goals. An institution also needs a more detailed operational plan, which sets intermediate goals, defines the personnel and other resources, which are required to achieve those goals, and provides a detailed budget. Such operational planning is typically conducted on a yearly basis, which allows for regular assessment of progress towards long-term goals, and for adjustments in approaches and methods. Institutions that perform regular operational planning are more able to adjust to changing conditions, and are more likely to effectively to address the identified environmental issue.
Presence of monitoring and evaluation system
Monitoring and evaluation system is a primary management tool that enables programme/project staff to monitor progress of implementation. The system works by reporting on the provision of inputs, progress of activities, status of outputs and results by comparing to the pertinent plan of work incorporated in the programme/project document. The presence of such a system ensures that, among other benefits, the most basic sustainability enabling activities are monitored and rectified in the course of implementation.
Presence of a system for the regular assessment of staff training needs
The knowledge, skills, and abilities needed by staff are constantly evolving. New institutional functions and activities, new technical requirements for ongoing functions and activities, and staff turnover all affect staff duties. An institution that does not regularly assess staff training needs assessment will identify staff training needs and may cause an institution to reconfigure staff assignments to meet the challenges of the identified environmental problem effectively.
Presence of a system for assessing the effectiveness of staff training in terms of the quality of the training provided and the appropriateness of the training for institutional needs.
Training should transfer knowledge, skills, and abilities to staff to improve performance in tackling a particular environmental problem. In many cases, however, no test of the validity of this assumption exists. Effective staff training includes measures of whether knowledge, skills, and abilities were actually transferred (e.g., using pre- and post-tests); whether the knowledge transferred is relevant to the objectives and the activities of the institution(e.g. through systematic review of training curricula and outcome measures); and whether the new knowledge was actually applied. An institution with a regular system for assessing these measures will gain more from staff training than an institution that does not, and thus will be better able to serve its clients.
(iv) Financial Sustainability Indicators
Financial sustainability refers to having enough reliable funding to sustain the environmental objectives and results envisaged in the concerned environmental programmes/projects. To be sustainable, such funding should be generated from a country's or region's own resources. Local resources need to be available to replace donor funding as current project input is halted, as donor funding declines or is completely withdrawn. The phasing in of local resources and phasing out of donor funding must be foreseen and built into the project/programme design.
Under financial sustainability are Public Sector and Private Sector Financing categories, which refer to activities designed for financing environmental activities in general and the particular environmental issue/problem addressed by the programme/project under evaluation.
Public Budgetary Allocations
Governments usually make budgetary allocations to public services/sectors, including environmental activities, according to set priorities. It can, therefore, be conceived that , the particular environmental issue(s) addressed by the programme/project under evaluation shares resources from such public budgetary allocations. Within this context, evaluation investigates how the particular environmental issue features within the National Environmental Plan of Action, if there is any, by establishing:
The availability of national environmental budgetary allocation both in amount and as a percentage of the national development budget.
The ranking of environmental budget compared to other development sectors.
The priority ranking of environmental issues within the National Environmental Plan of Action or agreed regional priorities.
The annual budgetary allocation for the environmental issue the programme/project under evaluation addresses.
Private Sector Financing
Here, evaluation should investigate whether or not national governments or
regional organizations involved in environmental activities have functional
resource mobilization mechanisms to supplement public financing for environmental activities. Evaluation, therefore, should address:
The availability of resource mobilization mechanism for the environment, outside public budgetary resources;
The presence of committed NGOs with reliable funding sources for the particular environmental issue;
The presence of institutionalized and functional government policy aimed at raising revenues for the identified environment issue through user fees;
The availability of sufficiently aware and committed private sector with demonstrated will to contribute towards financing the identified environmental programme.
Donor Funding Expectations
Developed countries, multi-lateral financial institutions, and regional
development funds do donate or loan funds for various environmental activities. Access to such funds, for particular environmental issues, depends on various factors such as, relevant policies of donors and financing institutions and the policies and priorities of beneficiary countries.
Evaluation should establish the general trend in the availability of such funds for
particular environmental issues and the knowledge of the availability of such funds and hence the attitudes, plans and actions taken by the particular government/national executing agency to secure supplementary funds needed to ensure sustainability of programme/project objectives and results.
Source of Information:
Programme/project document, programme/project progress reports, self-evaluation fact sheets, pertinent government gazettes, charter of national executing agency, national development plans and budgets, Strategic and Operational plans of national executing and collaborating agencies responsible for national environmental management.
Impact Indicators
At the higher level, impact is measured by the extent to which implementation of divisional programmes and projects bring about change in the environment. In most cases, impact on the environment can only realistically be determined in the long term, rather than in the short-term. Determination of environmental impact can best be established through a post-project-completion follow-up activity comparing baseline data. Further, it is difficult to attribute all environmental changes to UNEP activities.
At a lower level impact refers to the extent to which a project catalysed, stimulated and initiated a wide range of actions, internal and external to institutions directly involved in environmental management. At this level, impact is determined by examining how activities influence policies, strategies and other activities of cooperating agencies, governments and partners and on the mobilization of funds for related environmental activities.
(i) On UNEP's Role
The extent to which UNEP programmes/projects:
Offered opportunities for strengthening collaboration among governments and institutions and provide UNEP with a much better regional presence on issues of environmental policy;
Expanded UNEP's interagency activities;
Identified gaps in knowledge that suggest the need for further action research by UNEP;
Strengthened UNEP's role as the authority that sets the environmental agenda, and that promotes coherent implementation, and an advocate for the global environment;
Enhanced the visibility of UNEP;
Facilitated sharing of information with governments, NGOs and the private sector on UNEP's priorities and urgent global environmental issues.
(ii) On Governments and National Institutions
The extent to which:
UNEP projects caused greater awareness about global environmental issues among government policy makers resulting in better coordination of national institutions;
Successful projects and activities were replicated in other regions of the same country, or other countries in the region;
Methodologies and approaches that UNEP programmes and projects introduced have been adopted in policy formulation process and action;
Information contained in project reports and project outputs is used by national institutions other than national focal points for particular environmental issues;
National institutes of higher learning use country studies or various environmental issues conducted by UNEP projects;
Project activities and methodologies were used as a basis for further academic research, such as, doctoral thesis;
Caused the signing of new and in some cases strengthening an existing international convention or protocol on particular environmental problems;
Assisted countries to upgrade their capacities in particular environmental areas addressed by the project;
Major regional action plans developed by UNEP over the years, such as MAP, are being taken as a model by other regions to develop their own regional programmes;
Contributed to the empowerment of national constituencies for environmental advocacy.
(iii) On UN System Agencies and other Organizations
The extent to which UNEP programmes/projects:
Approaches, activities, methodologies and training materials conceptualized and developed by UNEP were replicated by other UN agencies;
Caused the institutionalization of environmental consideration in the project design processes of UN and other agencies;
Increased interest of the private sector in project results and studies;
Increased cooperation of the private sector in regard to the development of regulatory policies for the protection of the environment;
Created interactions/synergies with other projects and organizations e.g. through joint activities.
(iv) On Funding for Environmental Programmes/projects - Donor countries and
Multilateral Funding Organizations
The extent to which UNEP programmes/projects:
Influenced policies of donor governments and multi-lateral funding organizations to support UNEP and related environmental activities;
Provided seed money, which stimulated local and national governments to launch more comprehensive follow-up projects/activities;
Attracted sufficient donors for the support of succeeding phases or similar projects, encouraged by the performance of UNEP programmes/projects;
Facilitated the initiation of resource mobilization activities in terms of voluntary collaborations by the private sector, banks, companies, foundations and individuals towards identified environmental issues;
Resulted in broadening UNEP's donor base and increase in voluntary contributions from selected countries.
Source of Information:
Programme/project progress reports, terminal reports, self-evaluation fact sheets, discussions with programme/project staff, resident UN Agencies directly or indirectly involved in environmental activities, discussion with relevant national or regional or inter-governmental organizations, relevant national and regional NGOs, other stakeholders.
