1. How can evaluations be independent if they are conducted by the Evaluation Office?
The Evaluation Office reports to the UNEP Executive Director, whereas all operational staff in UNEP (managing and overseeing projects) report through their Division Directors to the Deputy Executive Director. In this way, the work of the Evaluation Office is independent of the operational side of UNEP.
2. If the Evaluation Office is independent, why does it use external consultants to undertake evaluations?
External evaluation consultants introduce an additional level of independence and impartiality to the evaluations since they are external to the organization. In addition, successful evaluation of the broad spectrum of activities that UNEP undertakes can only be done by contracting external experts with the specific technical expertise for each of the subject-specific projects. The Evaluation Office also uses external evaluation consultants because the demand for evaluation is far greater than could be met by Evaluation Office professionals alone.
3. How are consultants selected for evaluations?
Selection criteria for evaluation consultants are determined separately for each evaluation. This assessment takes into consideration the thematic area, regional focus, language requirements, strategic importance and complexity of the project to be evaluated. In many cases a consultancy opportunity will then be advertised. Evaluation consultants’ CVs are then screened against these criteria and the strongest candidates are invited to go through a competitive interview process. Evaluation consultants are also carefully vetted for conflict of interest to ensure their lack of previous or anticipated future (in the coming six months) involvement in the project or other work for the UNEP Units and key partners involved in the project.
4. How can I become an evaluation consultant for the Evaluation Office?
The Evaluation Office welcomes Curriculum Vitae for qualified evaluation consultants. Prospective consultants can register on the UN INSPIRA portal and add their details to the Consultants General Roster, selecting the Evaluation category. If your CV/profile meets the selection criteria for a specific evaluation, Evaluation Office staff will contact you for more information. Check out the Evaluation Office linkedin for consultant vacancies.
5. What kind of evaluations does the Evaluation Office do?
UNEP regards evaluation as a key component in effective Results Based Management. The Evaluation Office has responsibility for managing and conducting evaluations of projects and subprogrammes as well as strategic and thematic evaluations and other studies of management interest in the organization. The Evaluation Office carries out project evaluations across all funding sources including projects supported by the Global Environment Fund. Each sub-programme is evaluated once during two Medium Term Strategy cycles. The Medium Term Strategy can be evaluated at three stages during the four year cycle: formative stage, mid-term and end of cycle. At the end of each biennium, the Evaluation Office prepares a biennial Evaluation Synthesis Report, summarizing the performance for the organization through trends and patterns observed from completed evaluations at all levels. The predominant focus in UNEP evaluations is on the achievement of sustainable and up-scalable results at outcome level and long-lasting impact, along with factors that affect such performance.
6. How do you decide what gets evaluated?
According to the Evaluation Policy, all UNEP projects need to allocate funds for evaluation and their performance needs to be assessed when they finish. This can take place either by carrying out a Terminal Evaluation (a process managed by the Evaluation Office) or a Terminal Review (a management-led process managed by the Project/ Task Manager that is subject to 'validation' by the Evaluation Office. The Evaluation Office prepares an annual list of projects expected to reach operational completion.
- Selection of a project for evaluation by the Evaluation Office is more likely where:
-
Projects make a large contribution to the UNEP PoW results framework.
-
Projects contribute to parts of the UNEP PoW that have had less evaluative attention in the recent past.
-
Projects that, together with others, form a coherent portfolio/cluster evaluation.
-
Projects that represent an area of work of strategic importance to UNEP, (either perceived to have contributed to past strategic achievements or anticipated to play a strategic role in the near future).
-
Among Global Environment Facility (GEF) projects, when UNEP acts as both the Implementing and Executing Agency; or projects that coordinate a GEF programme.
-
Projects that are implemented in geographic regions that have previously been less evaluated and/or projects managed by Regional Offices.
-
Projects that fall under the scope of upcoming strategic evaluations (e.g. Subprogramme evaluations).
-
- Selection of a project for evaluation by the Evaluation Office is less likely where:
-
Organisational units and the work of Project/Task Managers have been exposed to frequent evaluation in the past.
-
Projects have been evaluated frequently in previous phases.
-
Projects have output-oriented designs.
-
Projects have low levels of expenditure.
-
A GEF Project is a ‘Child Project’ within a GEF Programme.
-
7. What happens to the evaluations when the evaluation report is finalized?
The Evaluation Office communicates the evaluation findings to the project stakeholders and publishes finalized evaluation reports in UNEP's online document repository (UNEP Wedocs). The evaluation recommendations are converted into a Recommendations Implementation Plan and the Evaluation Office tracks the compliance of implementation of the recommendations every six months for a total period of 12 months, as appropriate. The process involves close interaction with UNEP project/ programme managers. Up-dates on the status of compliance with recommendations, by Division/ Regional Office are regularly sent to the Senior Management Team every six months as an integral part of performance monitoring and levels of compliance with recommendations are summarized in the biennial Evaluation Synthesis Report.
8. What are good lessons and recommendations?
Good lessons and recommendations are integral components of a good evaluation report, since they often serve as the ‘take-home’ messages that continue to have relevance after evaluations are completed. Whereas recommendations provide concrete action points to UNEP and its partners on how to solve a particular problem affecting the project, lessons are more general and describe good practices with a wider application. All lessons and recommendations should be clearly derived from the findings of the evaluation, and this context should be briefly presented when formulating them. Recommendations should be feasible to implement within the timeframe and resources available and clearly indicate who should do what and by when. Since the purpose of a recommendation is to solve a problem, it is okay to propose alternate solutions to the problem when relevant, and to assess the pros and cons of each option. Lessons are also derived from real project experiences but they propose action with a wider applicability by promoting practices that were found to be beneficial or discourage practices that were found to cause problems to successful project implementation.
9. How are evaluation terms of references developed?
The Evaluation Office has developed a template for evaluation terms of references (ToR) that is based on internationally accepted evaluation norms and standards from Organisation for Economic Co-operation and Development/Development Assistance Committee, the United Nations Evaluation Group, and the Global Environment Facility Evaluation Office, among others. The Evaluation Office regularly reviews the ToR against experience from evaluations, new trends in evaluation, feedback from evaluation consultants and changes and needs within the organization. The template is updated so that it remains relevant and ensures that evaluations are of high quality and respond to organizational and donor needs. The final ToR is specifically tailored for each evaluation by Evaluation Office staff, in collaboration with the project team. Whilst for accountability reasons, the standard criteria against which projects are evaluated remain the same, tailoring the ToR enables the evaluation consultant to focus attention on issues particularly important to a given project. This increases the relevance and utility of the evaluation findings.
10. How do you assess the quality of evaluation reports?
The Evaluation Office uses a set of questions to assess the quality of the zero-draft and the final evaluation report as well as the quality of the entire evaluation process. These questions are annexed to the evaluation terms of reference and presented in the final evaluation report. The main purpose of this assessment is to increase the transparency of the evaluation process and to enable the evaluation audience to read the evaluation report against this quality assessment. Recording the report quality at zero-draft and final report stage allows the Evaluation Office to better capture this review and revision process. In addition, the Evaluation Office staff are intimately involved in the evaluation process and may add Evaluation Office ratings as a second column in the ratings table in cases where they find that evaluation ratings are insufficiently grounded in evidence, reflect internal inconsistency, or have not been established in a similar way among different evaluation consultants.
11. Where can I find information on official UN rules for consultants?
UNEP works under the rules and regulations set by the United Nations Secretariat. Information on official rules and regulations for consultants can be found in a document issued by the United Nations Secretariat – Administrative instructions for consultants and individual contractors, ST/AI/2013/4.
12. Who evaluates the Evaluation Office?
The Evaluation Office is part of the United Nations Evaluation Group, which has a Framework for Professional Reviews of the Evaluation Function of United Nations Organizations. In line with this Framework, the Evaluation Office is also periodically peer-reviewed by members of the United Nations Evaluation Group. The last peer-review took place in 2012 and the peer-review report is publicly available.
The UN Office of Internal Oversight Services (OIOS) publishes biennial comparative assessments of UN Secretariat evaluation functions.
The evaluation reports of UNEP projects supported by the Global Environment Facility (GEF) go through a quality assessment by the GEF Evaluation Office who periodically publish statistics of the quality of evaluation reports produced by the different GEF Implementing Agencies.
The Multilateral Organisation Performance Network (MOPAN) regularly conducts assessments of UNEP, which reviews the evaluation function and use of evaluations, among other things, as part of the indicator framework applied.
