The Zenex Foundation contributes to improved education outcomes for historically disadvantaged learners in South Africa. Through the interrogation of its own programming, the Foundation emphasises the importance of high quality evaluations for learning which, Dr Fatima Adam explains, are generally underpinned by high quality commissioning processes.
Why do your projects require independent evaluations?
Zenex recognises that transforming education is a complex endeavour and change is often incremental. It is thus very important to find meaningful opportunities to reflect and learn from previous initiatives on an ongoing basis. For us, it is critical to apply an external evaluation lens to our work, to better understand and then adapt, improve or share it. An external trained eye can be helpful when evaluating interventions, and can help to make findings more credible. Evaluations are particularly important when it comes to strategic and innovative work.
How does Zenex commission evaluations?
First, we ensure that all our projects are clearly described and have explicit theories of change, including specifying key activities, outcomes and impact. Second, we make clear the purpose for commissioning an evaluation. Evaluations can be commissioned for various purposes, such as to improve programme designs and implementation, inform decision-making, establish outcomes and impact and contribute to knowledge. The third is developing clear questions that we want answered and aligned with the purpose. The fourth is knowledge of the costs of the different type of evaluations. For instance, evaluations that test young learners in diverse languages are geographically spread and are more costly in rural areas.
Once we have designed a project and scope of work, we call on the evaluators to pitch. We also ask evaluators to comment on the scope of work and evaluation questions. Zenex then short- lists and selects the preferred provider based on an assessment of their proposals against a set of criteria. Once the evaluator is appointed, Zenex engages them on whether the project is evaluable, whether the questions can be answered and how they propose to answer these questions. Often, engagement between the evaluator and Zenex involves a robust discussion about sampling, design, methods and the trade-offs that need to be made based on cost and feasibility.
How does this process prevent confirmation bias?
The evaluation questions must be carefully set and be broad enough to prevent biases from cropping up. In other words: did the implementation go as planned; where were the challenges and opportunities; where did we achieve some changes in behaviour, and where did we not? The responsibility lies with the evaluator to ensure that the questions allow for critical engagement with the design and implementation of the projects. Donors also need to ensure that evaluations are not just accountability exercises, but are used to build the knowledge around what works and what doesn’t. Ultimately, evaluations should be shared with others, to build a community of practice.
Who should lead the commissioning of an evaluation?
The donor has to lead. In my opinion, however, the donor should assemble a steering committee or consult an expert on the terms of reference. With Zenex, in some instances we design a set of questions and review them with a monitoring and evaluation (M&E) expert, before sending out the call for agencies to pitch. It is also useful to have a selection committee review all shortlisted pitches. This committee could include a project implementer, M&E expert and a subject matter or content expert.
Once an evaluator has been appointed, the donor must closely manage the evaluation by troubleshooting with the evaluator and feeding results and data back to the steering committee.
What key information should be included in the terms of reference and proposals?
Terms of reference are developed by the donor and should include the project description; a very clear description of what you want answered – including questions on the process, outcome and impact; the purpose of the evaluation and what proposals should contain.
Proposals are submitted by the potential evaluators and should demonstrate an understanding of the project; a critique or appraisal of the proposed evaluation questions; demonstration of alignment between the proposed questions and the project design; an explanation of how they will answer the questions, the evaluation design and research methods; what type of sampling they will do and why; CVs and costs.
At what stage of the commissioning process should the budget for an evaluation be decided on?
The evaluation should be budgeted for at the beginning of a project. There are two options available to the commissioner of the evaluation. Option one is to set aside 10% to 15% of the project budget for evaluation. Option two is to estimate a budget based on possible evaluation design and methods that could be used. In both instances robust engagement must be held with an evaluator to agree on possible trade-offs on design, methods and sampling in relation to costs and the evaluation purpose.
DR FATIMA ADAM, Programme director of research and evaluations, Zenex Foundation, www.zenexfoundation.org.za , info@zenexfoundation.org.za
