By Amanda Jitsing
Picture this scene: It is 10 am on a Monday morning, and the room is filled with evaluators, programme managers, implementers, funders, policymakers and beneficiaries. As joint stakeholders vested in the success of their programme, they have gathered to agree on an evaluation framework which will guide the evaluation. However, whilst some of them understand the meeting’s purpose and know how to proceed with the task at hand, others have never even heard of a ‘theory of change,’ let alone previously seen an evaluation framework. The meeting progresses then, with a few voices tending to dominate the discussion as the others scramble to keep up with proceedings.
Does this sound like effective stakeholder participation in evaluation to you?
The short answer is probably not. In this scenario, there is little room for meaningful engagement between different stakeholders. The voices of implementers, funders and policymakers are largely subdued, and none of the beneficiaries have been invited to the meeting. A number of things might have gone wrong with the meeting. Perhaps the meeting’s facilitator was not adept at encouraging discussion. It could also be that stakeholders were not prepared or they were not the right people to attend this meeting. Nevertheless, the outcome is that minimal stakeholder engagement took place during this workshop.
There are two main reasons why stakeholder participation in evaluation is important. Firstly, evaluation theories such as Patton’s Utilisation-Focused Evaluation are built on the notion that if stakeholders are engaged throughout an evaluation, they are more likely to use its results. Secondly, stakeholders often hold valuable information about the programme. They can help evaluators understand the programme better and identify the wide range of issues and challenges experienced by the programme.1 This information influences how the evaluation is designed and questions are formulated.
Stakeholders typically participate in an evaluation by sitting on an evaluation steering committee or attending stakeholder workshops at key points throughout the evaluation. In my experience, the level and quality of stakeholder engagement in these forums tends to vary considerably. An evaluation where the level of stakeholder engagement is weak undermines the credibility of the results, irrespective of how rigorous the evaluation is. If stakeholders are not engaged throughout the process, they are less likely to understand, accept and use the results.
It is not uncommon to find that stakeholders are invited to the steering committee or stakeholder meetings simply to ensure representation. In other words, the primary goal is to get as many stakeholders onto the committee, irrespective of whether they have an interest in the programme or would contribute meaningfully to the evaluation. Not only does this make the evaluation steering committees unwieldy, it undermines the objective of stakeholder participation.
On the other hand, there are also examples of good practice, where stakeholder engagement has been thought through, and efforts made to find the right people for the steering committee. In these cases, the evaluation process and content of the report is enriched through stakeholder participation.
What does it take to get stakeholders to participate meaningfully in evaluations? The first step is finding the right stakeholders, those who have knowledge of, and an interest in the programme.2 From past experience, I have often seen evaluation steering committees made up of senior programme managers. While this group of stakeholders may have strong knowledge of the strategic objectives of the programme, they tend to have limited knowledge of the day to day operations. Hence, it is necessary to bring in a mix of programme managers and implementers to generate balanced input from individuals with an understanding of the strategic intent and those involved in the execution of the programme. These two perspectives can enhance the evaluation design. One way of finding the right stakeholders is to create a stakeholder map at the start of the evaluation to identify the types of stakeholders, how they are connected to the programme and their roles in the evaluation process.
Another important group of voices that should be included is the beneficiaries of the programme themselves. Beneficiaries provide a “reality check” on evaluations and bring in the perspectives of those who the programme is designed to help.3 Unfortunately, they often cannot be included in evaluation steering committees due to practical and time constraints. For example, while teachers might be able to validate or refute the findings on their experiences of coaching programme, it is unlikely they would be able to attend regular evaluation steering committee meetings. Nonetheless, where possible, it is critical that the voices of beneficiaries are included in the evaluation process. This can be done by creating specific opportunities for beneficiaries to participate in the evaluation.
Once the composition of the evaluation steering committee has been balanced, one needs to ensure the invited stakeholders remain engaged throughout the process and contribute both effectively and meaningfully. Although, most evaluation steering committees begin with many members, the numbers tend to dwindle as the evaluation progresses. Stakeholders, under pressure from other work commitments, stop attending or come to meetings sporadically. This can be disruptive for an evaluation, particularly if it delays decisions on key milestones delivered during the evaluation process.
Creating practical commitment mechanisms is one way in which to deal with this challenge and foster the desired meaningful participation. Evaluation steering committees should consider agreeing on a schedule of meetings upfront, requiring stakeholders who cannot attend a meeting to provide written feedback, and asking members who miss a certain number of meetings to leave the steering committee or to nominate another representative for the duration of the evaluation.
In practice, there are budget and time constraints that limit the number of stakeholders that can participate in an evaluation. This is why making an effort to find the right stakeholders is a crucial part of evaluation planning and must be done before the evaluation starts. Ultimately, meaningful stakeholder participation is about finding the right number of diverse stakeholders, who are willing and able to share their knowledge of the programme and give feedback to the evaluators.
- 1.Greene, J.C (1987) Stakeholder Participation in Evaluation Design: Is it worth the effort? Evaluation and Programme Planning. Vol 10. Pp. 379-394.
- 2. Markiewicz, A & Patrick, I (2017) Developing Monitoring and Evaluation Frameworks. SAGE Publications, Kindle Edition.
- 3. Markiewicz, A & Patrick, I (2017) Developing Monitoring and Evaluation Frameworks. SAGE Publications, Kindle Edition.
Source: Lessons and Reflections about Implementing M&E in South Africa: An anthology of articles by Zenex staff, M&E experts and NGOs. www.zenexfoundation.org.za
