Trialogue’s retrospective evaluation of the MTN Awards for Social Change sheds light on the importance of advancing monitoring and evaluation (M&E) practices within non-profit organisations (NPOs). As NPOs strive for greater social impact and transparency, the review considers how the awards fostered better M&E frameworks in participating South African organisations.
Sound M&E practices are the tools that enable NPOs to effectively deliver on their objectives. They help organisations to systematically track their progress, identify areas for improvement and demonstrate the outcomes of their programmes to stakeholders, funders and beneficiaries. Good data can help NPOs to make better decisions allocate their limited resources more effectively and reinforce the organisations efficiency and sustainability.
However, limited resources and capacity constraints can challenge NPO’s ability to effectively deliver M&E. The MTN Awards for Social Change, launched in 2019 by the MTN Foundation and Trialogue, set out to encourage and reward good M&E practice in the non-profit sector.
“We believe that effective monitoring and evaluation typically make for a more efficient, transparent and targeted programme, which enables NPOs to use resources more strategically and deliver greater impact,” said Arthur Mukhuvha, General Manager of the MTN Foundation.
“We were delighted with the quality and number of the entries over the past four years which reflected the non-profit sector’s commitment to measuring their impact in our society. The competition served as encouragement for organisations to reconsider their M&E practices and explore how to enhance their approaches over time. We take this opportunity to thank Trialogue for the successful running of this programme; collaboratively we have positively impacted the NPO community” added Mukhuvha.
About the MTN Awards for Social Change
The competition, which ran for four years, offered a total of R1 million in prize money to winning NPOs in three categories based on their size – small, medium and large. Each year a fourth bonus prize winner was awarded.
Small NPOs were classified as having an annual income of less than R5 million, medium NPOs had an annual income greater than R5 million but less than R15 million, and large NPOs had an annual income greater than R15 million. The winner of each category was awarded R300 000 and the winner of the bonus category, for an NPO that provided the best evidence of advanced M&E practices received R100 000.
The competition attracted enthusiastic participation from NPOs with 70 entering in 2019 and 141 in 2020. The complications caused by the Covid-19 pandemic saw these figures drop to 50 in 2021 and 53 in 2022.
Entrants were scored according to their general M&E practices, project design, project outputs and project outcomes. A bonus category was scored according to the consideration of ethical practice when undertaking monitoring and evaluation activities, the theory of change, the unintended outcomes identified, the understanding of longer-term systemic changes, the exploration and establishment of causality and the consideration of all findings that can impact on success and how these were addressed.
The MTN Foundation concluded the funding of the competition at the end of 2022. Trialogue then conducted a review to assess the impact of the awards on the practice of M&E within participating organisations. A case study was also documented to reflect the extent to which the competition prompted awareness, learning and application of effective M&E principles for an NPO.
Research methodology behind the review
Trialogue’s review of the awards combined quantitative analysis and qualitative research methods to gather data. This provided a detailed view of the overall impact the MTN Awards had on the M&E practices of participating non-profits.
The judges’ scores for the top five shortlisted organisations that entered the competition each year were analysed, together with a comparison of the number of organisations that entered through the years, and the number of repeat applications.
A survey assessed the perspectives and experiences of competition entrants over the years. This was intended to investigate the potential improvements and shifts around the awareness of M&E, knowledge of M&E, M&E practices and processes, as well as any other potential effects that participation in the competition may have given rise to.
Trialogue also conducted in-depth interviews with three of the five judges who adjudicated the competition.
MTN Awards review findings
Trialogue’s review of the project identified the following trends in scores;
- Improved M&E practice scores. Entrants generally demonstrated an improvement in their M&E practice scores over the years. Small and large organisations consistently scored higher over the progression of the competition. Medium-sized organisations also showed a progressive increase in their average scores, with the exception of a slight dip in scores between 2020 and 2021. Large organisations’ average scores tended to be consistently higher than those of smaller organisations. This can most likely be attributed to larger organisations being required to undertake M&E as a funding requirement, having the necessary capacity to do so and therefore more likely to have better M&E systems in place.
- Improved project design. Small and large organisations demonstrated a consistent improvement in project design over the four-year period. However, medium-size organisation showed a slight decrease in their average project design scores between 2019 and 2020, before steadily increasing through to 2022. Project design among large organisations scored consistently higher than smaller organisations, indicating better M&E systems designs.
- Limited project output improvement. The average scores for project outputs across small, medium and large organisations showed a stark increase between 2019 and 2020 and then a dramatic decrease from 2020 to 2021. This most likely reflects the impact of Covid-19 on the sectors’ output ability in 2021.
- Small organisations delivered consistent improvement in project outcomes. Small organisations were the only ones to show a consistent increase in their average score across the years. Medium and large-sized organisations showed an increase in their average scores on project outcomes between 2019 and 2020 but a continual decrease from 2020 to 2021 and 2021 to 2022. This trend may also be explained as the effect of Covid-19.
Meanwhile, the qualitative evaluation of the effects of the awards on organisations’ M&E practices revealed the following:
- Improved internal knowledge of M&E. More than half of respondents (57%) indicated that the awards influenced their internal knowledge of M&E, either partially (39%) or substantially (17%). A third of respondents believed that competition participation did not influence their organisation’s internal knowledge of M&E.This improvement was more apparent among small organisations than medium or large ones.
- Improved M&E knowledge influenced the organisation. The majority (87%) of respondents who acknowledged a partial or substantial improvement in M&E knowledge through the awards project indicated that this improvement was also applied more broadly in their organisations. They reported improved overall M&E processes (85%), the incorporation of M&E into their project designs (32%) and the inclusion of this knowledge in reporting to stakeholders (29%). Respondents also noted improved awareness when designing and reporting on projects and capacitation of M&E staff on M&E processes
Additional outcomes and benefits of the MTN Awards
The review also considered additional outcomes of project participation, besides its direct effect on M&E practice. Respondents reported:
- Increased awareness around M&E processes and implementation.
- Implementation of new processes within organisations.
- Generation of additional publicity.
- Generation of additional funding.
- Other outcomes such as improved M&E processes, inclusion of M&E in budgets, improved reporting systems, strengthened credibility with funders, and strengthened funding applications.
Prize money used to reinforce M&E
The winners who responded to the survey reported that they had used the prize money to strengthen their M&E systems and processes. They used the money to;
- Expand unstructured supplementary service data (USSD) developments.
- Scale up projects used for entry into the competition.
- Increased sustainability funds, or the organisation’s financial reserves.
- Strengthened general M&E programmes within organisations.
MTN Awards bolstered NPO M&E efforts
The MTN Awards for Social Change provided a valuable opportunity for participating organisations to reassess their M&E efforts and fostered a drive for continued improvement in this practice.
One survey respondent noted, “It is important to measure impact accurately and good to see donors care about this critical aspect of community development and NPO credibility.”
The application process helped participants to evaluate their M&E frameworks, improved their awareness and understanding of the significance of M&E and the importance of integrating it into organisational strategy and project design. Competition winners often reinvested the prizes directly back into M&E, demonstrating the recognition of overall value of sound M&E practices for oraganisations.
The competition reinforced the MTN Foundation’s dedication to development, garnering positive media attention and enhancing its credibility in the sector. Participant feedback underscored the value of MTN’s support for M&E, affirming its role in advancing developmental efforts within the South African non-profit sector.
Read the case study of Neighbourhood Old Age Home’s (NOAH) participation MTN Awards project, which documents their dramatic progress in M&E practice over the years of their involvement in the competition. This case study was produced as part of the Trialogue Review of the MTN Awards for Social Change 2019 – 2022
