Tag Archive for: Monitoring & Evaluation and Impact Assessments

EWSETA: Post-Implementation Review of the War on Leaks project

Mthente Research and Consulting Services was appointed by the Energy and Water Sector Education and Training Authority (EWSETA) to conduct a post-implementation review of the War on Leaks (WoL) project. Launched in 2015 as a presidential initiative in response to South Africa’s pressing water conservation challenges, the WoL project aimed to address the inefficient use of water and reduce losses caused by physical leakages and commercial inefficiencies. The review provided EWSETA with critical insights into how well the project performed against its original objectives and offered valuable lessons to inform the planning and execution of future initiatives of a similar scale.

To conduct the review, Mthente adopted a mixed methods approach grounded in the OECD/DAC evaluation criteria—relevance, effectiveness, efficiency, impact, and sustainability. The study included a comprehensive document review, a quantitative survey with 200 project beneficiaries, and qualitative interviews with key stakeholders, including representatives from the implementing agent (Rand Water), project managers, and training providers.

The triangulation of data from multiple sources enabled Mthente to assess the successes and challenges experienced during implementation, as well as the broader impact and sustainability of the project. The findings offer EWSETA a well-rounded understanding of the WoL project’s achievements and limitations, serving as a foundation for improved project design, implementation, and evaluation in the future.

ECGB: Organisational Maturity Assessment

The Eastern Cape Gambling Board (ECGB) is responsible for regulating all gambling activities in the province per the Eastern Cape Gambling Act (Act 5 of 1997, as amended) and the National Gambling Act (Act 7 of 2004, as amended). This includes issuing and revoking gambling licenses, protecting the public from illegal gambling, and ensuring the exclusion of problem gamblers. The ECGB’s mission is to promote a socially responsible industry through regulation, licensing, and enforcement, ensuring the Eastern Cape gaming industry is honest and competitive, contributing to society, and providing excellent service to stakeholders.

In line with this mandate, the ECGB commissioned Mthente Research and Consulting Services to conduct an annual organisational maturity assessment, with the most recent one completed in the 2023/24 financial year. This assessment was based on the ECGB’s organisational maturity framework and an electronic tool, which evaluates the performance of its governance processes. The 2024/25 Organisational Maturity Assessment reaffirms the organisation’s strong and stable performance by consistently maintaining its maturity rating.

Review of Municipal Annual Financial Statements (AFS) Support Initiative in selected municipalities of KwaZulu-Natal (KZN)

Mthente Research and Consulting Services was appointed by the KZN Treasury to undertake an evaluation of the review of the Municipal Annual Financial Statement (AFS) support initiative. The principal objective of this study was to determine whether the AFS Reviews provided to delegated municipalities are being implemented as specified and to ascertain if there has been an improvement in the quality of AFS submitted and audits opinions expressed on the AFS submitted and audits opinions expressed on the AFS. The initiative, spearheaded by the KZN Provincial Treasury’s Municipal Accounting and Reporting Sub-Programme, aims to enhance financial governance and accountability within the regional municipalities by providing necessary support and monitoring their financial management practices. This support is crucial, particularly in ensuring compliance with the Generally Recognised Accounting Practice (GRAP) and the Municipal Finance Management Act (MFMA). 

Mthente’s methodology included a desktop review, which involved an extensive review of critical documents provided by the KZN Treasury, the development of a Theory of Change (TOC) for the Municipal AFS Review Support Initiative. This TOC was crafted to accurately reflect the initiative’s intended outcomes, impacts, and the essential assumptions underlying its strategic objectives, followed by a workshop with key stakeholders to confirm and refine the proposed TOC. Semi-structured interviews with 16 key municipal officials were conducted, and the key findings were analysed and presented in a report.

ECSA: Developing a Monitoring and Evaluation Framework

The Engineering Council of South Africa (ECSA) is a statutory body established by section 2 of the Engineering Profession Act, Act 46 of 2000 (EPA). ECSA’s primary role is to regulate the engineering profession in terms of the EPA. The primary functions include developing policies, setting standards that align with local and international benchmarks, and adhering to global instruments. This also encompasses the accreditation of engineering programs, registering individuals as professionals in different registration categories, maintaining their registration through Continuing Professional Development (CPD), and regulating the conduct of registered professionals. To fulfil its mandate, ECSA develops and reviews policy and standards, conducts feasibility studies, and carries out research for informed decision-making. 

Mthente Research and Consulting Services was commissioned by ECSA to develop an effective model for monitoring and evaluating its statutory functions. The objective of this project was to ensure that ECSA has robust tools and processes to measure the performance of its statutory functions, allowing for continuous improvement in how it governs the engineering profession, ensuring regulatory compliance, improved performance, and alignment with best practices locally and internationally.

ITC ILO: Strengthening the employment impact of South Africa’s industrial policies

The Department of Trade, Industry, and Competition (dtic) plays a crucial role in providing leadership on South Africa’s trade policy, promoting economic growth and development, industrial upgrading and diversification, and poverty reduction through sustainable employment and job creation. 

One area where improvements to the current context is needed is the regular monitoring of employment creation in key sectors. As part of the master plans, the Government commits to providing a set of fiscal and financial incentives, which may include tax concessions (foregone revenue), grants, or preferential financing. The expectation is that beneficiaries of these measures will invest and create jobs. Monitoring investment and employment trends is vital to ensure compliance with the commitments made about these incentives, or for the adaptation and improvement of the sectoral plans. 

The International Labour Organisation (ILO) aimed to support the dtic in enhancing its monitoring functions within the context of its broader monitoring and sectoral intelligence work. To this end, Mthente Research and Consulting Services was appointed by the International Training Centre Of The International Labour Organisation (ICTILO) to strengthen the employment impact of South Africa’s industrial policies. The primary objective of the report was to provide a baseline appraisal of the dtic’s Monitoring and Evaluation (M&E) system. This included assessing the current data sources used by the dtic, their compatibility with relevant standards and requirements, and whether the M&E system aligns with best practices. The report examined how various data points and sources are consolidated to form a coherent understanding of employment trends, identifying any existing gaps or bottlenecks in the current system, and assessing the system’s adaptability to emerging industrial trends and policy shifts.

An M&E appraisal was conducted to assess the effectiveness of the dtic’s M&E system in measuring employment outcomes within the automotive and Retail–Clothing, Textile, Footwear, and Leather (R-CTFL) sectors. This assessment was informed by a desktop review of documents. Online engagements were also held with dtic personnel to gain an understanding of the monitoring and evaluation system. The M&E appraisal process evaluated the dtic system based on Otundo’s (2019) twelve primary components for effective and efficient M&E systems and followed the ILO template for evaluability reviews. This included examining internal logic and assumptions, the quality of indicators, baselines, targets, milestones, the feasibility of means of verification/measurement and methods, human and financial resources, and stakeholder engagement.

To improve the M&E system, a technical workshop was held to validate the appraisal of its M&E system. This workshop also facilitated the co-designing of a roadmap for the M&E system in collaboration with dtic staff and key stakeholders from the Department of Employment and Labour and Productivity SA.

NACS Implementation Plan and Monitoring Framework

GIZ

The Transparency, Integrity, and Accountability Programme (TIP) aims to support both state and non-state actors in the implementation of the National Anti-Corruption Strategy (NACS) in a comprehensive manner that involves the entire government and society. TIP focuses on building the capacity of anti-corruption advocates within the government, civil society, and the business sector.

The programme provides technical assistance to the National Anti-Corruption Advisory Council (NACAC) and its Secretariat to help develop an integrated monitoring and evaluation system for the NACS and assess its impact. This support includes revising the NACS implementation framework and finalising the NACS implementation plan along with its corresponding monitoring framework. The M&E framework and process will be used as a mechanism by which information will be collected, captured, collated, stored, and used, to assess the progress achieved by NACS on its programmes. 

Mthente Research and Consulting Services was appointed to assist the NACAC’s Monitoring, Evaluation, Reporting, and Learning (MERL) workstream in revising the NACS implementation framework and finalising the NACS implementation plan and its monitoring framework. This comprehensive strategy delineates strategic objectives aimed at fostering citizen participation, ethical leadership, professionalism, transparent governance, a culture of reporting, protection of vulnerable sectors, and the fortification of anti-corruption agencies. Recognising the need to coordinate anti-corruption efforts, the strategy emphasises the establishment of robust implementation structures and monitoring measures to holistically address corruption at multiple levels.

Since corruption is a complex phenomenon, it presents a formidable challenge in the development of an implementation framework, especially linear models of addressing change. Corruption, driven by multiple interconnected factors, often leads policymakers to address symptoms rather than root causes. 

To assist the NACAC MERL Workstream with the revision of the NACS implementation framework and the finalisation of the NACS implementation plan and its monitoring framework, Mthente’s scope of work was informed by guidance from the technical committee and stakeholders throughout its execution.

Outcome Evaluation of two SA Tourism Implementation Programmes

South African Tourism (SA Tourism) is the official marketing agency responsible for promoting tourism on behalf of the South African government. It is legally constituted under the Tourism Act (No. 3 of 2014), with the mandate to position South Africa as a premier travel and business events destination.  SA Tourism’s mission includes “marketing South Africa both internationally and domestically to increase the volume of tourists and the value they add to the economy”.  To achieve these objectives, SA Tourism implements targeted marketing initiatives designed to promote the country’s diverse tourism offerings, stimulate economic growth, and contribute to job creation. In FY23/24, SA Tourism launched several strategic campaigns, including the Global Brand Campaign (Live Again Campaign) and three localised global tourism campaigns, namely, the Hugh Masekela Campaign; the Easter Gospel Celebrations Campaign; and the Siya Kolisi Campaign. Given the strategic importance of these campaigns, Mthente was commissioned to conduct an outcome evaluation to assess their effectiveness and provide insights to enhance future marketing efforts. The evaluation employed a qualitative approach, utilising semi-structured interviews to gather in-depth insights from key stakeholders involved in SA Tourism’s marketing campaigns. The qualitative approach was chosen to allow for a detailed exploration of participants’ experiences, perceptions, and evaluations of the campaigns. Key findings from the data collection and analysis phase highlighted areas where SA Tourism’s marketing strategies have successfully reinforced South Africa’s global tourism brand, while still drawing out key recommendations for further improvement of the effectiveness in driving direct tourism growth and economic inclusivity.

Evaluation of four development funds in the agricultural sector in South Africa.

The Competition Commission of South Africa (CCSA) is a statutory body established to regulate competition in the country. It is responsible for investigating and evaluating restrictive business practices, abuse of dominant positions, mergers, and conducting market inquiries. CCSA also engages in advocacy to promote equity and efficiency in the South African economy. As a public entity, CCSA operates in compliance with the Public Finance Management Act and the Constitution of South Africa. Currently, CCSA seeks a qualified service provider to conduct a telephonic survey to assess the impact of four development funds it created between 2015 and 2020 in the agricultural sector. Mthente was appointed by the CCSA to conduct a micro-assessment, determining the impact of four development funds created by the Commission between 2015 and 2020 in the agricultural sector in South Africa. Working with a database of over 1000 beneficiary contacts, Mthente carried out a telephonic survey to assess the impact of the funds. In addition to the quantitative findings, Mthente conducted in-depth interviews with selected beneficiaries to develop insightful stories of change, highlighting the impact that the funds had on the lives of these beneficiaries.

NLC Evaluation of Small Grants (2018-2022)

The Legislative mandate of the NLC includes inviting applications from registered organisations to apply for grants (small, medium and large) in support of worthy good causes.  Non-profit entities working for the public good are eligible for funding from the National Lotteries  Commission. Mthente was commissioned by the NLC to undertake an evaluation of the small grants disbursed over the past five years (2018-2022). The study examined the efficiency of the small grant category in enabling first-time and emerging applicants to access funding, the impact of the funding on the objectives of grantees, the sustainability of grantees and the positive or negative unintended consequences for grantees and the NLC. Through comprehensive surveys with NLC grantees during the period in question and in-depth interviews with various NLC implementers, Mthente produced a comprehensive evaluation report that met the intended objectives. Overall, the study provided the NLC with a better understanding of the function and value of small grants in supporting emerging / grassroots organisations, and how this contributed to NLC’s funding objectives.  

Valued Citizens Initiative: Outcome evaluation of the iSIQALO II programme

Mthente was commissioned by the Embassy of France in South Africa to conduct an outcome evaluation of the Valued Citizens Initiative,  iSIQALO II programme ‘Strengthening the Capacity of Social Work Professionals to Help Children and the Youth Manage their Sexuality and Deal with Vulnerability Factors to Prevent Unintended Early Pregnancy in South Africa and Malawi”. The purpose of this evaluation was to assess the relevance and effectiveness of the iSIQALO II Professional Development Programme to empower social service professionals in Malawi and South Africa, to help children and the youth manage their sexuality and deal with vulnerability factors exposing them to unintended early pregnancy. Through a baseline and endline assessment that consisted of in-person fieldwork in South Africa and Malawi,  pre and post-surveys with social service professionals enrolled in the training, interviews with key implementers, teachers, parents and youth themselves were completed to be able to produce a baseline and endline report that met the intended objectives of the study.