Mthente Research and Consulting Services

Monitoring & Evaluation and Impact Assessments

Understanding the success of any project or programme requires strong key performance indicators that inform a systematic and evidence-based method to monitoring and evaluating. Our approach to Impact Evaluations is through Participatory Monitoring and Evaluation, which allows for assessment and learning that builds and empowers beneficiaries. It also allows for accountability and transparency.

Mthente conducts baseline, mid-term and end-term Impact Evaluations. Additionally, we undertake specialised evaluations such as: implementation and diagnostic evaluations. We work with organisations to measure the effectiveness of their programme activities, and determine the significance, impact and sustainability of the changes brought about by these activities.

See a selection of our past and current projects:

FEM Education Foundation NPC (FEMEF): a longitudinal impact programme evaluation of the Columba Leadership Programme

Mthente Research and Consulting Services (Pty) Ltd successfully bid for an evaluation commission by the FEM Education Foundation NPC (FEMEF) for a longitudinal impact programme evaluation of the Columba Leadership Programme.

The purpose of this evaluation is to assess the baseline outcomes in year 1 and provide a progress evaluation of the outcomes against the baseline at the end of year 2 and 3, as well as provide a detailed endline asssessment of the educational impacts against the baseline in year 4. This evaluation will include two components. Firstly, a desktop analysis of all existing evaluations conducted on Columba to date and a review of research conducted on similar interventions. Secondly, a longitudinal assessment of the outcomes and impact of the Columba Leadership program in accordance with its theory of change. A classical pre-post and group-comparison will be used to draw conclusions on progress within the control group and as compared to the treatment group.

It is expected that the customised indices developed during this evaluation to measure school culture and youth resilience will continue to be used past the life of the evaluation. As such, the tools developed will be designed with this consideration in mind. The customised indices will take into account the School Culture Index and Youth Resilient Leadership Index – both developed by Columba – as well as the theory of change, logic framework and other relevant literature from the desktop review.

It is expected that this report will be widely shared. This includes being published on the FEM and Columba websites. The final evaluation report might be used to influence government and other stakeholders.

Tshikululu – Due Diligence – NGO Early Development Expansion Programme

Tshikululu Social Investments commissioned Mthente Research and Consulting Services to conduct a due-diligence evaluation of a Faith-based Non-Government Organisation (NGO) based in Pretoria, and their Pre-school Forum Expansion programme. The programme had capacity to work with 43 early childhood development (ECD) centres and was proposing to expand working with 150 ECD centres in the next two years and 750 ECD centres in the next 6 years.

The objectives of this due-diligence study were to:

Define and analyse the organisation’s proposed operating model
Identify whether the model is worth replicating
Ascertain if the organisation has sufficient resource capacity
Understand the organisations finances and costing (financial due diligence)
Observe its impact on beneficiaries.

In order to make decisions on funding this programme, the due-diligence evaluation conducted research with the NGO, as well as conducting research with the ECD centre beneficiaries of the Pre-school Forum programme. In so doing, the NGO’s structure and overall financial, human resource, and governance capacity was investigated, as well as the structure, content and impact of the Pre-school Forum.

Evoke Leaders for Literacy Pilot: Impact Evaluation

In 2017 and 2018, World Vision South Africa and Pearson South Africa implemented two Evoke Literacy pilots. The first pilot was implemented at the community-level in Tusalushaka, Limpopo with in and out of-school youth. The second pilot was implemented at two secondary schools in Tembisa, Gauteng. Mthente conducted a quasi-experimental impact evaluation of the entire Evoke Literacy Pilot Program. The focus of the evaluation was to assess the impact of the pilot program by determining its effects on participants’: (a) 21c skills, (b) social innovation skills, and (c) knowledge of how children learn to read.

The evaluation included baseline and end line assessments for each pilot. The design was a pre-test / post-test using the Social Innovator’s Framework in a triangulated approach—self, peer, and professional questionnaires—as the basis for evaluating 21c skill development. The Evoke Literacy analytics (collected via the online platform) were further used to track skill development, and a Locus of Control questionnaire was used to evaluate individual beliefs about success in general. Data was also sourced from focus groups and interviews to support and inform the results of the quantitative analysis.

Gauteng Enterprise Propeller: Programme Impact Assessment

Gauteng Enterprise Propeller (GEP) is a Gauteng provincial government agency established under the auspices of the Department of Economic Development to provide support to small, medium and micro-sized enterprises (SMMEs) and co-operatives in Gauteng. To determine the sustainability of this support GEP commissioned Mthente to assess and measure the effectiveness and impact of their support programmes over the period 2012 – 2016.

The focus of the impact assessment was on two key GEP initiatives: the Investment Management Programme and the Business Development Support Programme. The focus of the assessment was to determine the contribution of these programmes to SMME sustainability, the indirect benefits to communities and the level of customer satsifaction.

The methodology informing the assessment included key stakeholder and expert interviews and 442 semi-structured interviews with Gauteng based SMMEs and Cooperatives that had been supported by GEP.

Gauteng Province, Office of the Premier: Evaluation of the Healthcare Improvement Workshop

Mthente has been commissioned by the Gauteng Planning Division (GDP) in the Office of the Premier, and the Department of Health, to undertake a rapid evaluation of their healthcare improvement workshop. The evaluation is to ascertain whether the workshop had an influence in motivating facility managers to implement improvement at facility level, if progress has been made towards achieving the workshop objectives, and to gather feedback on the implementation of workshop lessons.

Following a Theory of Change Workshop with key project stakeholders, 150 telephonic interviews will be held with workshop participants. Four focus group discussions will be held with workshop participants, including Facility Managers, District Managers and Quality Assurance Managers. On site observations will take place at four sites, and 200 members of the public at these sites will be randomly selected to be interviewed.

eThekwini Municipality: Evaluation Review & Assessment of the Sustainable Living Exhibition

Mthente has been commissioned by the eThekwini Municipality to undertake a review and evaluation of the Sustainable Living Exhibition initiative, from its inception in 2010 until 2016. The objective of the evaluation is to determine whether it is worthwhile to continue with the Exhibition.

To this end the focus of the review is to assess whether the goals and objectives have been achieved, to determine whether it will be cost-effective to host future exhibitions, and to provide recommendations on the way forward.

To support this, evaluation data will be collected from a sample of stakeholders and exhibitors from each year (2010-2016). This will be achieved through one-to-one interviews, focus group discussions and structured interviews.

Enhanced Civic Understanding & Engagement Project: End-of-Term External Evaluation

Mthente was commissioned to undertake an end-of-term evaluation of the two-year project entitled: “Enhanced Civic Understanding and Engagement (ECUE) Project,” which commenced in May 2015. The ECUE Project was funded by the European Union (EU) Delegation in South Africa, and was implemented by the Cape Town Refugee Centre (CTRC) in partnership with the Dullah Omar Institute for Constitutional Law, Governance and Human Rights (DOI) at the University of the Western Cape, and the KwaZulu-Natal Christian Council (KZNCC).

The objective of the evaluation was to “assess the progress” of the project according to its relevance, effectiveness, efficiency, outcome, and impact. This was achieved by assessing the project outcomes and impact on the lives of key beneficiaries who were directly involved in project activities in the Western Cape and KwaZulu-Natal, and by consulting the quantitative data in the form of records, attendance registers, and financial information. The target sites for the evaluation were Capricorn, Joe Slovo and Bellville in the Western Cape, and Isipingo and KwaPata in KwaZulu-Natal.

Impact Assessment Study of the Empowerment Codes of Good Practice for the B-BBEE Commission

The Broad-Based Black Economic Empowerment Commission (B-BBEE Commission), an entity established in line with section 13 B (1) of the Amended B-BBEE Act No. 48 of 2013, under the administration of the Department of Trade and Industry (the dti), has commissioned Mthente to conduct an impact assessment study of the empowerment Codes of Good Practice for the B-BBEE Commission. This study falls broadly within the objectives of the 2013 Amended Act and will inform whether the Codes have reduced or eliminated the fronting practices and fraudulent B-BBEE schemes that prevailed previously and whether government departments and public entities are effectively implementing the Act. Mthente will provide the Commission with key recommendations on initiative advancement including limitation or further modifications.

The South African Fashion Council: The Monitoring and Evaluation of Projects Funded

The South African Fashion Council (SAFC) commissioned a range of baseline, interim and end-term evaluations on funded projects.

The evaluations will:

serve to assess the impact and outcomes of the eight projects on the development of the fashion value chain in South Africa;
serve to improve the quality of proposals submitted according to the guidelines developed by the SAFC; and
provide data to measure any future project interventions and improve the rate of project success through outlining best practice and lessons learnt through case studies.

This research will be used to inform the SAFC’s strategy and funding to ensure the development of the fashion value chain across the sector and the development of participating designers into sustainable businesses.

The Department of Small Business Development: Evaluation and review of the Craft Customised Sector Programme (Craft CSP)

The Department of Small Business Development (DSBD), in conjunction with the Industrial Development Corporation (IDC) commissioned an external evaluation of the Craft Customized Sector Programme (Craft CSP). The main purpose of the Craft CSP is to define the critical steps and key interventions that would increase the competitiveness of the sector and enable its successful commercialisation. The recommendations of the Craft CSP evaluation will be used to improve the Government’s investment in the sector by looking at where the sector is and should be in order to improve the growth and development of the sector by the role-players themselves. The findings and recommendations will ultimately be used to refine the Craft CSP policy document.

The evaluation and the review will:

provide an overview of the status of the craft sector;
investigate the economic and social impact of the Craft CSP in different locations;
examine the extent of support provided by the Craft CSP programme;
quantify the value of government investment in the craft sector and compare to it that of other similar-sized economies;
conduct a full review and impact study on the implementation of the Craft CSP;
reflect on how the programmes of the Craft CSP impact beneficiary organisations, enterprises and the sector;
indicate the scope and reach of the programmes;
reflect on beneficiaries of the programmes and consequent economic spin-offs;
reflect on the shortcomings of the Craft CSP programme;
indicate the extent to which the five KPAs have been implemented; and
provide a set of recommendations on how the Craft CSP can be maximised and improved for more effective implementation.

See other work for the IDC: http://www.idc.co.za/tenders/T44-07 13_appendix_1_mthente_market_study_report.pdf

Networking HIV/AIDS Community of South Africa: Baseline Capacity Assessment:

The Networking HIV/AIDS Community of South Africa (NACOSA) has commissioned a baseline capacity assessment on the organisations implementing its Orphans and Vulnerable Children Community System Strengthening Project (OVC CSS). The programme was developed in consultation with national and provincial government in six provinces, and seeks to strengthen and expand community support systems for orphans and vulnerable children and their families, using NACOSA’s tested Community Systems Strengthening model. This assessment will serve to inform the current levels of capacity of new partner organisations, and strengthen the knowledge base around the current status of Community Based Organisation (CBO) capacity in South Africa.

The objectives of this assessment are to:

determine the current status of CBO capacity to deliver efficient, effective, relevant and sustainable OVC interventions to communities in need;
review the organisational systems, technical capacity, leadership/governance, adaptive and networking/linkages capacity of each CBO; and
gather data on current community responses and existing systems for supporting vulnerable children and households.

National Development Agency: Sustainability Fund Evaluations

The National Development Agency (NDA) commissioned a progress evaluation of the Eksteenskuil Agricultural Cooperative Grapevine Project. This is an income generation project previously funded by the NDA. Sustainability funding specifically aims to give a ‘cash injection’ to projects identified by the NDA as potentially sustainable, but in need of a ‘boost’. To address key challenges affecting the sustainability of the grapevine project, Eksteenskuil successfully applied for sustainability funding from the NDA to the value of R248 449 for the purchase of 5 500 vented picking trays.

The research objectives of the evaluation sought to assess the following:

if the sustainability funding agreement between the NDA and the Project has been implemented, and if not, why not;
if the goods/equipment agreed to be delivered to the project have been delivered by the NDA, and if not, what had delayed the delivery of the goods/equipment;
if the sustainability funding by the NDA is adding any measurable value to the project; and
the strengths, weaknesses and challenges that have affected the implementation of the sustainability funding of the project, from the perspective of both the project staff and the NDA staff.

Gauteng Enterprise Propeller: Impact Evaluation of SMME Support Programmes

Mthente has been commissioned by the Gauteng Enterprise Propeller (GEP), to evaluate the impact and measure the effectiveness of its programmes over the last four years. The programmes support the establishment, growth and sustainability of Small, Medium and Micro Enterprises (SMMEs) and Cooperatives in the Gauteng Province.

In addition to measuring effectiveness, the research will identify programme failures and make recommendations for improvement in the rollout of GEP’s programmes for the coming financial years. The evaluation includes a survey amongst 1 500 of the SMMEs on GEP’s database.

The Tshwaranang Legal Advocacy Centre: An Evaluation to Inform the Strategic Re-Positioning of the Organisation

The Tshwaranang Legal Advocacy Centre (TLAC) (commonly referred to as ‘Tshwaranang’) commissioned an evaluation and strategic positioning exercise. This occurred at a critical time for Tshwaranang after a funding shortfall that led to staff retrenchments, the relocation of the office and the near closure of the organisation. The outcomes of the evaluation process informed a repositioning strategy, which Mthente facilitated with staff and Board members. The strategy focused on gaining strategic insight into future opportunities and direction for the organisation.

Gauteng Department of Health: Evaluation of the Outcomes of Ward-Based Door-to-Door AIDS Education Programme in Gauteng Province

The Gauteng Department of Health commissioned an evaluation to determine and assess the outcomes of the Ward-based, Door to Door HIV and AIDS Education Programme. The Ward-based, Door to Door HIV and AIDS Education Programme has been implemented in informal settlements and high risk areas in Gauteng to (i) improve HIV prevention through combination prevention; (ii) address health care through testing, treatment and adherence; and finally, (iii) identify and refer orphans and vulnerable children (OVC) for poverty relief or referral (iv) and poor households. This holistic effort is aimed at mitigating the effect of HIV/AIDS on residents.

The objectives of the evaluation are to: measure the results and outcomes of the programme in the five municipalities where the programme was implemented; measure the effectiveness of the referral systems from the program to NGO and government services; document community responses to the programme; analyse the community responses; and recommend processes, systems and methods to strengthen the ward-based door-to-door HIV education programme in Gauteng. 104 households were successfully surveyed during the pilot study. The full survey will be implemented in mid 2016, where a total of 2,100 households will surveyed.