Organizations engage in impact evaluation to acquire information about the observed changes or ‘impacts’ produced by an intervention or an organizational change. For many organizations, an impact evaluation is used to improve or reorient an intervention or to make better and more informed decisions about whether to continue, discontinue, replicate or scale up an intervention. With donors and taxpayers directing significant amounts of funds to aid in addressing some of the most pressing social issues through organizations’ programmes, understanding how that money is being utilized, the outcomes of those programmes, and their impacts on stakeholders cannot be undermined. 

In facilitating this, monitoring and evaluation of data and information on key indicators must be collected and established in order to ensure validity and reliability. Thus, an impact evaluation requires a well knit monitoring and evaluation (M&E) system; more importantly, a valid impact evaluation requires a reliable monitoring and evaluation (M&E) system.

A monitoring and evaluation (M&E) system is used to assess the success and performance of programmed activities and projects. This generally involves tracking progress with respect to the project’s initial objective and analyzing data captured and measured on an ongoing basis to improve, and assess the programme’s performance. A strong M&E system can provide important information about the quality of programme implementation, which is especially useful for impact evaluation. A strong M&E can also provide:

• Factual, quantitative, and qualitative information about the programme implementation
  Most programme implementations deviate from the original plan due to logistical and political challenges. Understanding how the programme has diverged from or adhered to this plan is key to attributing the observed change in the outcomes to the programme. Frequent data monitoring can provide the information about the programme implementation.

• Salient data about whether targets have been met 
  Evaluations can help assess implementation fidelity, success in targeting planned beneficiaries, validation of key assumptions hypothesized in the theory of change, and providing important information about the field efficacy of the planned implementation. Programme evaluation can also help assess if a programme has progressed enough to do an impact evaluation.  

Below are some key points to keep in mind when developing a reliable M&E system which can be used to carry out a valid impact evaluation: 

• Articulate the theory of change with inputs from all the stakeholders including the evaluation team.

• Establish monitoring and information systems to track inputs, activities, processes, and immediate outputs.

• Develop key indicators and protocols for measuring attributable change. Do an evaluability assessment and discuss why an impact evaluation is important.

• Use robust and regular qualitative data and approaches to inform, validate or understand exceptions to the overall theory of change.

• Understand and analyze sources of bias (specifically, selection bias and programme placement bias).

• Set up explicit or implicit counterfactuals.

• Pay close attention to challenges to external validity: use disaggregated data and attend to important areas like unintended consequences for different target sub-groups, gender impacts, equity and heterogeneity in general.

• Train program staff and M&E professionals in protocol development, data collection, methods, analyses and interpretation.

• Analyze and understand design efficacy, implementation fidelity and causal impact.

• Review, reflect and update the system based on impact evaluation findings.

• Undertake cost and cost-effectiveness studies.

A strong M&E system is a long-term instrument that can aid in improving the implementation of existing programmes as well as informing decisions about new ones. ASSIST Creativelab helps organizations run monitoring and evaluation studies across sectors and thematic areas for impact interventions.

Engage with us today!

Sources:

Puri, J. and Rathinam, F. (n.d). Designing an M&E System for Impact Evaluation: Tips for Program Designers and Evaluators. Green Climate Fund. https://ieu.greenclimate.fund/blog/designing-me-system-impact-evaluation-tips-program-designers-and-evaluators