![Sample our Politics & International Relations journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5947/TOC-Subject-Sample-2021-Politics-International-Relations.jpg"})
Understanding about what constitutes development effectiveness requires elaboration in order for evidence to add greater value. Development in the twenty-first century is often complicated, if not complex, and correspondingly is most likely to be achieved through diagnostic, contextual approaches to experimentation and innovation. Impact planning, assessment, reporting, and learning systems (IPARLS) can successfully contribute to development effectiveness because they integrate key lessons learned about both the more successful generation and utilisation of evidence, and are aligned with the nature of twenty-first century development. Impact evaluations should be embedded in IPARLS in order to be more legitimate and better used. The systematic application of comparison and triangulation is the platinum standard of rigour for impact evaluations. Impact evaluations can be further improved if they are theory-based, investigate descriptive and causal inference, analyse casual mechanisms, and focus on contextual elaboration. Case and comparative case study designs for impact evaluation remain essential and these approaches can be made more rigorous given recent methodological advances. Impact evaluations and IPARLS will and should be judged by their contribution to greater understanding of development effectiveness and ultimately improved development.
Acknowledgements
The ideas in this article have benefitted from intensive discussions with numerous individuals especially David Bonbright, Fred Carden, Sarah Earle, Nancy McPhereson, Zenda Ofir, and Patricia Rogers (see Bonbright et al. 2009b). It draws heavily from (Khagram et al. 2008) and (Thomas 2007). Generous financial support for many of the ideas found in this article was provided by the Rockefeller Foundation, the International Development Research Centre, and the Bill and Melinda Gates Foundation. It is based on a plenary presentation at the Perspectives on Impact Evaluation Conference, Cairo, Egypt, on 30 March 2009 by Sanjeev Khagram.
Notes
1. For three very different views on this convergence, see Rodrik (2008), Ravallion (2008), and Quinn Patton (2008b).
2. See, for example, Chibber (2005) for a focus on a single institution or The World Commission on Dams (2008) for a focus on the sector of large dams in the context of sustainable water and energy resources development.
3. Among the foundational texts in the field of evaluation that argue and elaborate on this are Bamberger et al. (2006) and especially Quinn Patton (2008a).
4. The emerging IPARLS approach has been collaboratively developed by iScale, Keystone Accountability, and Global Action Networks-Net.
5. IPARLS are similar and attempt to practically integrate Ravallion's ‘10 steps to making impact evaluations more relevant’ (Ravallion 2008) and Michael Quinn Patton's 11 lessons learned about evaluation (Quinn Patton 2008b).
6. There is a tendency for practitioners to see the word ‘theory’ and assume that it is not practical. This perceptual barrier is not helpful, as all human action and interpretation is ‘theory-laden’. In addition, as the explicit use of more rigorous social science methodologies for evidence production in evaluation becomes more common, so should the more explicit use of rigorous social science methodologies for theory of change and theory of action. Applied, evaluation, and action-researchers should, however, recognise that there is a range of rigorous approaches to theory generation similar to the range of rigorous social science research methodologies such as formal modelling, grounded theory, systems thinking, and so forth.
7. This is why it is important that larger organisations play the role of formulating, sharing, and further elaborating broader theories of change. Government ministries responsible for a sector and funding agencies of various kinds along with networks working more broadly within a field are well suited to play this role.
8. This is a role that evaluators and evaluation consultants can play more actively and rigorously. Good evaluators and evaluation will not only assist in developing theories of action for their partners and clients but also link them to broader theories of change. These theories of change are often likely found in academic and scholarly literature in addition to more applied sources. If these links are more systematically made, much broader social learning about what works, under what conditions, and why will be generated. Sadly theory-based evaluation in general remains the exception rather than the norm. In a recent survey of development practitioners for example, only about one-quarter responded that theory-based evaluation was among the methods they most used and only about one-third responded that theory-based evaluation was among the methods with which they were most familiar (see Adrien and Jobin 2009).
9. This part of the IPARL systems is very similar to Lant Pritchett's notion of ‘MeE’ (or ‘big M, little e, big E’), in his ‘Political Economy of Aid Evaluation: How to Build Effective and Sustainable Movements’, presented at New York University, February 2009.
10. Most evaluative and other forms of research that generate credible evidence fail to reflect back upon what has been learned about the a priori theory of change or theory of action. This is a great loss in learning and improved practice because it is the theory of change and theory of action that informs future policy and practice within the context of the initiative and especially in other contexts – the credible evidence provides a signal of what approaches and lessons should potentially be adopted and adapted.
11. More and better impact evaluation does not substitute for more and better monitoring and evaluation. Linking and integrating the two in mutually reinforcing ways as proposed in the IPARLS approach is the ideal. The World Bank (2008) recently reported that on the one hand the number of impact evaluations had more than doubled which was positive news but that, on the other hand, the quality of monitoring and evaluation was modest or negligible in two-thirds of projects for which data were available which is disturbing.
12. Keystone Accountability proposes a ‘Feedback Principle for Social Reporting’ – credible public reporting by any organisation includes not only the logic and evidence for the results, but also: (1) what the organisation's primary constituents say about what it says it has achieved, and (2) how the organisation proposes to respond to constituency feedback. See http://www.keystoneaccountability.org.
13. Since the 1960s an entire industry has emerged to drive companies to remain accountable to their customers. The relevance of this customer satisfaction industry to constituency voice is apparent from leading publications in that field. Both as a tool to guide employee performance and as a way to map and mine constituency engagement with the company (commonly referred to as ‘customer loyalty’), customer satisfaction is to consumerism what constituency voice purports to be for development. Constituency voice also stands on the shoulders of the sub-discipline of evaluation known as ‘participatory evaluation’. Participatory evaluation methods are typically jointly planned and managed by constituents themselves (mainly primary constituents, staff, and other local stakeholders such as local officials). They have been used in various cases, and are increasingly seen as a component in larger evaluation frameworks (Bonbright et al. 2009a).
14. This article remains largely within positivist and Bayesian epistemologies and ontologies for research and evaluation, albeit with many openings for more constructivist and interpretivist perspectives.
15. Ethnographic designs are another type that is not addressed because of the focus on methods within largely positivist and Bayesian epistemologies and ontologies in this article. Ethnographic approaches that fit within these philosophies of science and social science are akin to the case study methods discussed in this section.
16. The other options for the ‘methods you mostly used’ question are primarily a list of various different types of experimental, quasi-experimental, and statistical designs. Either the categories of case study and qualitative impact evaluation approach should have been disaggregated or the experimental, quasi-experimental, and statistical designs should have been aggregated. No categories for mixed-method designs were included.
17. Emile Durkheim (1894) also argued that certain factors are internal to the object of analysis and cannot be removed. Furthermore, he argued that the comparative strategy seeks to examine and identify patterns, not observational units of analysis, because cause and effect are continuously linked; variation in one produces variation in the other.
18. Yin (1987) rejoins that single experiments could receive the same critique.
19. This table has been adapted from Brady and Collier (2004).
20. An initial version of these principles was first shared at the International Conference on Perspectives on Impact Evaluation, Cairo, Egypt, 29 March–2 April 2009. Several leading development and evaluation professionals contributed to their drafting, in particular, David Bonbright (Bonbright et al. 2009b), Fred Carden, Sarah Earle, Sanjeev Khagram, Nancy McPhereson, Zenda Olfir, and Patricia Rogers. The modifications included in the version presented in this article are the sole responsibility of the authors.