As we all know, RBF is being used in many settings to improve health care utilization and subsequently, healthcare outcomes. A range of evaluations have been conducted and are ongoing to assess the programs and learn about their impact. Apart from the clear lack of strong quantitative evidence that includes credible estimates of the counterfactual (i.e. what would have happened without RBF), an additional theme has emerged – it’s not enough to understand whether RBF works and which outcomes it affects in a particular context, but we need more information about how to interpret the results. Why did some indicators move and others not? What was the impact of the context and non-RBF parts of the health system? And conversely, how did RBF change the health system?
In the last few years, many have asserted that identifying the factors that determine success and failure of RBF programs and understanding the “why” and “how” of implementation is key to moving forward: why does RBF work in some settings and not in others, how is implementation being done, are key assumptions being met? The recent Cochrane Review authors state that RBF “depends on the interaction of several variables and contextual factors” and others have encouraged a systems perspective and approach in design and evaluation of RBF programs to determine what works, how, for whom and under what circumstances.
Most evaluations of RBF are based on the premise of a causal chain model that linearly links inputs to outcomes through activities, processes and outputs, and - this just won’t do for answering the “why” and “how” questions. Such a model fails to take into account two important elements: behavioral responses and feedback loops between different levels, notably health workers, health facilities, health systems, communities and the wider political economy. What is really needed in this case, is a conceptual framework that helps visualize how a RBF program works and how it impacts health. It needs to identify key elements for measurement and the links between these key elements. And it needs to include changes - material, behavioral, and incentive changes.
Over the past months, we worked with teams that are implementing RBF programs on the ground to transform what we had (a linear causal framework that formed the basis of impact evaluations) into a conceptual framework that would serve a more holistic function - be the basis for learning about RBF in multiple ways. The theory of change model that underlies a linear causal framework was retained, but we complemented it with some key additional elements:
- expectation theory to take human behavior and incentives into account in an explicit way
- social ecological theory to take into account the crucial feedback loops between facility-based RBF programs and the wider environment within which those facilities operate.
Of course, conceptual frameworks for programs should never be developed in a vacuum – they need to be designed with and by program implementers and evaluators together. So we asked the nearly 100 participants at the 4th Annual Results-Based Financing Impact Evaluation Workshop in Istanbul, Turkey this year help us improve the framework, or come up with their own competing framework. Interestingly, quite a few of the 14 country teams took our proposed framework and adapted it to their own needs.
A well designed framework can be helpful not only in visualizing what happens “in theory”, but also to test whether our assumptions about how the program works are actually correct. In other words, it can help guide learning. Once we know where the “triggers” and “transmission mechanisms” are likely to be, we can decide what needs to be measured and how to go about it. And different elements of the conceptual framework will need to be measured with different approaches, including impact evaluation, monitoring, process evaluation and documentation. In addition, some elements may be more amenable to quantitative analysis, while others need to be investigated with qualitative analysis or through mixed methods.
At the Istanbul workshop, we also asked teams to identify how they would use the conceptual framework to learn about their RBF programs – and most teams came up with a pretty comprehensive timeline of their learning activities, combining impact evaluation with monitoring and process evaluation, and documentation, and adding qualitative research to quantitative analysis. Countries as diverse as Nigeria, Zimbabwe, Kyrgyzstan, Cameroon and CAR are moving ahead with the plans they developed – hopefully by the next workshop in 2013, we will be able to share experiences and make tools and instruments available for wider use.
Incorporating this “big-picture” approach to building evidence on RBF poses definite challenges, especially for those involved in implementing and evaluating RBF programs on the ground. Therefore, it is critical that the experience that is gathered and tools that are developed be shared widely, so that learning about RBF is a community undertaking, rather than the onus being on individual research teams.
1 Eichler et al., 2009, Eldridge and Palmer 2009, Ireland et al., 2011
2 De Savigny et al., 2009; Fretheim et al., 2012
Recent comments
5 weeks 6 days ago
7 weeks 6 days ago
19 weeks 2 days ago
21 weeks 4 days ago
22 weeks 3 days ago
22 weeks 4 days ago