Evaluation and learning have always been at the core of the Health Results Innovation Trust Fund (HRITF). Established in 2007 to understand whether results-based financing (RBF) can be effective in improving reproductive, maternal, neonatal and child health (RMNCH), the HRITF has supported 28 countries in their efforts to pilot RBF, and it has instituted an evaluation portfolio that includes 24 impact evaluations and eight mixed methods evaluations.
The depth and scale of the HRITF learning portfolio is nearly unprecedented. The portfolio’s highly rigorous evaluations give it internal validity, while the relatively narrow, well-defined intervention set across a large number and variety of different contexts lend it considerable external validity.
As the HRITF operational and evaluation portfolios begin to mature with all country programming and evaluations now initiated, the HRITF has refreshed its learning strategy to reflect the in-country and cross-country learning that the portfolios provide. The aim is to leverage these component evaluations into composite learning.
Mirroring the extensive consultation process to ensure that the design of each RBF operation and each evaluation meets the needs of in-country learning and advances global knowledge, the priorities for the latest iteration of the HRITF learning strategy were established through a series of stakeholder consultations and other inputs from country representatives, practitioners, donors, communities of practice, thought leaders, evaluators, researchers, and the broader literature.
The result of this process is a prioritized strategy that recognizes that, although the evaluation resources of the HRITF have largely been committed, there are opportunities to investigate evaluation questions that demonstrate high demand, relatively low cost, and have readily available data from the HRITF portfolio. The HIRTF will focus on leveraging existing and upcoming data collection efforts to generate further learning within countries, while emphasizing further learning across the portfolio. The learning strategy also seeks to take advantage of the complementarity of the mixed methods approaches found in many of the RBF evaluations. The process identified five topics prioritized for aggregated learning across the impact evaluations:
- Effectiveness of RBF,
- Equity implications of RBF,
- Strength of incentives and their effects,
- Effects of RBF on unincentivized outcomes (sometimes termed “spillover” effects), and
- Effects of RBF on quality of care.
The HRITF will continue to monitor RBF results, and it will leverage that monitoring by learning from implementation. There was also a strong sense that the strategy could include two topics to continue learning from implementation itself: (i) verification and (ii) how RBF interacts with and strengthens health systems.
Importantly, the exercise of curating topics for further learning also identified questions that could not be included in the HRITF learning strategy solely out of consideration for the additional cost involved beyond the committed evaluation resources of the HRITF. These lines of inquiry would provide tremendous insights, and although the HRITF itself cannot take up these topics, the broader health research and evaluation community is encouraged to consider them for further exploration:
- Rigorous analysis of the contextual and design factors that influence RBF effectiveness to understand the conditions that are likely to lead to success,
- System dynamics modelling coupled with agent-based modelling to understand how RBF interacts with the health sector as a system,
- Post-endline follow-up surveys of an impact evaluation to study the long term effects and sustainability of RBF,
- Case studies on how RBF influences some individual components of health systems—e.g., governance, data and health information systems, purchasing mechanisms and financial flows,
- Cost effectiveness analysis using IE results and synthesizing cost effectiveness analyses across contexts to better understand the value for money of RBF, and
- Case studies to explore the interaction and integration of RBF and the private sector.
Notwithstanding these foregone opportunities, as learning activities are completed, the HRITF team will continue to disseminate RBF-related knowledge within and across countries to inform evidence-based practice and decision-making among practitioners, researchers, policymakers, donors and other stakeholders. A variety of dissemination methods will be used as appropriate for different audiences, both within and across countries.
The HRITF will continue to support country learning and country evaluations. Furthermore, it will endeavor to leverage opportunities to aggregate knowledge across its uniquely large portfolio to gain rare generalizable insights. Concurrently, it will continue to disseminate lessons from country evaluations and topical learning.