There is a great deal of evidence of “what works” in global health. Translating this evidence into results, however, is a challenge and requires acute attention to its delivery or implementation.
Effectively implementing evidence is an active process and requires a continuous cycle of learning. It is a science that involves the systematic identification and analysis of the factors that affect implementation, and the series of actions that are taken to address them. Knowledge dissemination and sharing is also a part of this science— the science of delivery.
The Health Results Innovation Trust Fund (HRITF), which is funded by the Governments of Norway and the United Kingdom, supports the World Bank’s results-based financing (RBF) for health programs. With their inherent focus on results, strategic use of data, and emphasis on knowledge sharing, the RBF programs in the HRITF portfolio exemplify the science of delivery in action.
Strategic Use of Data
The strategic use of data is central to the science of delivery, and the performance of RBF programs is systematically monitored and evaluated using a combination of quantitative and qualitative data. This data is then used to make informed decisions about program design and implementation, or as a springboard for further inquiry.
Quantitative, operational data is collected for many RBF programs using “performance dashboards.” Operational data highlights trends in program performance on quality scores and indicator achievement. This data is collected on a monthly and quarterly basis, which allows for a “real-time” analysis of program effectiveness, and is analyzed at the facility, regional, and national level. To ensure validity, operational data is independently verified.
The RBF program in Nigeria was piloted in primary health centers and hospitals in three states, Adamawa, Nasarawa, and Ondo. Operational data collected during the program’s first year showed general improvement in quality scores in the health centers in Adamawa and Ondo implementing performance-based financing (PBF)—the RBF approach used in Nigeria. The story was different in Nasarawa: operational data showed marked improvement and then a drastic decline. Hospitals in Nasarawa and Ondo improved their average quality scores, while those in Adamawa improved and then declined. By monitoring operational data, the team has been able to hone in on underperforming states and health facilities and to address specific implementation challenges.
Qualitative methods are often used in program monitoring. The data that these methods produce explains how and why different behavioral, organizational, and other contextual factors affect RBF program implementation and performance.
In the Gambia, focus groups and in-depth interviews with key stakeholders helped the team to identify the barriers to the delivery and the utilization of health services, and the findings that emerged from this qualitative research have informed the design of the country’s RBF program. The Gambia program design includes both supply- and demand-side interventions because the findings showed that there were barriers at both the health system and community level, such as insufficient funding and cultural obstacles, respectively.
The team in Zimbabwe implemented a mixed-methods process evaluation to identify different factors that contributed to variation in provider performance. The results of key-informant interviews and group discussions with program stakeholders showed—among other findings— that remote facilities earned lower incentives and that rigid procurement rules delayed local purchases, which affected performance. The team is now revising the criteria that characterizes a facility as “remote,” and is reviewing the performance-payment calculation formula to include a higher “remoteness bonus.” Procurement rules are also being streamlined to prevent delays and improve performance.
These innovations on program design and implementation create what World Bank President, Jim Kim, calls “virtuous cycles” of learning. By using data to track program performance, RBF teams can continuously learn from implementation and make improvements until the desired results and outcomes are delivered.
Impact Evaluations Contribute to Global Evidence Base on RBF
Impact evaluations (IE) form the core of RBF’s learning agenda. The results of these IEs contribute to the global evidence base on RBF’s effectiveness as an approach to improve health outcomes and to strengthen health systems. They also offer useful insights on how practitioners can most effectively design and implement RBF programs.
To date, four impact evaluations of RBF programs have produced final results. Those in Argentina and Zimbabwe, and Zambia have shown positive results—that RBF programs help get high-impact, quality health care to poor women and children around the world. The IE results from the RBF program in the Democratic Republic of the Congo (DRC), however, were not as encouraging. The design of DRC’s program was expected to yield increases in service utilization and coverage, but the results showed that there were not significant increases in either.
Though unexpected, the results of the IE in the DRC have provided valuable lessons on the importance of RBF design and implementation. One lesson centered on the design of the program’s payment mechanism and its effect on financial planning at the facility level. The mechanism led to payment volatility, and the subsequent ambiguity about expected revenues may have caused financial-planning challenges at the facilities. Another lesson was learned about management autonomy. The results showed that the flexibility provided to health facility managers in distributing payments among staff led to a more egalitarian allocation. These lessons and others have informed the redesign of the DRC’s RBF program, which has been extended under a new operation.
Knowledge Sharing and Dissemination
RBF knowledge and learning is actively shared and disseminated, which is an important component of the science of delivery. The RBF website and related social media channels are particularly effective for engaging on results-based financing with a diverse, global audience. By sharing foundational information and experiential knowledge through these channels, knowledge tools, such as the RBF E-Learning Course and the PBF Toolkit, have helped RBF practitioners to be more effective in their work.
A variety of learning events and forums further build RBF awareness, capacity, and knowledge. Learning events, like the Annual Results and Impact Evaluation Workshop, provide the opportunity for RBF stakeholders to share results and knowledge, discuss implementation experiences, and learn from peers and technical experts.
Science of Delivery in Action
The RBF programs in the HRITF portfolio are at the forefront of putting the science of delivery into action. Their intrinsic focus on results facilitates the collection and strategic use of data in making informed decisions on implementation. Support for these programs’ knowledge-sharing and dissemination initiatives also helps to ensure that RBF evidence—the “what works”—is effectively delivered.