Resources

Training, motivating and retaining human resources for health (HRH) is crucial for the improvement of health outcomes, especially in low and middle-income countries (LMICs) where human resources availability and management have been recognized as one of the key health system’s barriers to the achievement of the Millennium Development Goals. In recognition of the limitations of current financial incentives and/or remuneration levels, Performance-Based Financing (PBF) mechanisms have been introduced in many LMICs in recent years.

This project took advantage of an existing field experiment funded by the World Bank (WB) that introduced PBF for some randomly selected primary care facilities in Benin in 8 pilot districts since March 2012. While the impact evaluation of the WB project focuses mainly on the effects of the financial incentives on the uptake of health services and health outcomes in the population, the current project complements this evaluation by collecting detailed information of individual health care provider performance and motivations, in order to test the impact and causal pathways through which financial incentives operate.

Results from the study suggest that the PBF intervention in Benin improved some aspects of the performance of health workers: positive impact on quality of care and responsiveness towards patients but has no significant impact on clinical productivity. There was no difference on the volume of activities in facilities that received the conditional bonus compared to those that received additional resources unconditionally. However, there was a positive impact of the bonuses on a variety of measures of quality of care provided by the staff (observed and reported by patients), which provides some evidence of the successful emphasis on quality of the PBF intervention in Benin. The absence of impact on clinical productivity can be explained by the strong focus of the PBF scheme in Benin on the quality of care. The absence of negative impact on the number of patients seen also suggests that HWs were operating below their production constraint.

Comparing facilities that received the PBF bonuses to the facilities in the pure control group, we found relatively small and inconsistent effects of PBF. The PBF facilities were not associated with higher volume of activities, but there was a slight reduction in costs associated with PBF. There was no evidence of a difference in the resources of facilities available to facilities in the PBF group compared to the pure control group. For some of these outcomes, this lack of results might be partly due to the lack of power to detect differences in outcomes measured at the facility level. Besides, the lack of equivalence of the pure control group makes it harder to control for all potential factors that may explain differences (or lack thereof). Further analyses will use matching techniques to reduce differences between PBF and pure control facilities.

Impact Evaluation Toolkit

Tutorial Video

Need More Information on Impact Evaluation Methods?

The Impact Evaluation in Practice guide is a companion piece of the Impact Evaluation Toolkit. It helps policymakers and practitioners understand available impact evaluation methods, and make decisions based on evidence of what works.

Experience from the Field: Country Spotlights (PDFs)

Our Country Spotlights give you real-life experience from the field in designing and implementing impact evaluations. Scroll over the world map and learn from what other countries have been doing.

Afghanistan: Data Access Policy [PDF, 222KB]
Argentina: Using Administrative Data [PDF, 237KB]
Benin: Design & Implementation Challenges [PDF, 285KB]
Cameroon: Sampling Frame & Political Buy-In [PDF, 392KB]
Central African Republic: Integrating Impact Evaluation & Intervention [PDF, 302KB]
Democratic Republic of Congo: Team Building & Post-conflict Settings [PDF, 291KB]
Nigeria: Why do Impact Evaluation? [PDF, 447KB]
Rwanda: 1-Using Power Calculations in Policy Dialogue/2-Team Building & Local Capacity/3-Using Results Dissemination for Policy Dialogue [PDF, 383KB]
Zambia: Data Collection Challenges [PDF, 310KB]

In this module, you will learn why it's important to evaluate Results-Based Financing interventions. You will find out how to establish a theory of change and a results chain, and how these help develop hypotheses that an impact evaluation can test through evaluation questions. You will gather food for thought by browsing through first-generation questions on whether Results-Based Financing works, and second-generation questions on how Results-Based Financing can work better. At the end of this module, you should know how to develop impact evaluation questions relevant to the context you work in. These questions should respond to key aspects of interventions and fill knowledge gaps on your topic of interest.

Module 1: Choosing Evaluation Questions - Guidelines


Toolbox

1.01 Theory of Change Graph [PDF, 46KB]
1.02 Results Chain Template [PDF, 149KB]

In this module, you will find out how to set up an impact evaluation team, as well as define the roles and time commitments of each team member. At the end of this module, you will be ready to start building and contracting a team that can achieve the tasks you need to accomplish.

Module 2: Building the Impact Evaluation Team - Guidelines


Toolbox

2.01 Principal Investigator TOR [PDF, 640KB]

2.02 Evaluation Coordinator TOR [PDF, 584KB]

2.03 Data Analyst TOR [PDF, 579KB]

2.04 Local Researcher TOR [PDF, 477KB]

2.05 Power Calculation Expert TOR [PDF, 510KB]

2.06 Data Quality Expert TOR [PDF, 627KB]

2.07 Qualitative Principal Investigator TOR [PDF, 571KB]

2.08 Qualitative Field Worker TOR [PDF, 681KB]

2.09 Cost-analysis Expert TOR [PDF, 580KB]

2.10 Technical Assistance Firm TOR [PDF, 690KB]

 

In this module, we will teach you how to design your impact evaluation. Steps include:

  1. defining output and outcome indicators of interest that translate research questions into observable variables;
  2. determining an identification strategy, including the "arms" of a study that respond to research questions and guarantee a valid comparison group;
  3. making sure the sample has sufficient power to detect changes across a representative population;
  4. knowing which data will be needed to measure indicators of interest;
  5. how the timeline, team composition and budget must align and adjust simultaneously; and
  6. how to plan for dissemination activities. At the end of this module, you will know how to design an impact evaluation.

Module 3: Designing the Impact Evaluation - Guidelines


The Toolbox

3.01a RBF Indicators 1 [PDF, 823KB]
3.01b RBF Indicators 2 [Excel, 54KB]
3.02 WHO Indicators [Zip, 2.15MB]
3.03 IE Design Paper Template [Word, 30KB]
3.04 IE Budget Template [Excel, 14px]
3.05 Ex-ante Power Calculation [PDF, 392KB]
3.06 Power Calculations Binary var. [PDF, 170KB]
3.07 Power Calculation References [Zip, 2.55MB]

In this module, you will learn essential steps that lay the groundwork for data collection. Steps include:

  1. scheduling data collection,
  2. reaching formal agreements ex-ante with stakeholders on data ownership,
  3. making sure participants in the survey are protected by defining a research protocol and getting the study ethics right,
  4. determining a data entry strategy and what's needed from survey firm(s),
  5. hiring one or more survey firm(s), and, finally,
  6. adapting and preparing data collection instruments and entry tools.

At the end of this module, you will be ready to go into the field.

Module 4: Preparing the Data Collection - Guidelines


The Toolbox

4.01 Gantt Chart [Excel, 14KB]
4.02 MOU on Data Access [Word, 38KB]
4.03 Research Protocol [Word, 67KB]
4.04 Informed Consent [Word, 19KB]
4.05 Health Facility Survey Firm TOR [Word, 34KB]
4.06 Household Survey Firm TOR [Word, 36KB]
4.07 Data Collection Budget [Excel, 18KB]
4.08 Equipment for Biomarker Data [Word, 15KB]
4.09 Health Facility Questionnaires [Zip, 3.64MB]
4.10 Household Questionnaires [Zip, 3.07MB]
4.11 Community Questionnaires [Zip, 97KB]
4.12 Costing Questionnaires [Zip, 265KB]
4.13 Data Entry Program [Zip, 15.8MB]
4.14 Anemia Referral Guidelines [Word, 14KB]
4.15 Anemia Referral Form [Word, 15KB]
4.16 Translate Questionnaires [Zip, 918KB]
4.17 Institutional Review Board TOR [Word, 31KB]
4.18 Accurate Translation Certification [Word, 15KB]

In this module, you will dig into data collection activities and data quality safeguards. You will see how to define the sample and sampling frame in the field and create unique identifiers for geographical areas and study arms. You will understand the benefits of data collection pre-test and proper data entry management plans. You will learn to plan for fieldwork supervision, recruit and train field teams, conduct a pilot test and, finally, how to manage fieldwork and ensure regular reporting on fieldwork activities. At the end of this module, you should know how to provide sufficient training, support, supervision and communication channels to ensure quality fieldwork activities and data.

Module 5: Implementing the Data Collection - Guidelines


The Toolbox

5.01 Interview Duration Sheet [Excel, 11KB]
5.02 Enumerator Evaluation [Excel, 16KB]
5.03 Survey Progress Report I [Word, 22KB]
5.04 Survey Progress Report II [Excel, 18KB]
5.05a Household Field Manual [Word, 2.37MB]
5.05b Household Training Program [Excel, 15KB]
5.05c Household Training [Zip, 5.79MB]
5.06a Health Facility Field Manual [Word, 206KB]
5.06b Health Facility Training [Zip, 12.7MB]
5.07 Training, CAR & Cameroon [Zip, 3.26MB]
5.08 Health Facility Supervisor Checklist [Word, 22KB]
5.09 Health Facility Arrival Checklist [Word, 20KB]
5.10 Health Facility Supervisor Form [Word, 22KB]
5.11a Daily Listing U5 Exit Interviews [Word, 21KB]
5.11b Daily Listing ANC Interviews [Word, 19KB]
5.12 Cash Management Sheet [Excel, 35KB]

In this module, you will find out why properly storing and documenting data matters, how this can be done and who is responsible for it. You will discover the benefits of the World Bank data catalog and its key features. You will also be able to distinguish which types of data should be stored, where to store it and who should access it. At the end of this module, you will know how to ensure that survey respondents are not put at risk, how to keep the outcome of the survey safe with minimal risk of damage or loss, and how to provide easy access for the impact evaluation research team.

Module 6: Storing and Accessing Data - Guidelines


The Toolbox

6.01 Data Catalog Deposit Form [Word, 126KB]
6.02 Nesstar Data Storage Templates [Zip, 28KB]
6.03 Login Data Management Toolkit [MHT, 92KB]
6.04 How to Access Data Catalog [PDF, 194KB]

In this module, you will learn the functions of each impact evaluation report (baseline, midline, endline), get useful hints on how to clean the data, create variables including indicators of interest, and how to validate the design of the impact evaluation at baseline. You will receive guidance on impact analysis to supplement the impact analysis methods presented in Impact Evaluation in Practice (Gertler et al. 2016). Finally, you will learn how to build your analysis in the spirit of efficient dissemination to key stakeholders. At the end of this module, you should be ready to start analyzing the data while both preserving its integrity and planning for sharing results.

Module 7: Analyzing Data and Disseminating Results - Guidelines


The Toolbox

7.01 Household Baseline Report
7.01a Handbook for Report [PDF, 650KB]
7.01b Indicators: Rwanda [Excel, 60KB]
7.01c Stata Do Files: Rwanda [Zip, 135KB]
7.01d Power Calculations: Rwanda [Zip, 759KB]
7.02 Health Facility Baseline Report
7.02a Suggested Report Outline [Word, 15KB]
7.03 Community Health Worker Baseline Report
7.03a Stata Do Files: Rwanda [Zip, 23KB]
7.04 Stata ado file Balance tables [Zip, 207KB]
7.05 WHO Anthro package [Zip, 748KB]
7.06 Stata training [Zip, 2.34MB]
7.07 Stata training Validate Design [PDF, 1.41MB]

In this module, you will learn how monitoring and documenting program implementation can help inform the relevance of impact evaluation design and analysis. You will discover examples of program implementation adjustments that can affect the impact evaluation, and why it is important to monitor program implementation. You will also learn how complementary data sources can help save monitoring costs and time. At the end of this module, you will know what to monitor, what can be used to monitor interventions, and how documenting results can help ensure the impact evaluation stays relevant over time.

Module 8: Monitoring and Documenting RBF Programs - Guidelines


The Toolbox

8.01 Monitoring Indicators: Rwanda [Excel, 10KB]
8.02 Field Supervision Visit: Rwanda [Excel, 24KB]

  • Exploring the Impact of Performance-Based Financing on Health Workers’ Performance in Benin
    Size: 2.41 MB

Resource Information

Document Type: (PDF) Download
Author/s: Mylene Lagarde, Samantha Burn, Lionel Lawin, Kefilath Bello, Jean-Paul Dossou, Patrick Makoutode, Beatrice Sourou Goufodji, Christophe Lemiere, Maud Juquois
Countries: Benin
Date of Publication: November 2016

Share This Resource