EVALUATION DESIGN

The Government of Armenia, with the technical support of the U.S. Agency for International Development have introduced a national Performance-Based Financing (PBF) program at the primary health care (PHC) level.  Under this program, a bonus is paid annually to PHC providers based on their level of performance, in addition to capitated payments. The goal of the program is to improve the provision and quality of maternal and newborn health (MNH) and non-communicable disease (NCD) services by: 1) increasing the level and the frequency of incentive payments, 2) strengthening the verification of results, and 3) introducing three payment indicators for reproductive health services (cervical cancer screening, quality of prenatal care and additional laboratory examination from prenatal care).

Since the performance-based incentive mechanism in Armenia is scaled-up to all PHC providers, it is not possible to evaluate its impact using a prospective impact evaluation. Instead, the evaluation will focus on measuring how rapid feedback mechanisms provided by patients can help improve accountability and raise the quality of care in selected health facilities. The evaluation proposed to rigorously test a mechanism that allows patient feedback using a computer assisted phone interviews (CAPI) survey. Based on patient responses, the aggregated quality score at the health facility level will be calculated and shared in three ways that are thought to influence the quality of care and accountability.

In the first case, the quality score will only be disseminated to the health facility and the supervisory authority in the health system.

In the second case, the quality score will be shared between the health facilities, supervisory authorities, and publicly disseminated.

Finally, in the last scenario, the quality score will be communicated to the health facility and the best performing facilities with good performance (based on the CAPI survey) will be offered additional non-financial rewards.

To measure the impact of the CAPI-based patient feedback mechanism, a rigorous impact evaluation will randomly allocate the participating health facilities in four groups.  The first three groups will include health facilities each implementing one of the three aforementioned scenarios of sharing the quality score of a health facility. The fourth group will be a control group, thus the health facilities in this group will not be implementing any interventions. The underlying question is: does rapid feedback mechanism by patients contribute to improving the coverage and quality of care of NCD and MCH services, and, which of the three variations in design described above lead to greater gains in quality of care?

FINDINGS

The pilot for the impact evaluation to test appropriate tools (SMS or Computed-Assisted Telephone Interviewing) for survey has been completed in 2016 before scale-up. The conclusion was that the response rate was much higher using a CAPI survey. Thus, this method will be used for the scale-up of the pilot program and the evaluation.