Perspective of an Impact Evaluation (IE) Front Line Agent: What are the lessons learned from the 2012 IE RBF Workshop?

Evaluate the problems and successes of baseline data collection. For the representatives of survey firms, one of the “workshops within the workshop” entailed a review of the problems encountered during the preparation and implementation of baseline surveys of HRITF impact evaluations. Two moderators  (Beatriz Godoy and Juan Muñoz from Sistemas Integrales), who specialize in survey preparation, implementation, and analysis, were tasked with identifying the problems common to all teams and, working collaboratively with the teams, proposing practical solutions to these problems. My blog on impact evaluation in the field (to be posted later this week) discusses the successes and problems that we shared during the workshop on the impact evaluation of PBF in Cameroon. It was reassuring, during this workshop, for my team to discover that countries as a whole shared the same problems encountered in the field (budget constraints, transport and life span of materials, the length of the F1[i] health facility questionnaire, availability of health facility staff relative to requests made of them by patients, time spent in health facilities, the isolated nature of some areas, etc.). We learned that the data entry process should be decentralized as soon as possible and that more time should be spent on training data collection agents, which will enhance mastery of tools and thus certainly permit time to be saved in interviews, and lastly, to document more extensively the database through the production of metadata. Consequently, it was agreed that tools should be designed with greater involvement of all stakeholders and that special attention should be paid by the financial partners to the budget constraints faced by survey firms.
 

Better understand the theoretical and practical aspects of evaluation methods. With regard to the data analysis workshop (led by Damien De Walque and Jed Friedman, World Bank), we obtained more information, among other things, on one of the main methods to be used in the case of Cameroon (double difference), taking into account the identification strategy used in the context of the impact evaluation, namely, the random assignment of intervention and comparison groups. We also gained a better understanding of the angle from which databases are analyzed, that is, not merely as an isolated survey, but more as a substratum that makes it possible to ensure internal and external validity, which is essential to any impact evaluation, in particular the balance of study groups or arms. The theoretical part of the verification methodology of this balance was supplemented later on by the presentation of the elements in the  Impact Evaluation Toolkit, which contains small tools or practical programs in STATA^© or Excel, which are ready for use, are flexible, and can be easily implemented.

 

An effective training approach. In more general terms, what did I learn from the workshop? In terms of form, it should be borne in mind that we had the opportunity, during the course of this workshop, to work with the finest government, survey firm, and World Bank actors in the area of the implementation of impact evaluations of RBF projects. Contacts were facilitated by the methodological approach adopted during the course of the workshop—a learning and knowledge approach, suited to adult training, put participants in a relaxed mindset from the beginning of the workshop, using a fun exercise that had each participant playing the role of “a survey agent,” who had to use all his persuasive talent to capture the interviewee’s attention for one minute, in a bid to obtain information. Through this seemingly innocuous exercise, which was, however, aimed at learning more about what others were doing and the level of intervention in RBF projects, all barriers were broken in a matter of minutes, people opened up quickly, all participants soon felt at ease, and information was exchanged throughout the workshop in an atmosphere where time flew for many persons, despite the time spent on intensely enriching activities. For this alone, I would like to thank the organizers, since the success of a workshop is often attributable to a very small number of factors.
 

Expand the scope of stakeholder action: A holistic vision of the evaluation. From a substantive standpoint, we learned a host of lessons from our exchanges during the course of this workshop. As the representatives of a survey firm, the exchanges allowed us to see beyond the baseline and even endline survey. We were able to envision our involvement from a holistic standpoint, based on a vision of both the baseline and endline surveys, as well as all that could take place between the two surveys and would serve as a confounding factor. For any survey firm, this entails obtaining the resources to be a stakeholder in any routine or qualitative research that would allow for tracking any activity or change in the baseline survey and the follow up survey. In other words, we realized the utility and importance of comparing the information from other RBF monitoring and/or evaluation sources with the information derived from the impact evaluation. Furthermore, for me, the Toolkit, which was briefly presented during the workshop, of which we received the web link and I had received a preliminary version in August 2012 (after data collection and processing, as a member of the evaluation team funded by the HRITF), is a key tool for persons preparing to launch their baseline survey, as well as for those who have already done so but wish to take a necessary step back to better focus on the future in the context of this impact evaluation process.
 

Long-term commitment of survey firms. In addition to these lessons learned and the workshop itself, partnerships and capacity-building in the context of RBF programs and their evaluations can be improved. We, along with other survey firms, expressed the need for study tours in the countries conducting evaluations of this nature, in addition to email exchanges and gatherings such as this workshop, with the aim of more effective sharing and dissemination of the benchmark practices of each. This poses a problem with the formulation and drafting of the budgets of countries and companies, with the budgets of the latter being allocated by bidding for each survey rather than for the entire evaluation process. These budgets should be harmonized from the perspective of the survey company as a partner assisting the government throughout the impact evaluation process. From the standpoint of the survey firm, this process seems to be sequenced in terms of each impact evaluation, giving the impression that the invention phases of the survey company are somewhat discrete while in reality, evaluation is a continuous process.
 

To conclude, I can simply say that the IFORD team and the remaining PBF impact evaluation team in Cameroon went to this workshop satisfied that they had successfully conducted the baseline survey despite the problems, and with a desire to share and learn more about this process from others. We are returning with many challenges, but also certainly armed with better information and, without a doubt, with strong resolve to get ourselves to be a stakeholder in the impact evaluation in Cameroon to the end of the process.

 

---