This blog is cross-posted on The World Bank Development Impact Blog

On October 3rd, I sent out a survey asking people what was the biggest, most embarrassing, dramatic, funny, or other "oops" mistake they had made in an impact evaluation. Within a few hours, a former manager came into my office to warn me: “Christel, I tried this 10 years ago, and I got exactly two responses.”

I’m happy to report I got 49 responses to my survey. My initial idea was to assemble a “Top 10” of mistakes, so I promised the ten winners they would get a small prize. Turns out, assembling a "Top 10" was a bit tricky, but here’s my attempt at classifying the information I got.

 #1 - A first batch of comments were stories of random, funny things that happened in impact evaluations. Here’s one that cracked me up in my office on a Friday afternoon:

A researcher launched a baseline survey in Liberia, and hired a local organization of ex-combatants turned enumerators. The two criteria for enumerators were that they have to be literate and have to be able to drive a motorcycle, as the enumerators would have to ride between villages through three provinces. During the initial training, the enumerators the organization identified were fantastic - smart, hard-working, and kind. On the day of launch of the survey, 14 motorcycles were delivered. Everyone was packed up and ready to go for a few weeks. Then the enumerators got on the motorcycles. About two-and-half miles after driving, it became clear that while the enumerators were literate, they definitely had never driven motorcycles before and essentially lied during the interviews. What was supposed to be a two-and-half hour drive took nine hours going around 15 kilometers per hour. While nobody got really hurt, the enumerators crashed into large mounds of dirt on the side of the road many, many times over the next few days. The researcher noted he had never gotten back from field work with such a layer of dirt encrusted into his skin.

 Another story was just as random, but maybe not as funny:

A team implemented a follow-up survey with 6,000 households in northern Bangladesh about three years after the implementation of an ambitious anti-poverty program. The "oops" was that about 1/3 of the data was gathered during the month of Ramadan. Although most of the team was Muslim, noon anticipated the effect of Ramadan on, for example, nutrition data.

While these two mistakes were kind of random and hard to predict, I did find that most responses reflected much more systematic issues. My next four items in the "Top 5" are of the more systematic type:

# 2 -Hiring the wrong person(s) on the impact evaluation team, or not defining their responsibilities well:

"The evaluation manager decided that surveys do not deliver any credible results”

“We hired local consultants (recommended by colleagues) that were not up to the task”

“My Research Assistant took big shortcuts”.

# 4 -Issues in the design and (preparation of) data collection:

This was by far the most represented category. Issues abound about the “lack of IDs."

“The survey firm failed to include student ID numbers in the administration of the student test. Good luck merging two years of student data by long last names that all start with an 'R.'"

“After the questionnaire was finalized and translated, the cover sheet for the 'mothers' module was one line too long and disturbed the formatting. My colleague decided to lop off the top line to make things fit for printing. Unfortunately, it was the line that included the code linking each mother, who answered the module to the household register.”

“The Cover page of the questionnaire was never printed, so the contact information for the respondents was completely missing.”

Further issues were related to correct specification of the terms of reference of survey firms.

“Verbal agreements with survey firms about changes to the TOR – they are extremely costly. Lo and behold, after field work and most payments are processed, the data were never found, or were held back strategically.”

"We specificed in the TOR that the data from the survey belonged to us. But we never said anything about the data log, and without it, we could not interpret the data, so we were unable to use them. Such a waste!"

#5 - Monitoring the intervention after the baseline seems to be another source of issues: