PRETESTING AND LEARNING FROM MISTAKES
There is no way to emphasize sufficiently the importance of pretesting any survey instrument. No matter how much you do to prepare a culturally appropriate questionnaire, it is absolutely guaranteed that you will have forgotten something important or that you will have poorly worded one or more vital element. These glitches can only be identified by pretesting.
If you are building a self-administered questionnaire, bring in a dozen or more pretest respondents and sit with them as they fill out the entire instrument (Sheatsley 1983). Encourage them to ask questions about each item. Your pretest respondents will make you painfully aware of just how much you took for granted, no matter how much ethnographic research you did or how many focus groups you ran before making up a questionnaire. There is no gurantee, by the way, that a dozen pretest respondents are enough. If you’re still learning a lot after a dozen pretest respondents, then bring in some more.
For face-to-face interviews, do your pretesting under the conditions you will experience when the survey is underway for real. If respondents are going to come to your office, then pretest the instrument in your office. If you are going to respondents’ homes, then go to their homes for the pretest (box 9.6).
Never use any of the respondents in a pretest for the main survey. If you are working in a small community, where each respondent is precious (and you don’t want to use up any of them on a pretest), take the survey instrument to another community and pretest it there. This will also prevent the pretest respondents in a small community from gossiping about the survey before it actually gets underway. A ‘‘small community,’’ by the way, can be ‘‘the 27 students from Taiwan at your university” or all the residents of an Indonesian rice-farming village.
If you have a team of face-to-face interviewers, make sure they all take part in the pretest—and be sure to do some of the pretesting yourself. After the pretests, bring the interviewers together for a discussion on how to improve the survey instrument. Ask them if people found some questions hard to answer—or even refused to answer. Ask them if they would change the wording of any of the questions. Check all this yourself by watching a couple of interviews done by people on your team and note when informants ask questions and how the interviewers respond. That way, you can train interviewers to respond in the same way to questions from informants.
As you conduct the actual survey, ask people to tell you what they think of the study and of the interview they’ve just been through. At the end of the study, bring all the interviewers back together for an evaluation of the project. If it is wise to learn from mistakes, then the first thing to do is find out what the mistakes are. If you give them a chance, your respondents and your interviewers will tell you.