Evidence for the Effectiveness of Interviewer Feedback, Instructions, and Commitment

Feedback, Instructions, and Commitment

Early Studies

Marquis (1970) conducted the first experiment on the interviewing techniques for his dissertation. He found that interviewer Feedback statements provided only after respondents reported instances of morbidity (e.g., chronic health conditions) led to significant increases in such reports, compared to a control condition in which interviewers provided no feedback. The experimental evidence supported the idea that social reinforcement could improve response accuracy, on the assumption - supported by record check studies (e.g., Madow 1967) - that more reporting is more accurate.

In this study, as in many subsequent experiments conducted by the Cannell team, the interviewers were trained to administer both experimental and control interview treatments, thus serving as their own controls. The alternative design would be to assign different interviewers to different experimental treatments. Each design embodies trade-offs. The first design risks interviewers inferring that a particular method is preferred and behaving so as to support the suspected hypothesis, or misapplying the techniques, thereby attenuating the experimental effects. The second design risks confounding experimental effects with interviewer differences. Cannell and colleagues' choice of the within-interviewer design rests on their practice of not revealing the hypothesis and carefully training a single set of interviewers to follow each questionnaire script strictly. Cases were randomly assigned to treatments. The interviewers and the respondents were all white. The sample was households selected from neighborhoods with similar socioeconomic characteristics and age ranges, a procedure designed to control extraneous demographic variance that could muddy the comparison of interviewing treatments.

The next experiment (Marquis, Cannell, and Laurent 1972) was the first to examine a questioning technique that evolved in later studies into Instructions. The experimental treatments involved the length of questions asked, reinforcement (Feedback), and reinterviews. Question length was manipulated by adding what was intended to be redundant verbiage - not more memory cues or socio-emotional material. The idea, following research by Matarazzo, Wren, and Saslow (1965), was that - as noted in interview observation studies described above - more lengthy speech by the interviewer would lead to more lengthy responses, which, in turn, would yield more health information. This notion, from research on behavior modification, as was contingent reinforcement (feedback), is an example of how Cannell and colleagues melded theories from social psychology with empirical induction.

Respondents in this study were members of a prepaid health insurance plan. Medical record information from respondents' clinic visits for 6 months in 1968 was to serve as the validating criterion for interview reports and allow for detection of both under- and over-reporting. The study's findings were mixed and not easy to explain. One reason for the mixed findings may have been the quality of the record information; there was some concern that the record information was not accurate. The study budget limited the sample size and aspects of the experimental execution. Further, the experimental findings were moderated by education.

Respondents with higher and lower education reacted differently to the question length and reinforcement treatments. Reinforcement seemed to reduce error for respondents with less than high school education but not for those with more years of education. The reverse was true for longer questions. All in all, the study seemed to produce more questions than answers. What carried forward was the continued belief that interviewer reinforcement could be a useful tool and that some form of question lengthening was worth further exploration. The study's authors reasoned that question lengthening might have cueing effects, might suggest that the inquiry is important or that taking time to think about the question is a good idea. The length of the question itself might be a way to provide time for the respondent to think (Blair, et al. 1977; Cannell, Marquis, and Laurent 1977:70).

The next several years saw further refinement of the question length and feedback techniques and the addition of the Commitment procedure to the suite of interviewing approaches examined in experimental studies. The Commitment procedure is a global motivational tool for the interview. It seeks an explicit agreement from the respondent that he or she will undertake actions to reach the goal of rendering complete and accurate information. The procedure was tested in an experiment that examined effects on health experience questions. The experiment included an expanded definition of accurate reporting. The dependent measures were intended to capture the amount of information provided by respondents, as well as the specificity of the response (e.g., number of mentions to open questions, number of doctor visits reported, number of symptoms reported for the pelvic region of the body, specific dates for medical appointments or for which the respondent was limited in his/her activities, and evidence of checking external sources of information before responding) (Oksenberg, Vinokur, and Cannell 1979a).

Instructions grew out of the earlier work on long questions. Cannell and colleagues decided that, rather than speculating about the possible reasons why long questions seemed to have a useful effect, it made sense to insert messages at the beginning of the questionnaire (e.g., a global instruction about the need for accuracy, completeness, and precision) and also just prior to selected questions to inform respondents how to achieve those results. The procedural Instructions focused on, for example, telling respondents to take time and search memory before answering a query.

Instructions were studied along with Commitment and Feedback in a separate experiment (Oksenberg, Vinokur, and Cannell 1979b). The questionnaires in both studies became a script, beginning with requests for Commitment, then global Instructions, and then questions preceded by Instructions and followed by contingent Feedback. The findings of the studies are summarized in Cannell, Miller, and Oksenberg (1981). In brief, the combined forces of the three techniques showed promising results when compared with a control condition containing only questions to be administered. The dependent variables were, again, measures of the amount of information respondents provided and the precision with which it was rendered. Another, smaller experiment (Miller and Cannell 1977) comparing one treatment with commitment, instructions, and feedback with a control that did not include these techniques found higher reporting of behaviors likely to be under-reported (e.g., time watching television, "X-rated" movie attendance) and lower reporting of behaviors likely to be over-reported (e.g., book reading) in the experimental condition. Miller and Cannell (1982), in a national telephone study, found that Commitment and Instructions produced more reporting of socially undesirable attitudes toward television watching, as well as more responses to open questions and reports of health conditions, while the Feedback treatment, when combined with the other techniques, did not add to these effects.

In the late 1970s, the University of Michigan's Survey Research Center (SRC) undertook a methodological study to evaluate the use of random-digit dialing (RDD) sampling and telephone data collection for the collection of health information, specifically to test the feasibility of conducting the National Health Interview Survey (NHIS) by telephone (Thornberry 1987). As part of the mode comparison study (comparing the telephone survey conducted by SRC to the in-person NHIS collected by the U.S. Bureau of the Census for the National Center for Health Statistics), the telephone sample cases were randomly assigned to one of two treatments. The control group was intended to replicate the procedures used by the Census Bureau in which interviewers were limited to a predefined set of behaviors - asking the question as worded and using standard interviewing procedures such as neutral probes. In the experimental treatment, Commitment, Instructions, and Feedback were incorporated into the questionnaire. The researchers examined rates of reports of various health behaviors, with the hypothesis that higher rates of reporting of health incidences were indicative of higher quality data. The study found that for several health behaviors (bed days, work loss days, days with reduced activity, acute conditions, chronic conditions), the experimental form of the questionnaire led to increased levels of reporting.

The authors acknowledged that there was not, at the time, "unequivocal acceptance of the underreporting hypothesis" (Thornberry 1987:23). Marquis, who worked with Cannell on a number of the studies reviewed here, attacked the record check evidence that was used to support the idea that many events are under-reported (Marquis 1978; Marquis, Marquis, and Polich 1986). He and colleagues dismissed findings of "reverse" record check studies, in which the sample of respondents is chosen from records of events - such as the hospitalization and doctor visit studies conducted by Cannell and Fowler discussed earlier - arguing that this design cannot discover over-reports of these experiences. They analyzed the results of "full" record check studies - where the sample and the record evidence are independent - of sensitive behavioral events. They argued that these studies produced more evidence of over-reporting of sensitive experiences than under-reporting. They further argued that the main issue with reports of sensitive information is reliability, not response bias. In response to such criticisms, Cannell and colleagues examined alternative explanations for the increased reporting of health events resulting from the experimental treatments, for example, over-reporting due to telescoping. After considering alternative explanations, the researchers concluded the "most tenable hypothesis is that the experimental techniques facilitated accurate reporting on health variables in this study" (Thornberry 1987:25). We will return to this debate below.