You can’t fix what you don’t measure.

—William Thomas Lord Kelvin (1893)

The selection of measures to include in a study is perhaps one of the most important, and often the most challenging, aspects of behavioral intervention research. Selection must be carefully planned at the inception stage of intervention development. Measures provide answers to questions regarding whether the intervention “works,” for whom, and to what extent and provide evidence that is used for decision making regarding an intervention at all phases of the pipeline. A common source of misleading results from intervention trials often stems from inadequate attention to the choice of measures—a mismatch between the intent of the intervention and the measurement strategy.

Different measures can relay different stories about the impact of an intervention, so measures need to be carefully aligned with the research questions of interest and what the intervention intends to change, modify, or impact. Consider, for example, a study that is evaluating a new software tool to aid Internet searching. The new tool is being compared to the standard search tool available on the browser. The outcome measures include indices of user performance (e.g., time, errors) as well as user perceptions of usability. If the performance data indicated that the study participants performed an information search task more efficiently using the new tool, one might conclude that the tool is effective and should be adopted. However, if the usability ratings indicated the tool was cumbersome and difficult to use, decisions about implementation of the tool might be different as user perceptions of the usability of technology are strongly related to technology uptake.

In general, the type and quality of the measures included in an intervention research trial are critical to (a) answering questions related to the study goals and hypotheses, (b) detecting change attributable to the intervention, (c) one’s ability to compare findings to previous research, (d) determining the type of statistical analyses that needs to be employed, (e) the internal validity of the study, and (f) furthering theoretical knowledge within the treatment domain area. The choice of measures influences other design considerations in a trial such as the frequency and length of assessments and needed sample size.

Consider an intervention research trial that involved a comparative analysis of the impact of training methodology on the ability of older adults to search the Internet to find credible health information. Two methods were compared: a standard classroom approach led by an instructor; and an interactive, multimedia approach where the instructor acted as a coach, provided one-on-one feedback, and the students completed online exercises. On the basis of the findings, the investigators claimed that the standard method was superior to the interactive method in terms of teaching older adults. However, the claim was based solely on the ratings of three instructors with respect to ease of implementation of the method; there were no indices of student learning or student evaluative ratings. Further, the method rated as easier was already in place in the senior community center where the training evaluation took place. Clearly in this case, it would be hard to make a convincing argument for adopting the new interactive method of training as compared to a scenario where there were findings favoring that method, which were based on student learning achievements and student evaluations.

In this chapter, we discuss (a) the role of measures in behavioral intervention research, (b) criteria for measure selection, (c) the types of measures available, (d) methods for collecting outcome data, and (e) the role of technology in measurement.

< Prev   CONTENTS   Source   Next >