Subjective Evaluations

Subjective evaluation methods involve determining the importance of the outcome of an intervention through ratings or assessments from the targeted individual, family members/friends, or clinicians who have contact with the individual. The issue that is assessed is whether the individual or those in contact with the individual detect a meaningful change in some outcome measure—for example, an individual feels that he or she has better coping skills if the treatment or intervention is valuable and acceptable. The need to show that a treatment is feasible and acceptable within community settings is becoming increasingly important given the current emphasis on translational research within the social and behavioral sciences. Interventions have little chance of succeeding in the community if the target population finds them cumbersome or is unwilling to accept or implement them or if the outcomes are not perceived as important.

Social Validity. In this regard, measures of social validity have been discussed as important indicators of clinical significance. Social validity is a multidimensional construct that includes both acceptability and importance (Foster & Mash, 1999). There are three distinct but related elements of intervention programs that can be assessed for social validity: (a) the goals of treatment (e.g., enhancement of problem-solving skills), (b) the treatment procedures (e.g., home-based intervention approach), and (c) the outcomes produced by the treatment (e.g., reduction in the frequency of disruptive behaviors). Generally, treatment goals are assessed for both their importance and acceptability; treatment procedures are assessed for their acceptability; and treatment outcomes are assessed for their importance (Is the outcome meaningful to the individual?) and derived benefits.

Assessment of social validity is typically achieved by using subjective evaluations, which involve having study participants rate interventions in terms of their overall value, the extent to which the intervention was helpful or beneficial, the acceptability of the intervention protocols, and whether they would recommend the intervention to others in similar circumstances. In the review of caregiver intervention studies conducted by Schulz and colleagues (2002), 14 of the 43 reviewed studies included measures of social validity. The typical finding was that the majority of participants rated the intervention as helpful, beneficial, or valuable. However, the authors warned that the results should be interpreted with caution with respect to generalizability as they are based on rating of participants who choose to remain in the study. In addition, respondents may have been biased in their ratings in a desire to please the study interventionists, and they may have felt a need to provide positive ratings to bring value to their efforts in participating in the study.

It is also important to note that subjective ratings may not necessarily correlate with actual behavior—a caregiver may indicate that increasing support from other family members is an important outcome, but they may choose not to participate in conference calls with other family members. There may also be a lack of congruence between ratings of importance and ratings of acceptability, so decisions may have to be made regarding the relative importance of these criteria. Careful attention also needs to be paid to the selection of individuals chosen to do the evaluations. For example, health care professionals may have different ideas from caregivers about the importance of intervention goals.

When choosing how to measure social validity, it is important to consider the purpose of the assessment and the phase of the pipeline. The intervention development phase might involve the use of focus groups or pilot testing to evaluate the contents of the intervention protocol. At all phases of the pipeline, it is important to identify relevant aspects of the intervention to be included in the evaluation. For example, in an evaluation of the Personalized Reminder Information and Social Management (PRISM) system (see Chapter 14), an evaluation was included that assessed the value of the overall system and each of the features with respect to whether they improved the ability to perform various everyday activities; the usefulness of the system and the features; and the adequacy of the training protocol. Also included were measures of system usability and acceptability.

Finally, when assessing the goals of an intervention, it is important to distinguish between ultimate or distal goals (e.g., reduced stress, improved relationship with a spouse) and more instrumental goals (e.g., enhanced problem-solving skills, improved communication skills). Instrumental goals are outcomes that are hypothesized to be related to ultimate outcomes. The distinction between ultimate and instrumental intervention goals has important implications for defining the clinical significance of an intervention. Of course, all measures of social validity should be pilot tested with representatives of the target population.

< Prev   CONTENTS   Source   Next >