We have described the importance of considering a number of issues before embarking upon an evaluation of a behavioral intervention. The most important consideration is determining how one’s specific research questions and hypotheses will contribute to knowledge in the existing scientific literature. One of the most important tasks for the investigator is to carefully construct a priori hypotheses, select reliable and valid dependent measures, and to specify an effect size that would have to be achieved to be considered clinically meaningful and practically important. Statistical significance merely refers to the confidence that obtained results are reliable and not a function of chance. While a nonreliable result is not worth pursuing, it is equally improper to present a trivial (albeit reliable) result as important. Before the results of a study are analyzed, investigators should have a clear indication as to their criterion for statistical significance, distinguish primary analyses from secondary analyses, and use appropriate corrections for multiple post hoc tests that raise the possibility of Type I error rates. There are a number of correction procedures available, some very stringent and others less so.
The levels of measurement have profound effects on how obtained data are analyzed and interpreted (see as well Chapter 14). In general, interval-level data maximize obtained information and allow for the greatest flexibility in analyzing test results. Since many models depend on measures of fit, it is imperative to include or account for important variables that will be related to outcome. An appreciation of mediator and moderator variables is very important in many analyses such as path analyses, other regression-based approaches, and any longitudinal data analyses from classic least square models to growth curve and random effects mixture models based on maximum likelihood procedures.
In situations where there are natural hierarchies and individuals are nested within these hierarchies, multilevel modeling approaches are powerful approaches to account for the nonindependence of observations. Growth curve and growth mixture models enable the investigator to examine trajectories of growth over three or more time points and they provide a meaningful way to test the effects of different variables on the individual trajectories of change over time.
Despite the plethora of newer statistical approaches, an old adage is true. No amount of sophisticated data analytic approaches will be able to compensate for the lack of methodological rigor in study design and measurement. Random assignment and blinding raters to expected outcomes are vitally important as are the expectancies of the research participants themselves. In any treatment study, those who drop out of studies are often those who are different from completers and may be the ones who did not derive actual treatment benefit. As a result, ITT and CACE approaches must be strongly considered.
Finally, despite our attempts to recruit research participants who are representative of a particular population, those who volunteer for research studies may differ in many critical ways from target clinical populations, and efficacy in a particular trial may not generalize to effectiveness with clinical groups that do not closely resemble the sample in which a study is based. This points to two pressing needs in the field: (a) independent replication studies (unfortunately, negative results are not often published) and (b) meta-analyses where effect sizes can be pooled across a number of studies to arrive at a conclusion as to the efficacy of a specific set of interventions. Despite our attempts to examine effects at a group level, individuals have complex and varied responses owing to unique and individual characteristics. Recent attempts to more fully incorporate individual-level participant data in our analytic models may result in better tailored interventions that can be appropriately targeted to those in need.