THE MEANING AND MEASUREMENT OF CLINICAL SIGNIFICANCE
Defining Clinical Significance
“Significance” generally refers to the quality of being important (Merriam-Webster, 2014). In behavioral intervention research, we are generally concerned with two types of significance: statistical significance and clinical significance. Statistical significance is based on probability likelihood and is typically operationalized at an alpha level of .05 or .01. Observing a significant difference between two treatment groups means that there is a reliable difference between two groups on a chosen outcome measure or only a 5% (a = .05) or a 1% (a = .01) likelihood that the difference was due to chance. Statistical significance is influenced by factors other than the relationship between the independent and dependent variables such as sample size and variability in the data. In addition, statistical significance does not indicate the magnitude of the difference. The effect size statistic is an indication of the magnitude of a treatment effect and often thought of as a measure of practical significance. Effect sizes are often calculated using Cohen’s “d” (the difference between the means, M1 - M2, divided by the standard deviation of either group) and sometimes interpreted as small (0.0-0.2), medium (0.3-0.5), or large (0.6-0.8), but as cautioned by Cohen (1988), there is some risk in using these operational guidelines given the diversity of behavioral research.
Clinical significance has evolved as a means to determine the impact of an intervention or treatment and refers to the importance of the effect of an intervention and whether it makes a difference in the lives of individuals (Kazdin, 1999). For example, in the case of an intervention to treat individuals with severe depression—has the treatment moved the patient to remission or a more functional level? Measures of clinical significance are usually used as a supplement to measures of statistical significance and are intended to address the issue of impact of an intervention. For individuals, this is clearly important, as an intervention should effect a change on some outcome that is impactful in their lives. Among caregivers, this might include reduced burden, better coping skills, enhanced social support, or the ability to keep the person they live with at home with life quality. Among overweight adults, this might be weight loss, enhanced mobility, or lower cholesterol. Clinical significance is also pivotal to health care professionals and social agencies/policy makers. Clinicians who are faced with choosing among available treatments are often faced with a dearth of information regarding the impact or practical relevance of research findings for individuals. Social agencies and policy makers are also increasingly asking for evidence about real-world effects of treatments when making decisions about investing in intervention programs. Generally, tests of statistical significance are not sufficient to yield evidence that a treatment is worthwhile. Thus, assessment of clinical significance adds a critical dimension to the evaluation of treatment effectiveness that is not captured by standard statistical evaluation methods.
It is important to recognize that what constitutes clinical significance depends upon the problem or issue that is being addressed and the goals of the intervention or treatment (Kazdin, 1999). For example, in the case of a cognitive training intervention directed at schizophrenic patients, it would not be reasonable to assume that patients would obtain normative levels of cognitive functioning. Instead, in this case, improvements in quality of life (QoL) and/or the ability to perform everyday activities could be markers of clinical significance.