What Defines Improvement?
The definition of improvement in performance following cognitive or functional skills-enhancing interventions depends on the goal of the assessment. Clinical treatment will have a very different set of standards from a regulatory efficacy trial. Further, improvement can be indexed in several ways. These improvements can be defined, in hierarchical order of rigorousness, as statistically significant, clinically meaningful, definitely nonrandom, and normalization of functioning,
Statistically Significant. Statistically significant is the criterion for demonstrating differences between active treatments and control conditions. This is the lowest bar for empirically defined improvement, because it is largely dependent on the sample size of the study. This criterion does not depend in any way on the level of baseline performance and does not require any predetermined end of study level of performance. Although the lowest of the bars that we are discussing, it is still important. Any intervention that does not separate from an inactive treatment cannot be seen to provide reliable improvements. This criterion is also required for an intervention to receive regulatory approval.
Clinically Meaningful. Clinically meaningful is a higher bar than statistical significance. This threshold would imply a certain average degree of improvement in performance for the populations treated. Required as part of this criterion is some notion of what the size of such a change would be and this requires information obtained from other sources other than statistics such as actual indices of functioning in everyday settings. Embedded within this concept is the expectation that a certain amount of improvement in cognition or functional skills would be associated with a certain amount of improved everyday functioning. For instance, several different studies of functional measures have identified threshold levels of cognitive performance consistent with achievement of functional milestones such as independence in residence, or ability to manage finances or medications, or improvements in driving skills. Intervention-related improvements that reach these thresholds would be possible indices of clinically meaningful change (see Chapter 17 for a more detailed discussion of this issue).
Definitely Nonrandom. When a group of participants receives treatment, even if the benefit is statistically or clinically significant for the group, there is likely to be variation in response among the people treated. The assessment of improvement for individuals differs from that for groups in that to be certain that an individual has improved to a level greater than chance, a host of influences on performance such as practice effects requires consideration. The “reliable change index” has been developed in order to quantify whether an improvement in one person exceeds what is expected based upon general influences (Heaton et al., 2001). The reliable change index statistic incorporates the test-retest reliability of the measure and establishes a range of scores that exceeds this level of change. The confidence interval of the reliable change index is typically set at 90%, meaning that there is only a 1 in 10 chance that the threshold amount of change would have occurred by random factors alone.
With commonly used outcome measures for clinical trials in humans, the typical level of change required to define a definitely nonrandom change is in the range of about 1.0 standard deviation (SD) (Leifker et al., 2010). This is a fairly high bar, but in several previous cognitive treatment studies, the group improvements in cognitive outcomes have been as great as 0.8 SD (Bowie et al., 2012; Fisher et al., 2009). This could mean that a number of people treated in those studies manifest individual improvements that are definitely nonrandom.
Normalization of Functioning. This is the highest bar and is not necessarily a goal of every intervention study. Normalization of functioning would imply two things: substantial improvement in functioning that is entirely within the normal range, or improvement on the part of individuals to at least their pre-illness level of functioning if not better. The normal range of functioning is typically defined as within 1.0 SD of the population mean or higher. Further, if an individual’s performance was within that range prior to illness, then their posttreatment functioning should be within that range as well. Normalization is a high bar because many individuals whose performance is slightly below the cutoff for normal cognitive functioning ( — 1.0 SD) are functioning adequately in their lives.