Design #3: Time Series
Time series designs (TSD) and time series analysis (TSA) have been infrequently applied and generally under-utilized as a design to evaluate the “effectiveness” of HP-DP programs (Windsor et al., 2004). Biglan, et al. (2000) noted: “Greater use of interrupted times-series experiments is advocated for community intervention research.” Time series designs enable the production of knowledge about the effects of state- and country-wide intervention health policies in circumstances in which a randomized trial is too expensive or simply impractical. Design #3 requires the availability and accessibility of an existing valid and complete data and information system for the target area, population, and problem. These data are essential to accurately describe past, current, and future incidence rates of a specific risk factor(s) or event(s) and impact rates for multiple years, for example, DUI rates, or motorcycle injury rates.
A PHASE 4 Dissemination-Effectiveness Evaluation is the most likely type of evaluation to use a TSD to assess the impact of a new state-wide or system-wide public health program. This design is especially appropriate for evaluating the effects of a new health policy, tax, law, or the systemwide dissemination and adoption by an agency and staff of a new “HP-DP Best Practice” intervention to be delivered to all eligible clients. Design #3 can be considered if a program can:
- • Establish that a routinely reported data monitoring system exists for the HP-DP impact rate, is accessible, and is current;
- • Establish the validity, reliability, completeness, and stability of impact measurement and rates;
- • Establish the periodicity-pattern of the impact or outcome rate being examined for a large well-defined problem, system of care or services, population at risk, and a specific geographic area;
- • Document at multiple monthly, quarterly, or annual data points at least two to three years before, and two to three years after the HP-DP intervention was introduced; and
- • Introduce the HP-DP system-wide intervention at a specific time and, if well justified, to withdraw the intervention abruptly at a specific time period in the future.
The application of a time series design (TSD) requires that an adequate number of valid observations, data, and rates are available, preferably over a three- to five-year period before and after implementation of the HP-DP policy program, to document behavioral or health outcome rate trends. The observation points should occur at equal intervals—monthly, quarterly, semi-annually, or annually—and cover a sufficient time period to confirm pre-intervention and post-intervention variations for an impact rate. Observations and analyses of a behavior change trend over time, however, even with fewer data points (e.g., two to four baseline and two to four follow-up assessments covering a two-year period), may represent a significant improvement over Design #1.
Although most discussions of this method refer to a TSD as a “quasi-experimental design,” it is arguable that in some cases it should be referred to as an “experimental design.” An evaluation that applies a TSD, if successfully implemented, can produce results with high internal and external validity. Case study 4 in this chapter is an excellent example of the application of a TSD and analysis in the evaluation of the impact on surface miner injury rates of a national health and safety training policy and program. It had high internal and external validity for the US population of 110,000 miners and 10,000 mines. There are multiple examples in the literature.
If applying a TSD, a program needs to examine the extent to which the evaluation design can control for measurement, selection, and history biases. Because of the long duration of an evaluation, although selection and measurement are very important biases when a TSD is applied, historical biases are a central concern. People, places, environments, and conditions change over a 3-10 year period. The plausibility of the impact of factors such as weather, seasonality, major local or national historical events, and changes in health policies, taxes, or procedures must be examined. Because the principal issue in applying a TSD is to document the significance of a trend in an observed rate, the HP-DP treatment needs to be powerful enough to produce and sustain significant positive shifts in an impact rate beyond normal variations. The threat of external and internal historical events increases significantly with the duration of the evaluation.
An excellent example of the application of a time series design was published by the US National Bureau of Economic Research (2011) by Chen, Jin, Kumar, and Shi in “The Promise of Beijing: Evaluating the Impact of the 2008 Olympic Games on Air Quality.” They used local Air Pollution Index (API) data and Aerosol Optimal Depth (AOD) particulate data from NASA satellites from 2000 to 2009. The researchers confirmed, from thousands of observations in multiple cities, that the air quality (API Index) improved from 109 in 2000-2001 to 77 during the Olympic Games in 2008. It reverted to 83 one month after the games and to 96 within 12 months in 2009. The program to improve air quality in Beijing was one of the largest natural experiments in the literature. This complex, impact evaluation and analysis should be of interest to HP-DP graduate students for discussion in class about the application of a TSD in real-world situation.