Case Study i: Process Evaluation Model, Smoke Free Families Program

R. Windsor, H. Whiteside, Jr., L. Solomon, et al. “A Process Evaluation Model for Patient Education Programs for Pregnant Smokers,” Tobacco Control (2000) 9, Supplement III (see publication for references).

The following case study presents an example of the application of the PEM by the Smoke Free Families (SFF) Program of the National Program Office (NPO) funded by the Robert Wood Johnson Foundation (RWJF). The primary questions in a process assessment evaluation of a patient education program for pregnant smokers are the following: (1) What procedures should a trained professional routinely provide to a pregnant smoker at her first visit and at follow-up visits? (2) What is excellent clinical practice standard for counseling for this risk factor and patient population? (3) Were the new procedures based on normative criteria (evidence based) developed by a consensus of experts? (4) Did staff participate in the development of the implementation plan? and (5) Did staff perform/provide the procedures to patients as planned at each OB or prenatal care visit?

Identification of the number of patients screened and recruited each week who smoked—Procedure 1 (P1)—was the first task for all projects. Baseline data on patients who were screened and recruited, as well as refusals, documented the daily, weekly, monthly, and annual census for each project by site. Among each eligible cohort of 100 patients who smoke (A), a number of patients accept the program at each site (B). These criteria produce information to compute an exposure rate for each procedure (B/A = C). As noted in Table 5.7, each SFF study had planned at the first visit a smoking status and psychosocial assessment of patients—Procedures 2 (P2) and 3 (P3). Patients in this example were scheduled to receive each of the next seven procedures at future visits. This example indicates that intervention-patients will receive Procedures 7, 8, 9, and 10.

Under the National Program Office Process Evaluation Guidelines (1994-2004), each of the RWJF- SFF grantees had to define and to implement a set of new patient education and counseling procedures. As noted in Table 5.7, one of the first steps in the preparation of a process evaluation plan was to require each RWJF program to define its essential new patient assessment and intervention methods for each visit, and by patient-staff contact. A practice Performance Standard (D), based on Guidelines and

an expert panel review, defined the expected level of provider performance and levels of patient exposure to each procedure (P).

The National Program Office (NPO) used 100% as an absolute practice standard (D) for each procedure. An Implementation Index (E) for each procedure is derived by dividing the exposure rate (C) by the practice-performance standard (D). In this example, where the practice-performance standard (D) is 100%, the Implementation Index (E) and exposure rate (C) are equal. A composite of all Implementation Indexes (?E) provides a summary index of the successful delivery of a patient assessment and education program: a Program Implementation Index (PII). A PII > 0.90 is an excellent level of implementation for a multi-component program.

Illustrative data for the 10 clinical practice procedures for the E group patients are presented in Table 5.8. These data indicated that the project needs to increase patient exposure to Procedures 6-7-8-9-10. Each SFF grantee had the responsibility to apply the PEM to patients at all sites to produce implementation data for its procedures. A staff training plan can be prepared to improve a specific exposure rate (C) or implementation index (E), when problems are documented: for example, when an exposure rate or Implementation Index falls < 90%.

Patient Assessment and Counseling Procedures (P...)

An example of an unsuccessful implementation is presented in Table 5.9.

Data and the PII in Table 5.9 are excerpted from the publication. Although >100 was the target sample size for an E and C group, this process evaluation included only 42 E group patients.

After patients had undergone a telephone screening for inclusion in this study, each was asked to set a quit date within the next two weeks and mailed treatment materials. Women in the E group (usual care plus video) received the calendar, tip guide, and the six-video

table 5.8 Process Evaluation Example

Program Implementation Index =

Ze/P = 0.90 + 1.00 + 1.00 + 1.00 + 0.95 + 0.95 + 0.80 + 0.80 + 0.70 + 0.70/10 = 0.95

n

X = intervention group-component; O = patient observation-smoking status; P = procedure.

program. All follow-ups were conducted by telephone. Major patient assessments were conducted two to three days after a quit date, four to five weeks after the quit date, and one month postpartum. Abstinence, negative affect, coping stress, and self-efficacy were obtained by phone interviews only. No counseling was provided during any of the phone follow-up visits.

As noted in Table 5.9, the most significant difficulties encountered in this study were patient recruitment and retention. While there may be some appeal for using videos as a minimal intervention, the lack of personal contact may have contributed to very poor compliance. As noted, very low patient adherence was an issue in this study. Multiple factors, including poor commitment, high nicotine dependence, and affective disturbance, may have contributed significantly to the overall poor adherence to the patient education procedures and low cessation rates. The limited value of planning a future patient education program using six videos is self-evident.

As documented in the case study, the primary value of the PEM is that it can and does provide data for weekly and/or monthly progress reviews for each site and individual providers. It can be used to identify specific implementation problems by site and specific staff. In case study 1, for example, six videos for patients were created. The process data confirmed almost no use of the videos beyond video 3 or video 4. These data, combined with patient feedback, require a reduction of the number of videos. The routine application of the PEM documented the degree to which the clinical staff of the example had implemented all procedures as planned. The PEM provided empirical data about the feasibility of routine delivery and replicability of procedures at comparable settings. It is also the primary method used to prepare a cost analysis of new and existing health education programs. Future studies should apply the PEM in planning an evaluation.