Questions Based on the EPD

1. Did those who participated in the program perform significandy better on the test than those who did not?

2. Did supervisors evaluate students' drug audits as satisfactory?

3. Were students able to identify situations in which a law does or does not apply?

An evaluator will take this questioning phase one step further by asking some questions not specifically based on the EPD, but which are nonetheless applicable and valuable to the dialogue.

AN EVALUATOR'S PROGRAM DESCRIPTION DIALOGUE

Evaluator: As we discussed, the purpose of this meeting is to devise an evaluator's program description. Let's begin with my telling you howl understand the general purpose of the program ....

Program director: Let me interrupt to inform you specifically about our objectives and activities. One of our major objectives is to have students observe the effects of a variety of drugs on the human body.

Evaluator: Why is this important?

Program coordinator: Usually, pharmacy students learn about the effects of drugs from lectures, textbooks, and from observations of laboratory animals; they rarely are given an opportunity to see human reactions to drugs. We think this is an important part of a pharmacist's training because it makes students more aware of potentially dangerous drugs as well as what happens when certain drugs are taken in combination with other drugs.

Evaluator: I see. Making pharmacists more aware of effects of drugs is your goal, and observing drug reactions is an activity to achieve the goal.

Program coordinator: Yes, and in addition to the observation of human subjects, another activity for this goal is having students attend lectures on the nature and treatment of frequently occurring medical problems, like hypertension.

Evaluator: At the end of the program, how will you know if the students have become more aware?

Program director: All pharmacy students take a test that requires them to name the likely reaction when one or more drugs are administered to patients with certain medical conditions. We will consider the program a success if students who participated in the program perform significantly better on the test than students who did not participate in the program.

Evaluator: What other goals does the program have?

Program coordinator: Another major goal is to teach pharmacy students how to audit medical charts to assess the appropriateness of the medications. We do this first by teaching our students how to read medical charts and second by supervising them as they do chart audits in nursing homes.

Evaluator: How will you know whether or not students can perform these assessments?

Program director: When supervisors evaluate their students' performance, we'll know if they find them to be satisfactory.

Evaluator: How about the accuracy of the audit results?

Program director: The coordinator also suggested checking the audits, but I have faith that the supervisor's evaluation is sufficient evidence of program merit.

Evaluator: OK. Are there any other goals?

Program coordinator: No.

Evaluator: When I received some of the program brochures, I noticed that the legislation that funds this program requires that you teach students about laws relating to informed consent and malpractice.

Program director: You're right. That's not one of our prime goals, but we do devote part of our instruction to it. Obviously, we would not be eligible for funding if we overlooked it, so we discuss the laws and the professional situations in which they apply.

Program coordinator: We measure their understanding by asking students to cite laws and their provisions.

Program director: We also ask students to identify situations in which the law does or does not apply.

Table 4.1 EPD for the Pharmacy Training Program.

Questions Not Specifically Based on the EPD

1. How do the goals, activities, and outcomes of this program compare with other clinical programs for pharmacists?

2. What will be the benefits of increasing the program funds by 10 percent?

3. How well was the program managed?

With the EPD clearly outlined, the evaluator can begin to identify monitoring and evaluation activities. However, much of what goes into the evaluation activities is dictated by the purpose of the evaluation. In Chapter Three we discussed the reasons for evaluation in terms of whether it is to fulfill some mandate, to identify what is happening in a program, or to determine if one process is more effective than another. Using the preliminary information you glean from the EPD and the purpose of the evaluation, you will select a specific evaluation model. These models are covered in the next chapter.

Putting It All Together

As with any project, you have to start somewhere. The evaluator's program description is a great starting point for an evaluation. If you are not intimately familiar with the program you are about to evaluate, then you need to learn something about the processes that are planned or are already in process. Also, you can learn about program staff perspectives on the crucial elements of the program simply by asking questions to develop the EPD. What questions do others want answered? What other stakeholders may need to be considered (Quinones and Kirshstein, 1998)? In the case of the pharmacy program, you have already conferred with the program director and program coordinator. Some stakeholders that you may want to talk to include instructors, supervisors, students, state officials, and other health professionals.

An interesting side benefit of the EPD process is the rapport that you can begin to develop with the program staff. Reviewing goals, standards, and activities and asking the good questions that result in an effective evaluation will align the thinking and actions of all concerned. This brings to mind a quote attributed to Sir Josiah Stamp, who served at the Inland Revenue Department (England) from 1896 to 1919: "The government are very keen on amassing statistics. They collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the village watchman, who just puts down what he damn pleases." Unless a rapport, a trust, is built between the evaluator and the program staff, they may tell you anything they please, whether truth or falsehood, and you may never know the difference. Or they may tell you nothing, which is just as bad.

The EPD provides you with information that you can use to develop evaluation questions and the evaluation design format that will guide your actions from this point forward.

Key Words and Concepts

Evaluator's program description (EPD): A statement prepared by the evaluates after working with stakeholders to amplify and clarify all aspects of the program, including goals, objectives, activities, and anticipated outcomes

Evaluation questions: Questions that will be answered by the evaluation and are based on the EPD