Data Sources

Starting Point

It might be helpful, at this point in the book, to discuss Logic Models. Usually, logic models are a natural product of a strategic planning process. It is used to bring clarity to the planning and create consensus or a better understanding of the project. However, it can also help parties focus on the evaluation.

In a nutshell, people who use logic models see them as a collection of "if/then" statements. Thinking about elementary Algebra, "If X, then Y. If Y, then Z." The "X's, Y's, and Z's" are replaced with Inputs, Outputs, and Outcomes. The Inputs consist of the Program Investments, such as Staff, Time, Money, Volunteers, Materials, Technology, and Facilities that are used by a program to perform its activities and services. The Outputs consist of program Activities and Program Participants. Activities relate to the actions taken by program staff to bring about the desired changes in clients such as Training Sessions, Courses, Workshops, Services, Product Development, Counseling, Manufacturing Products, Partnering, and Assessing. The Program Participants refer to those recipients of the Activities, such as Students, Trainees, Clients, Employees, Stakeholders, Customers, and Decision Makers. The Activities are what "we do" to those who "we reach." Finally, the Outcomes are multilevel and consist of Short-Term, Medium-Term, and Long-Term Outcomes. Short-term outcomes might include the immediate results of an activity, such as Learning, Attitude, Knowledge, Performance, Skill, or Understanding. Medium-term outcomes might include results of activities after the recipients have been away from the activity for a while, back in their naturalistic environment, and had time to let them manifest in some Action, such as Behavior, Practice, Decision Making, or Performance. Long-term outcomes might include eventual changes in Conditions in the recipients' work environment, lifestyle, relationships, or perspective, such as Productivity, Sociability, Communication, Economics, Collaboration, or the Return on Investment realized by the recipient's or project's sponsor. So the basic structure of a Logic Model would look like Figure 6.1.

Figure 6.1 Basic Logic Model.

Basic Logic Model.

The function of the evaluator would be to work with the project staff to flesh out the structure of how the staff sees the project under review given the flow of this model. An example of such an application is depicted in Figure 6.2.

This application of the Logic Model to a marriage enrichment program begins by defining the inputs of staff, money, and so on. The Output-Activities include developing the program, conducting the sessions, and having the case manager work with the couples. The Output-Participants are those couples who were targeted for services and who agreed to participate. The Outcomes-Short-Term reflect the specific objectives that the project identified as results to be achieved at the end of the project. The Outcomes-Medium-Term are the changes that the couples should experience as they begin implementing these new skills. The Outcomes—Long-Term are the changes in family life that should be realized after the new practices are institutionalized by the families.

The picture painted by the use of a Logic Model gives the evaluator a clearer view of how the project components are expected to flow and what the project staff were thinking would happen before, during, and after the project was implemented. This activity goes a long way to assuring that the evaluator and project staff are all on the same page in their understanding of the project. Also, it gives the evaluator a notion about expected results (short-, medium-, and long-term) so that data sources (instruments) can be identified.

At the point you are ready to identify data sources for your evaluation effort, you already

Know who the audience is

Have had meetings with stakeholders to decide on evaluation questions

Have written program objectives in hand

Have decided on the program activities you will observe

Figure 6.2 Marriage Logic Model.

Marriage Logic Model.

As the program or project evaluator, you have already learned how to identify what a program intends to accomplish and how the staff anticipate meeting those objectives. You have also learned how to use that information to structure evaluation questions that will assist the staff in learning how their program works and might be improved.

If you know what you need to measure, you will now need to identify the data sources that will support your evaluation (or how you measure). The first step is to once again look back to the EPD process during which you asked stakeholders what, if any, data sources they currently had in place to monitor or measure program outcomes. If they are appropriate, you may want to incorporate those existing data sources into your evaluation design. However, if none of those in place are appropriate, or if they are outdated, incomplete, or unavailable because of privacy or other issues (Wholey, Hatry, and Newcomer, 1994; 2004), you will have to devise one or more sources suited to your evaluation.

< Prev   CONTENTS   Next >