DEVELOPING THE EVALUATION PLANS
When planning an evaluation, two documents are completed in as much detail as possible. The data collection plan lays the initial groundwork and answers the five key questions outlined in Table 3-6. Providing detailed answers to these questions up front sets the scope of the data collection process. An example of a completed data collection plan is shown in Figure 3-2. In this program, sales associates are selling an upgrade to an existing product using a mobile device. Level 1 and Level 2 collections are built into the learning program. Level 3 is collected with a web-based questionnaire sent by the sales coordinator. Level 4 data are monitored in the system and tracked by the evaluator and from system records.
Table 3-6. Data Collection Plan Key Questions and Descriptions
|
Key Question |
Description |
|
What do you ask? |
The answers to this question lie in the program objectives and their respective measures. |
|
How do you ask? |
How you ask depends on a variety of issues, including resources available to collect data. For example, Level 2 data may require tests, self-assessments, or exercises. |
|
Whom do you ask? |
Use the most credible source; sometimes this includes multiple sources. |
|
When do you ask? |
Timing of data collection is critical, particularly for application and impact measures. Select a point in time at which you believe application and impact will occur. |
|
Who does the asking? |
Typically, the system collects data at Levels 1 and 2. For the higher levels of evaluation, representatives of the evaluation team may be assigned specific roles. |
The second planning document is the ROI analysis plan, which includes the seven key information categories listed in Table 3-7. These seven key areas are addressed in detail in the ROI analysis plan, as shown in Figure 3-3. Two impact measures are monitored—monthly sales and time to first sale. Although the upgrade is released to all sales associates at the same time, not all the sales associates were using the mobile learning program. This provided an opportunity to compare a user group to a nonuser group (experimental versus control group). The profit margin for the upgrade was used to convert to money. The profit of the first sale was not included since it was included in the total sales of the upgrade.
Planning in detail what you are going to ask, how you are going to ask, who you are going to ask, when you are going to ask, and who will do the asking, along with the key steps in the ROI analysis will help ensure successful execution. Additionally, having clients sign off on the plans will ensure support for the evaluation approach when results are presented. These planning documents (Figures 3-2 and 3-3) are explained in more detail in chapter 11.
Figure 3-2. Completed Data Collection Plan
|
Program: Product Upgrade With Mobile Learning Responsibility: Date: |
||||||
|
Level |
Broad Program Objective(s) |
Measures of Success |
Data Collection Method/ Instruments |
Data Sources |
Timing |
Responsibilities |
|
1 |
REACTION & PLANNED ACTIONS Achieve positive reaction on: • Relevance to my work • Recommend to others • Important to my success • Intent to use |
Rating of 4 out of 5 on a composite of four measures |
LMS survey, built into program |
Participant |
End of program |
Program manager |
|
2 |
LEARNING Learn to use five concepts to sell new upgrade: • Rationale for upgrade • Features of upgrade • How upgrade will increase client profit • Pricing options • Implementation and support |
Achieve 4 out of 5 correct answers on each module Achieve 20 of 25 total correct answers |
True/False quiz |
Participant |
End of program |
Program manager |
|
3 |
APPLICATION/ IMPLEMENTATION Use of five skills: • Explain rationale for upgrade • Identify key features of upgrade • Describe how upgrade increases client profit • Identify pricing options • Explain implementation and support • Make the first call in 5 days |
Rating (4 of 5) on a 1-5 scale System check |
Questionnaire, webbased Performance monitoring |
Participant Salesforce. com |
1 month after program 1 month after program |
Evaluator |
|
4 |
BUSINESS IMPACT • Increase in sales to $10,000 per month • Sell first upgrade in 3 weeks |
Monthly sales per associate Actual sale |
Business performance monitoring Business performance monitoring |
Salesforce. com Salesforce. com |
3 months after program 1 month after program |
Evaluator |
|
5 |
ROI 30% |
Comments: |
||||
Table 3-7. ROI Analysis Plan Key Areas and Description
|
Key Area |
Description |
|
Methods for isolating the effects of the program |
Decide the technique you plan to use to isolate the effects of the program on the impact measures. |
|
Methods for converting data to monetary value |
Identify the methods to convert impact measures to monetary value. |
|
Cost categories |
Include the cost of needs assessment, program design and development, program delivery, evaluation costs, along with some amount representative of overhead and administrative costs for those people and processes that support programs. |
|
Intangible benefits |
List those measures you choose not to convert to monetary value are considered intangible benefits. |
|
Communication targets for the results |
Identify those audiences to whom results will be communicated. |
|
Other issues that may influence the impact or the evaluation itself |
Anticipate any issues that may occur during the learning process that might have a negative effect or no effect on impact measures. |
|
Comments or reminders to the staff managing the program |
Place reminder notes of key issues, comments regarding potential success or failure of the program, reminders for specific tasks to be conducted by the evaluation team, etc. |
