When planning an evaluation, two documents are completed in as much detail as possible. The data collection plan lays the initial groundwork and answers the five key questions outlined in Table 3-6. Providing detailed answers to these questions up front sets the scope of the data collection process. An example of a completed data collection plan is shown in Figure 3-2. In this program, sales associates are selling an upgrade to an existing product using a mobile device. Level 1 and Level 2 collections are built into the learning program. Level 3 is collected with a web-based questionnaire sent by the sales coordinator. Level 4 data are monitored in the system and tracked by the evaluator and from system records.

Table 3-6. Data Collection Plan Key Questions and Descriptions

Key Question


What do you ask?

The answers to this question lie in the program objectives and their respective measures.

How do you ask?

How you ask depends on a variety of issues, including resources available to collect data. For example, Level 2 data may require tests, self-assessments, or exercises.

Whom do you ask?

Use the most credible source; sometimes this includes multiple sources.

When do you ask?

Timing of data collection is critical, particularly for application and impact measures. Select a point in time at which you believe application and impact will occur.

Who does the asking?

Typically, the system collects data at Levels 1 and 2. For the higher levels of evaluation, representatives of the evaluation team may be assigned specific roles.

The second planning document is the ROI analysis plan, which includes the seven key information categories listed in Table 3-7. These seven key areas are addressed in detail in the ROI analysis plan, as shown in Figure 3-3. Two impact measures are monitored—monthly sales and time to first sale. Although the upgrade is released to all sales associates at the same time, not all the sales associates were using the mobile learning program. This provided an opportunity to compare a user group to a nonuser group (experimental versus control group). The profit margin for the upgrade was used to convert to money. The profit of the first sale was not included since it was included in the total sales of the upgrade.

Planning in detail what you are going to ask, how you are going to ask, who you are going to ask, when you are going to ask, and who will do the asking, along with the key steps in the ROI analysis will help ensure successful execution. Additionally, having clients sign off on the plans will ensure support for the evaluation approach when results are presented. These planning documents (Figures 3-2 and 3-3) are explained in more detail in chapter 11.

Figure 3-2. Completed Data Collection Plan

Program: Product Upgrade With Mobile Learning Responsibility: Date:


Broad Program Objective(s)

Measures of Success

Data Collection Method/ Instruments

Data Sources





Achieve positive reaction on:

• Relevance to my work

• Recommend to others

• Important to my success

• Intent to use

Rating of 4 out of 5 on a composite of four measures

LMS survey, built into program


End of program

Program manager



Learn to use five concepts to sell new upgrade:

• Rationale for upgrade

• Features of upgrade

• How upgrade will increase client profit

• Pricing options

• Implementation and support

Achieve 4 out of 5 correct answers on each module

Achieve 20 of 25 total correct answers

True/False quiz


End of program

Program manager



Use of five skills:

• Explain rationale for upgrade

• Identify key features of upgrade

• Describe how upgrade increases client profit

• Identify pricing options

• Explain implementation and support

• Make the first call in 5 days

Rating (4 of 5) on a

1-5 scale System check

Questionnaire, webbased

Performance monitoring


Salesforce. com

1 month after program

1 month after program




• Increase in sales to $10,000 per month

• Sell first upgrade in 3 weeks

Monthly sales per associate

Actual sale

Business performance monitoring

Business performance monitoring

Salesforce. com

Salesforce. com


months after program 1 month after program



ROI 30%


Table 3-7. ROI Analysis Plan Key Areas and Description

Key Area


Methods for isolating the effects of the program

Decide the technique you plan to use to isolate the effects of the program on the impact measures.

Methods for converting data to monetary value

Identify the methods to convert impact measures to monetary value.

Cost categories

Include the cost of needs assessment, program design and development, program delivery, evaluation costs, along with some amount representative of overhead and administrative costs for those people and processes that support programs.

Intangible benefits

List those measures you choose not to convert to monetary value are considered intangible benefits.

Communication targets for the results

Identify those audiences to whom results will be communicated.

Other issues that may influence the impact or the evaluation itself

Anticipate any issues that may occur during the learning process that might have a negative effect or no effect on impact measures.

Comments or reminders to the staff managing the program

Place reminder notes of key issues, comments regarding potential success or failure of the program, reminders for specific tasks to be conducted by the evaluation team, etc.

< Prev   CONTENTS   Next >