EVALUATION PLAN AND OBJECTIVES
The learning management system (LMS) was the primary vehicle used to launch, score, and track the modules and evaluations. At the close of each module, the learner was triggered to automatically complete a Level 1 evaluation. Figure 9-1 presents the full data collection plan.
Figure 9-1. Data Collection Plan
|
Evaluation Purpose: To demonstrate monetary benefits from impact of sales academy on sales performance Program: Online Sales Academy Responsibility: Consultant Date: March 3 |
||||||
|
Level |
Broad Program Objective(s) |
Measures |
Data Collection Method/Instruments |
Data Sources |
Timing |
Responsibilities |
|
1 |
Reaction, Satisfaction, and Planned Action |
Item #6 I am generally satisfied with this course |
Course evaluation hosted on online assessment Website. Distribution method: email to participant with hyperlink to website with course evaluation. |
Online assessment database |
Upon completion of course |
Facilitator |
|
2 |
Learning |
85% correct on mastery learning check |
Mastery learning checks hosted on online assessment website. |
LMS tracking system |
Before and after course |
Facilitator |
|
3 |
Application/ Implementation |
Increased skill assessment ratings on sales competencies |
Skill gap assessment launched through LMS; Automated through LMS |
LMS competency management system |
Before course and 90 days post online training |
Monitored by consultant |
|
4 |
Business Impact |
Increased new accounts; Increased $$ made from sales |
Database fields for number of new accounts and dollars from sales |
Sales workforce database |
180 days posttraining |
Consultant |
|
5 |
ROI |
Baseline Data: 5,000 new accounts opened $15,000,000 sold in those new accounts Comments: |
||||
The goal for satisfaction was set for an average rating of 90 percent. Mastery learning checks (Level 2) were embedded in the design of the e-learning modules. The learner was required to complete the mastery checks before completing the module. A goal of 85 percent correct was set for the mastery checks. Preand post-training skill-gap assessments (Level 3) were administered online to measure and track the level of sales competencies among the sales associates. A minimum score for the posttraining skill assessment, administered three months after training, was set for 80 percent. Other performance and business impact measures, which were tracked in an online sales workforce database, included the following:
O ability to contact 10 new prospects and conduct needs assessment on those prospects within a week after attending the sales academy
O number of sales proposals that the sales associates generated based on the assessments within 30 days
O number of new accounts opened
O weekly amount of dollars earned from sales in the first four weeks after sales academy
O monthly amount of dollars earned from sales thereafter O number of customers retained out of total customer load O dollar amount gained from new accounts
O reduction of time spent in training
O amount gained from cross-selling products (new initiative).
Figure 9-2 presents the ROI analysis plan. To isolate the impact of the e-learning program, participants and their managers were asked to estimate the impact of the e-learning program on the business results. Confidence levels for estimates were used to adjust for any indecision.
Program Cost
The total cost of the former sales academy was $18,890,800. This included analysis, design, marketing, delivery, evaluation, and overhead costs. Each sales associate already had an assigned personal laptop, and the company had already purchased an LMS; nevertheless, a proportionate amount of the hardware and software was allocated to this project. The costs of the program are illustrated in Table 9-1.
Figure 9-2. ROI Analysis Plan
|
Data Items (Usually Level 4) |
Methods for Isolating the Effects of the Program/ Process |
Methods of Converting Data to Monetary Values |
Cost Categories |
Intangible Benefits |
Communication Targets for Final Report |
Other Influences/ Issues During Application |
Comments |
|
Number of new accounts generated |
Control group |
Net profit from sales Cost of online sales academy Cost of lost productivity |
Customer satisfaction |
• Senior vice president of sales • Vice president of sales • Vice president of sales academy • Sales academy project work team • Company's online newsletter • LMS (messaging for registrants) |
• New product launch • Call center undergoing customer service training, which could affect customer satisfaction |
||
|
Dollars made from sales |
Control group |
Dollars made from sales in both groups preand posttraining; convert gross sales to net |
Table 9-1. E-Learning Cost Tabulation Worksheet
|
Item |
Itemized Cost |
Total Cost |
|
Upfront Costs |
||
|
Servers (to accommodate learning technology) |
$10,000 |
$10,000 |
|
Software (authoring software, LMS, survey software, virtual classroom setup): depreciation rate per year x number of years |
$60,000 |
$60,000 |
|
Hardware (PCs) |
$18,000 |
$18,000 |
|
Total Upfront Costs |
$88,000 |
|
|
Recurring Technology Costs |
||
|
Annual software maintenance |
$500 |
$500 |
|
Upgrades for software |
$300 |
$300 |
|
Total Recurring Technology Costs |
$800 |
|
|
Analysis Costs |
||
|
# of employees x average salary x benefits x # of hours on project |
$6,500 |
$6,500 |
|
Meals, travel, and incidental expenses |
||
|
Office supplies and expenses |
||
|
Outside services |
$2,000 |
$2,000 |
|
Equipment expense |
||
|
Other miscellaneous expenses |
||
|
Total Analysis Costs |
$8,500 |
|
|
Development Costs |
||
|
# of employees x average salary x benefit rate x # of hours on project |
$6,500 |
$6,500 |
|
Program materials and supplies |
||
|
CDs/diskettes |
||
|
Artwork/graphics |
||
|
Other |
||
|
Outsourced services |
$300,000 |
$300,000 |
|
Internal services (for example, information technology staff), including salaries and charge-backs for services: # of employees x average salary x benefit rate x # of hours on project OR amount billed by department |
$7,500 |
$7,500 |
|
Registration fees |
||
|
Other miscellaneous expenses |
$5,500 |
$5,500 |
|
Testing (alpha and beta testing): # of testers x average salary x benefit rate x # of hours on project |
$5,500 |
$5,500 |
|
Total Development Costs |
$325,000 |
|
Marketing Costs |
||
|
Marketing staff: # of marketing employees x average salary x benefit rate x # of hours on project |
$1,000 |
$1,000 |
|
Meals, travel, and expenses |
||
|
Office supplies |
||
|
Printing and reproduction |
||
|
Outsourced services |
||
|
Internal services |
||
|
Equipment expense |
$150 |
$150 |
|
Hardware expense |
$175 |
$175 |
|
Software expense |
$175 |
$175 |
|
Miscellaneous expenses |
||
|
Total Marketing Costs |
$1,500 |
|
|
Delivery Costs |
||
|
Participants' time in training: # of employees x average salary x benefit rate x # of hours of training time (tracked by either timestamp actual or average) |
$4,557,000 |
$4,557,000 |
|
Lost production (explain basis) |
||
|
Program materials and supplies, if required |
||
|
Instructor costs for synchronous learning |
||
|
Instructors' salaries and benefits |
||
|
Meals and travel expenses for synchronous learning, if applicable |
||
|
Outside services |
||
|
Facility/rental costs (synchronous, satellite studio, distance learning lab) |
||
|
Facilities expense allocation |
||
|
Hardware expense |
||
|
Software expense |
||
|
Miscellaneous expenses |
||
|
Total Delivery Costs |
$4,557,000 |
|
|
Evaluation Costs |
||
|
# of employees x average salary x benefit rate x # of hours on project |
$6,800 |
$6,800 |
|
Meals, travel, and incidental expenses |
||
|
Participants' costs for interviews, focus groups, surveys, and so forth |
||
|
Office supplies and expenses |
||
|
Printing and reproduction |
||
|
Internal services |
||
|
Outsourced services |
$8,000 |
$8,000 |
|
Hardware expense |
$100 |
$100 |
|
Software expense |
$100 |
$100 |
|
Other miscellaneous expenses |
||
|
Total Evaluation Costs for Program/Project |
$15,000 |
|
|
TOTAL PROGRAM/PROJECT COSTS |
$4,995,800 |
Source: Adapted from Phillips, J.J. (1997). Return on Investment in Training and Performance Improvement Programs. Boston: Butterworth-Heinemann.
