Measures to evaluate the effectiveness of a program can be designed and tracked through five distinct levels, as shown in Table 12-1. In addition to the five levels of data illustrated in this table, intangible benefits are reported for important outcomes that cannot be converted to monetary values.

The executive vice president of international sales requested that the return on investment (Level 5) be calculated for this program because of the high cost and potential business impact of the TLC program. Therefore, it became necessary to analyze data at the five levels, plus any intangible benefits.

Table 12-1. Five Levels of Data

Level and Type of Measure

Measurement Focus

Level 1: Reaction/Planned Action

Measures participant satisfaction and captures planned actions

Level 2: Learning

Measures changes in knowledge, skills, and attitudes

Level 3: Application

Measures changes in on-the-job actions and behavior

Level 4: Business Impact

Measures changes in business impact variables

Level 5: Return on Investment

Compares program benefits with the costs

ROI Model and Process

Executive management expressed concern that the process used to evaluate TLC was a credible process. Figure 2-3 in chapter 2 illustrates the ROI Process Model used to address this concern. This process has been applied in virtually every industry and in numerous international settings to evaluate training programs, HR programs, technology initiatives, and performance improvement programs. The process flows sequentially from step to step until the ROI is developed. The impact study captures both Level 3 (application) and Level 4 (business impact) data. The ROI (Level 5) is developed from Level 4 data. Improvements that cannot be converted to monetary values are reported as intangible benefits. A conservative approach is used to ensure that only benefits that can be legitimately attributed to the program are captured.

The Data Collection Plan and ROI Analysis Plan

After the business measures were determined and the framework for the TLC training program was known, the data collection plan focusing on Level 3 (application) and Level 4 (business impact) measures was developed. The Level 3 measures were behavior changes and frequency of application linked to the TLC program objectives. After exploring performance data availability in the ISO unit, the quality of the specific data, and the perceived linkage to the TLC program, the Level 4 measures were targeted and included in the data collection plan, which is presented in Figure 12-1.

The Level 4 data items were then transferred to the ROI analysis plan so that the planning process could be completed. The methods for isolation, data conversion, cost categories, and other items were determined and documented on the ROI analysis plan, which is presented in Figure 12-2.

Level 1 and 2 Data

Level 1 data were captured through an online feedback questionnaire that assessed course content, usefulness of the TLC program, and job applicability. Participants rated questions on a Likert-type scale from 1 to 7. Participant average for the overall course content was 6.6, and overall usefulness of the system was 6.5. Applicability of the course to the job was rated 6.8. Level 1 data are consolidated in the first three columns of Table 12-2.

Level 2 data were assessed using preand post-testing. The preand post-testing for TLC was designed based on job performance expectations. Subject matter experts (SMEs) determined the testing components, which were then validated by sales managers.

The SMEs, working with the program designers, validated program content based on competency requirements and skill gaps of the sales organization. They also provided input to design preand post-tests. Pretests were administered electronically at the beginning of each learning module to determine individual knowledge and skill gaps. The results showed that participants averaged a 50 percent knowledge level on the pretest and averaged a 91 percent knowledge level on the post-test. These Level 2 data are consolidated in the last two columns of Table 12-2.

Figure 12-1. Data Collection Plan: United Petroleum International

Program: Technology Learning Competency (TLC) Responsibility: Date:




Data Collection Method

Data Sources



Level I Reaction and Satisfaction

Employee positive reaction to:

• Appropriateness of the technology delivery program

• Usefulness of the TLC

• TLC application to the job

• Participants' perception and attitude

• Online questionnaire

• Participant

• End of each segment (3-5 modules)

• End of program

• Program coordinator

Level II Learning

Module learning assignment based on knowledge/skill gaps:

• Client partnerships

• Product pricing & contracting

• Identification & handling objections

• Skill gaps identified

• Learning occurs as gaps are closed through each module implemented

• Online pretest questionnaire on all modules

• One post-test by module

• Participant

• Prior to training to establish baseline

• Prior to each module as required

• At end of each module

• Program coordinator

Level III Application & Implementation

• Review post-course report and participate in follow-up planning meeting with managers

• Application of skills to achieve business goals

• Goals set and achieved

• Skills applied in sales planning and sales situations

• Follow-up questionnaire

• Participants

• Managers

• Coaching and planning session within two weeks of TLC

• Follow-up questionnaire four months after TLC

• Program coordinator initiates followup

• Manger and participants initiate planning and coaching

Level IV Business Impact

• Improve closing ratio

• Increased revenue

• Customer satisfaction

• Increase in monthly closes

• Increase in profit margin

• Customer satisfaction index

• Performance monitoring

• Performance monitoring

• Customer survey (existing)

• Sales record

– marketing

• Sales record

– Marketing

• Customer quarterly survey

• Monthly

• Monthly

• Monthly

• Program coordinator

Level V ROI

Because of strict requirement for development costs (see comments) an ROI at 20% will be acceptable.

Comments: Because training will be completed for all current engineers within the first year of roll-out, management desires to achieve a return on investment during the first year. Therefore, development costs will not be prorated over the life of the program as is customary.

< Prev   CONTENTS   Next >