The ROI Methodology is comprised of five key elements that work together to complete the evaluation puzzle. Figure 2-1 illustrates how these elements are interconnected to create a comprehensive evaluation system.

Figure 2-1. Key Elements of an Evaluation System

Evaluation Framework

The system begins with the five-level ROI framework, developed in the 1970s and becoming prominent in the 1980s. Today, this framework is used to categorize results for all types of programs and projects.

O Level 1 Reaction and Planned Action data represent the reactions to the program and the planned actions from participants. Reactions may include views of the format, ease of use, convenience, and fit. This category must include data that reflect the value of the program content, including measures of relevance, importance, amount of new information, and participants' willingness to recommend the program to others.

O Level 2 Learning data represent the extent to which participants have acquired new knowledge about their strengths, development areas, and skills needed to be successful. This category also includes the level of confidence of participants as they plan to apply their newly acquired knowledge and skills on the job.

O Level 3 Application and Implementation data determine the extent to which professionals apply their newly acquired knowledge and skills from the learning program. This category of data also includes data that describe the barriers preventing application, as well as any supporting elements (enablers) in the knowledge and skill transfer process.

O Level 4 Business Impact data are collected and analyzed to determine the extent to which applications of acquired knowledge and skills positively influenced key measures that were intended to improve as a result of the learning experience. The measures include errors, rejects, new accounts, customer complaints, sales, customer returns, down time, cycle time, job engagement, compliance, absenteeism, and operating costs. When reporting data at this level, a step to isolate the program's effect on these measures is always taken.

O Level 5 Return on Investment compares the monetary benefits of the impact measures (as they are converted to monetary value) to the fully loaded program costs. Improvement can occur in sales, for example, but to calculate the ROI, the measure of improvement must be converted to monetary value (profit of the sale) and compared to the cost of the program. If the monetary value of sales improvement exceeds the costs, the calculation is a positive ROI.

Each level of evaluation answers basic questions regarding the success of the program. Table 2-1 presents these questions.

Table 2-1. Evaluation Framework and Key Questions

Level of Evaluation

Key Questions

Level 1: Reaction and Planned Action

• Was the learning relevant to the job and role?

• Was the learning important to the job and success of the participant?

• Did the learning provide the participant with new information?

• Do participants intend to use what they learned?

• Would they recommend the program or process to others?

• Is there room for improvement in duration and format?

Level 2: Learning

• Did participants gain the knowledge and skills identified at the start of the program?

• Do participants know how to apply what they learned?

• Are participants confident to apply what they learned?

Level 3: Application and Implementation

• How effectively are participants applying what they learned?

• How frequently are participants applying what they learned?

• Are participants successful with applying what they have learned?

• If participants are applying what they learned, what is supporting them?

• If participants are not applying what they learned, why not?

Level 4: Business Impact

• So what if the application is successful—what impact will it have on the business?

• To what extent did application of knowledge and skills improve the business measures the program was intended to improve?

• How did the program affect sales, productivity, operating costs, cycle time, errors, rejects, job engagement, and other measures?

• How do you know it was the learning program that improved these measures?

Level 5: ROI

• Do the monetary benefits of the improvement in business impact measures outweigh the cost of the technology-based learning program?

Source: ROI Institute, Inc.

Categorizing evaluation data as levels provides a clear and understandable framework to manage the technology-based learning design and objectives and manage the data collection process. More importantly, however, these five levels present data in a way that makes it easy for the audience to understand the results reported for the program. While each level of evaluation provides important, stand-alone data, when reported together, the five-level ROI framework represents data that tell the complete story of program success or failure. Figure 2-2 presents the chain of impact that occurs as participants react positively to the program; acquire new knowledge, skills, and awareness; apply the new knowledge, skills, and awareness; and, as a consequence, positively affect key business measures. When these measures are converted to monetary value and compared to the fully loaded costs, an ROI is calculated. Along with the ROI and the four other categories of data, intangible benefits are reported. These represent Level 4 measures that are not converted to monetary value.

< Prev   CONTENTS   Next >