THE ROI METHODOLOGY

The ROI Methodology is comprised of five key elements that work together to complete the evaluation puzzle. Figure 2-1 illustrates how these elements are interconnected to create a comprehensive evaluation system.

Figure 2-1. Key Elements of an Evaluation System

Evaluation Framework

The system begins with the five-level ROI framework, developed in the 1970s and becoming prominent in the 1980s. Today, this framework is used to categorize results for all types of programs and projects.

O Level 1 Reaction and Planned Action data represent the reactions to the program and the planned actions from participants. Reactions may include views of the format, ease of use, convenience, and fit. This category must include data that reflect the value of the program content, including measures of relevance, importance, amount of new information, and participants' willingness to recommend the program to others.

O Level 2 Learning data represent the extent to which participants have acquired new knowledge about their strengths, development areas, and skills needed to be successful. This category also includes the level of confidence of participants as they plan to apply their newly acquired knowledge and skills on the job.

O Level 3 Application and Implementation data determine the extent to which professionals apply their newly acquired knowledge and skills from the learning program. This category of data also includes data that describe the barriers preventing application, as well as any supporting elements (enablers) in the knowledge and skill transfer process.

O Level 4 Business Impact data are collected and analyzed to determine the extent to which applications of acquired knowledge and skills positively influenced key measures that were intended to improve as a result of the learning experience. The measures include errors, rejects, new accounts, customer complaints, sales, customer returns, down time, cycle time, job engagement, compliance, absenteeism, and operating costs. When reporting data at this level, a step to isolate the program's effect on these measures is always taken.

O Level 5 Return on Investment compares the monetary benefits of the impact measures (as they are converted to monetary value) to the fully loaded program costs. Improvement can occur in sales, for example, but to calculate the ROI, the measure of improvement must be converted to monetary value (profit of the sale) and compared to the cost of the program. If the monetary value of sales improvement exceeds the costs, the calculation is a positive ROI.

Each level of evaluation answers basic questions regarding the success of the program. Table 2-1 presents these questions.

Table 2-1. Evaluation Framework and Key Questions

Source: ROI Institute, Inc.

Categorizing evaluation data as levels provides a clear and understandable framework to manage the technology-based learning design and objectives and manage the data collection process. More importantly, however, these five levels present data in a way that makes it easy for the audience to understand the results reported for the program. While each level of evaluation provides important, stand-alone data, when reported together, the five-level ROI framework represents data that tell the complete story of program success or failure. Figure 2-2 presents the chain of impact that occurs as participants react positively to the program; acquire new knowledge, skills, and awareness; apply the new knowledge, skills, and awareness; and, as a consequence, positively affect key business measures. When these measures are converted to monetary value and compared to the fully loaded costs, an ROI is calculated. Along with the ROI and the four other categories of data, intangible benefits are reported. These represent Level 4 measures that are not converted to monetary value.