ROI Process Model

The second part of the evaluation puzzle is the process model. As presented in Figure 2-3, the process model is a step-by-step guide to ensure a consistent approach to evaluating a learning project. Each phase of the four-phase process contains critical steps that must be taken to ensure the output of a credible evaluation. The ROI process is described in more detail in the next section.

Figure 2-2. Chain of Impact Reaction

Source: Phillips, P.P., and J.J. Phillips. (2005). Return on Investment Basics. Alexandria, VA: ASTD Press.

Operating Standards and Philosophy

The third piece of the evaluation puzzle ensures consistent decision making around the application of the ROI Methodology. These standards, called the 12 Guiding Principles of the ROI Methodology, provide clear guidance about the specific ways to implement the methodology to ensure consistent, reliable practice in evaluating learning through technology. When these guiding principles (shown in Table 2-2) are followed, consistent results can be achieved. Additionally, these principles help maintain a conservative and credible approach to data collection and analysis. They serve as a decision-making tool and influence decisions on the best approach by which to collect data, the best source and timing for data collection, the most appropriate approach for isolation and data conversion, the costs to be included, and the stakeholders to whom results are reported. Adhering to the principles provides credibility when reporting results to executives.

Figure 2-3. The ROI Process Model

Source: © ROI Institute. All rights reserved.

Table 2-2. 12 Guiding Principles for Effective ROI Implementation

1. When a higher level of evaluation is conducted, data must be collected at lower levels.

2. When an evaluation is planned for a higher level, the previous level of evaluation does not have to be comprehensive.

3. When collecting and analyzing data, use only the most credible sources.

4. When analyzing data, choose the most conservative alternatives for calculations.

5. At least one method must be used to isolate the effects of the solution/program.

6. If no improvement data are available for a population or from a specific source, it is assumed that no improvement has occurred.

7. Estimates of improvements should be adjusted for the potential error of the estimate.

8. Extreme data items and unsupported claims should not be used in ROI calculations.

9. Only the first year of benefits (annual) should be used in the ROI analysis for short-term solutions/programs.

10. Costs of the solution/program should be fully loaded for ROI analysis.

11. Intangible measures are defined as measures that are purposely not converted to monetary values.

12. The results from the ROI Methodology must be communicated to all key stakeholders.

Source: ROI Institute, Inc.

Case Applications and Practice

The fourth piece of the ROI Methodology evaluation puzzle includes case applications and practice, which allow for a deeper understanding of the ROI Methodology's comprehensive evaluation process. Case studies are a way to provide evidence of a program's success. Thousands of case studies across many industries, including business and industry, healthcare, government, and even community and faith-based initiatives, have been developed, describing the application of the ROI Methodology. The case studies in this book, all based on measuring the ROI in learning through technology, provide excellent examples of application of the ROI Methodology.

Practitioners beginning their pursuit of the ROI Methodology can learn from these case studies, as well as those found in other publications. However, the best learning comes from actual application. Conducting an ROI study around learning through technology will allow participants to see how the framework, process model, and operating standards come together. The first study serves as a starting line for a track record of program success.


The last piece of the ROI Methodology evaluation puzzle is implementation. While it is significant to conduct an ROI study, one study alone adds little value to your efforts to continuously improve and account for learning investments. The key to a successful learning function is to sustain the use of ROI. Building the philosophy of the ROI Methodology into everyday decision making is imperative for attaining credibility and consistency in learning effectiveness. Implementing the ROI Methodology requires assessing the organization's culture for accountability and its readiness for evaluating technology-based learning programs at the ROI level. It also requires defining the purpose for pursuing this level of evaluation; building expertise and capability; and creating tools, templates, and standard processes.

< Prev   CONTENTS   Next >