Data Analysis^[1]

The data collected during the data collection phase must be analyzed. When using the ROI Methodology, data analysis should include the isolation of program effects on the data, calculation of fully loaded program costs, conversion of data to monetary values when appropriate, and the ROI calculation, if needed. This chapter provides a brief overview of each of these issues.

ISOLATING THE EFFECTS OF LEARNING

Isolating the effects of a program on business impact data is one of the most challenging, yet necessary, steps in the ROI Methodology. When addressed credibly, this step links learning through technology directly to business impact.

Other Factors Are Always Present

In almost every situation, multiple factors affect business results. During the time that programs are implemented, many other functions within the organization may be attempting to improve the same metrics addressed by the program. For example, marketing projects are designed to improve sales; and while a technology-based learning program should help a sales professional positively affect sales, the marketing project should also be positively affecting sales. In addition to internal factors, external factors may be affecting business results during the time that the program is occurring.

Without Isolation, There Is Only Evidence, Not Proof

Without taking steps to show the contribution of learning, there is no clear business linkage; instead, there is only evidence that learning may have made a difference. When business results improve during the learning program, it is possible that other factors may have contributed to that improvement. The proof that the program has made a difference on the business comes from isolating the effects of the program or initiative.

Other Factors Have Protective Owners

The owners of the other functions that influence business results are convinced that their processes or programs make the difference. Other processes or programs, such as advertisements, events, reward systems, and job design all have protective owners, and their arguments are plausible. Therefore, program owners are under pressure to build a credible argument for their case to claim value-add to the organization.

To Do It Right Is Not Always Easy

The challenge of isolating the effects of the learning program on the business is critical and can be done; but it is not easy for complex programs, especially when strongwilled owners of other processes are involved. It takes a determination to address this situation every time an ROI study is conducted. Fortunately, a variety of approaches are available.

TECHNIQUES TO ISOLATE THE EFFECTS OF LEARNING

Before reviewing the specific methods, it is helpful to highlight two important issues. First, although it is possible to isolate for the effect of the program at Level 3 application, isolation is usually applied to Level 4 impact. The business impact connected to the program is the key issue; when calculating ROI, the improvement in business measures is reported in monetary terms, and more than any other level, it must be credible. After the business impact data have been collected, the next step is to isolate the effects of the program. This step demonstrates the proof that the program made a difference, whereas reporting results along the chain of impact only presents evidence of the connection between the program and business results.

Another important issue is to attempt to identify the other factors that have contributed to the improvement in the business results measures. This step recognizes that other factors are almost always present and that the credit for improvement is shared with other functions in the organization. Just taking this step is likely to gain respect from the management team.

Several potential sources can help identify these influencing factors. The sponsors of the project may be able to identify the factors. Subject matter experts, process owners, and those who are most familiar with the situation may be able to indicate what has changed to influence the results. In many situations, participants know what other factors have actually influenced their performance. After all, it is their direct performance that is being measured and monitored.

By taking stock in this issue, all factors that contributed to improvement are revealed, indicating the seriousness of the issue and underscoring how difficult it is going to be to isolate the effects of the programs. A variety of techniques can help address the isolation issue, which are discussed next.

• [1] For more detail on this methodology, see The Value of Learning: How Organizations Capture Value and ROI and Translate Them Into Support, Improvement, Funds (Phillips and Phillips, 2007, Pfeiffer).