To evaluate a technology-based learning program using the ROI Methodology, it is important to follow the step-by-step process to ensure consistent, reliable results. These 10 steps taken during the four phases of an evaluation project make up the evaluation process model.
The first phase of a successful application of the ROI Methodology is planning. The plan addresses the key questions about the evaluation and defines how to know when success has been achieved. The plan begins with developing and reviewing the program objectives to ensure that the application and impact objectives have been defined. Next, the data collection plan is developed, which includes defining the measures for each level of evaluation, selecting the data collection instrument, identifying the source of the data, and determining the timing of data collection. The baseline data for the measures being tracked should be collected during this time. The next step is to develop the ROI analysis plan. Working from the impact data, the most appropriate technique to isolate the effects of the learning initiative is selected. The most credible method for converting data to money is identified along with the cost categories for the program. Intangible benefits are listed and the communication targets for the final report are identified.
Once the planning phase is complete, the data collection phase begins. Level 1 and 2 data are collected as learning takes place, using common instruments such as questionnaires, completion of exercises, demonstrations, and a variety of other techniques. Follow-up data at Levels 3 and 4 are collected after the program is complete, when application of the newly acquired knowledge, skills, attitudes, and awareness becomes routine. After the application the consequences are captured as impact measures.
Once the data are collected, data analysis begins. As described earlier, the method for data analysis is defined in the planning stage; so data analysis is just a matter of execution. The first step in data analysis is to isolate the effects of the learning program on impact data. Isolation is often overlooked when evaluating the success of technologybased learning programs, yet this step answers the critical question, “How much of the improvement in business measures is due to this particular learning program?”
Moving from Level 4 to Level 5 begins with converting Level 4 impact measures to monetary value. This step is usually easy because most of the important impact data are already converted to money. If not, there are some easy techniques to use. The fully loaded costs are developed during the data analysis phase. These costs include needs assessment (when conducted), design, participants' time, overhead, and evaluation costs.
Intangible benefits are also identified during the data analysis phase. Intangible benefits are the Level 4 measures that are not converted to monetary value. These measures can also represent any unplanned program benefits that were not identified during the planning phase.
The final step of the data analysis phase is the ROI calculation. Using simple addition, subtraction, multiplication, and division, the ROI is calculated.
The most important phase in the evaluation process is the final report. Evaluation without communication of results is a worthless endeavor. If the key stakeholders are not aware of the program's progress, it is difficult to improve the process, secure additional funding, and market programs to other participants.
While there are a variety of ways to report data, a micro-level report of the complete ROI impact study is important. This is a record of the success of the learning program. (A macro-level reporting process includes results for all programs, projects, and initiatives, and serves as a scorecard of results for all initiatives.) An important point to remember, however, is regardless of how detailed or brief the report may be, the information in it must be actionable. Otherwise, there is no value in conducting the ROI analysis.