In this section, the program that has undergone a rigorous evaluation will shine! Provide the results for all levels of evaluation, beginning with Level 1, reaction and planned action. Explain the intent for gathering reaction data, providing the specific questions the reaction data answers, and report the results. Then move on to Level 2, learning. Explain why it's important to evaluate learning and the key questions that learning data answers and report the results. Next, move on to Level 3, application and implementation. This is one of the greatest parts of the story. Provide evidence that what was taught was used. Discuss how frequently and effectively knowledge and skills gained in the program have been applied by the participants. Discuss how the support system enabled participants to apply what they learned. Discuss the barriers to the transfer of learning gained, to behavior change, or implementation and application. It is important to explain what happened. For example, if the work environment did not support learning transfer, report that here. Also explain that when it was recognized (through the evaluation process) that a problem was occurring (the support system was not helping), that action was taken by talking with those who might know or who might provide information about how things could be changed to support the program next time.
Next, discuss Level 4, business impact, including how the program positively influenced specific business outcomes. Reinforce the fact that the effects of the program were isolated; it must be clear to the audience that other influences that might have contributed to these outcomes were taken into account. Describe the options for isolation and explain why those options were chosen.
Then, report on Level 5, ROI. First, explain what is meant by ROI, clearly defining the ROI equation. Address the benefits of the program, the Level 4 measures, and how they were achieved. Explain how the data were converted to monetary value and detail the monetary benefits of the program. Then, report the fully loaded costs. Recall that earlier in the evaluation methodology section of the report, the cost items were detailed, but a dollar value was not identified. It is here, after monetary benefits are reported, where the dollar values of the costs are outlined. The readers have already seen the benefits in dollar amounts; now provide the costs. The pain of a very expensive program is relieved because the audience can clearly see that the benefits outweigh the costs. Finally, provide the ROI calculation.
The last section in the detailed report concerns intangible benefits, which are those items that are not converted to monetary value. Highlight those intangible benefits and the unplanned benefits that came about through the program. Reinforce their importance and the value they represent.
Conclusions and Next Steps
Develop and report the program conclusions based on the evaluation, answering these questions:
O Was the program successful?
O What needs to be improved?
Explain the next steps, clearly pointing out the next actions to be taken with regard to the program. Those actions could include continuing the program, adding a different focus, removing elements of the program, changing the format, or developing a blended learning approach to reduce the costs while maintaining the benefits achieved. Clearly identify the next steps and set out the dates by which these steps will be completed.
The appendices include exhibits, detailed tables that could not feasibly be included in the text, and raw data (keeping the data items confidential). The final report is a reference for readers as well as a story of success for others.
Throughout the report, incorporate quotes—positive and negative—from respondents. While it is tempting to leave out negative comments, ethically, they should not be omitted and including them enhances the credibility and respect for the report. By developing a detailed comprehensive report, there will be a backup for anything communicated during a presentation. When conducting a future ROI study on a similar program, the road map is now clear. Table 5-2 presents a sample outline of a detailed report.
Table 5-2. Impact Study Outline for Detailed Report
Another important report is the executive summary. The executive summary follows the same outline as the detailed report (although it omits the appendices), and each section and subsection is not developed in such great detail. Clearly and concisely explain the need for the program, the need for the evaluation, and the evaluation methodology. Always include the ROI Methodology prior to the results so that the reader understands and appreciates it. The understanding and appreciation build credibility and respect for the results. Report the data from Level 1 through Level 5 and include the sixth measure of success—the intangible benefits. The executive summary is usually 10 to 15 pages long.
General Audience Reports
General audience reports are a great way to describe the success of programs to employees. General audience reports may be published in organization publications, like newsletters or in-house magazines; reported in management and team meetings, where a brief review of the report can be communicated in a meeting setting; and, finally, published as case studies. Case studies can be published internally and externally. There are many opportunities to publish the case study outside the organization, including trade or association publications or academic research publications. The key is to tell the story to show that the programs are working, and that when they don't work, steps are taken to improve them.
A final micro-level report is a single-page report. Success of a program should not be communicated using the single-page report until after the audience understands the methodology. If an audience sees the ROI of a program without having an appreciation for the methodology used to arrive at the number, the audience will fixate on the ROI and never notice, much less form a regard for, information developed in the other levels of evaluation. Therefore, single-page reports are used with great care, but they are an easy way to communicate results to the appropriate audiences on a routine basis.
Macro-level scorecards can provide the results of the overall impact of learning and development programs, such as technology-based learning programs. These scorecards provide a macro-level perspective of success and serve as a brief description of a program evaluation in contrast to the detailed report. They show the connection between the program's contribution and the business objectives. The method of isolation is always included on the report to reinforce that credit is given where credit is due. The scorecard integrates a variety of types of data and demonstrates alignment between programs, strategic objectives, and operational goals.