Categories/Levels of Data

Corresponding to the ROI Methodology, the categories, or levels, of data included those listed in Figure 7-1. Level 1 reaction data measured participant satisfaction and planned actions for implementing the learning. The organizational need was for the participants to perceive that the learning was relevant and important to successful job performance, and plan to use the learning on the job. The program objectives, in this case, included a mean rating of 4 out of 5 points for content relevance and importance, based on participant reaction surveys, and 80 percent of participants' identifying planned actions.

Level 2 learning measures participant acquisition of knowledge and skills, as well as changes in attitude. The need was for the immediate managers to learn how to effectively apply the five skills on the job, as determined by facilitator assessments of participant discussions, responses to questions, and completed assignments. Also, the facilitator administered preand post-program self-assessment profiles to gauge the participants' perceptions of changes in key behaviors related to the five skills. UWES surveys of the production unit direct reports were taken before and at day 45 and 90 of the initiative and compared to those of maintenance unit workers.

Level 3 application measured on-the-job use of the skills taught in the program. The organizational need was for the immediate managers to consistently and effectively apply the five skills, measured through participant self-assessments taken three months after the program. UWES data of production and maintenance direct reports also was taken at three months to assess changes in work engagement.

Level 4 impact measures changes in business impact. The organizational need was to reduce the percentage of controllable waste and rework generated by the production unit. Monthly percentages of controllable rework and waste were used to determine whether program objectives had been met. Also, one last UWES comparison of direct reports was taken six months after the program.

The Level 5 ROI calculation compared the program's benefits to its costs. In this case, a conservative 15 percent target ROI for the reduction of the production units' controllable rework and waste was established.

Data Collection Strategy

The data collection plan in Figure 7-2 shows the level of evaluation, broad program objectives, measures/data collected, collection methods, data sources, timing, and responsibility. Level 1 reaction data collected at the end of each session gauged the participants' perceptions of the program and their intent to apply what they had learned. In addition to Level 2 participant preand post-self-assessment profiles and facilitator appraisals of learning, the evaluation used work engagement data collected from the production business unit during the program, and compared it to the work engagement of the maintenance unit. Participant self-assessment profiles and work engagement data taken three months after the program gauged the Level 3 on-the-job application of participant skills. Level 4 data included work engagement measurements taken six months after the initiative and the percentage of controllable waste and rework generated by production at eight months. Level 5 ROI was a calculation of net program benefits over costs.

ROI Analysis Strategy

The ROI analysis for this project, shown in Figure 7-3, depended on tracking the percent of controllable waste and rework for the production business unit before, during, and after the learning initiative. Monetary values were calculated directly, based on the percentage of total product waste and rework generated each month. The researcher used two methods to isolate the effects of the program on controllable waste and rework. The strategy called for trend analysis of the monthly percent of controllable waste and rework per total product, less those outliers identified by PolyWrighton management as nonattributable to the production business unit. Management and participant estimates of the impact of the program and the level of confidence in those estimates were also taken and adjusted. Work engagement was not converted to a monetary value, but was listed as an intangible benefit. Fully loaded costs were calculated and verified by management to ensure the most conservative ROI possible.

Isolation Techniques

Participant and management estimates of the impact of the initiative on business results, corrected for estimate error, were used in conjunction with trend analyses of controllable waste and rework. Outliers identified by PolyWrighton management as nonattributable to the production business unit were removed from the trend analyses.

Data Conversion Techniques

The conversion from percent controllable waste and rework to monetary values was direct. As previously noted, it costs PolyWrighton $35,000 for 1 percent of product rework and $245,000 for 1 percent of product waste. Data not converted to monetary values, including work engagement, were listed as intangible benefits.

Cost Summary

The program cost categories shown in Figure 7-3 included the consulting fees for the learning needs assessment, program design and development, learning delivery, and program evaluation. Printing and supplies, participant salaries, facilities, and travel were also planned costs of the program. In actuality, all fees and expenses were waived, leaving only the cost of participant salaries for PolyWrighton to bear. However, in order to provide the most conservative ROI figure for decision makers, all costs were included in the calculation.

Figure 7-2. Data Collection Plan

Program: PolyWrighton Work Engagement Program




Broad Program Objectives


Data Collection Methods

Data Sources



1 Reaction

• Program content receives favorable ratings from participants

• Participants plan to apply the learning on the job

• Program content receives average favorable rating of 4 out of 5 for relevance and importance

• 80% of participants identify planned actions

• Questionnaires

• Participants

• End of each session during the program

• Facilitator

2 Learning

• Learn to effectively apply the five selfcoaching skills of selfmanaging, reflecting, acting consciously, collaborating, and evolving

• Learn to foster motivational work environments that increase engagement

• Participants demonstrate successful completion of program learning objectives outlined in the Facilitator and Participant Guides

• Self-assessment

• Work engagement

• Observations of performance, guided discussions, questioning, and assignments

• Skill development journals

• Pre-/post-selfassessment profile

• Utrecht Work Engagement Scale (UWES)

• Facilitator

• Participants

• Participants

• Direct reports of participants

• Throughout the 90-day learning program

• Day 1, Day 90

• Day 1, Day 45, Day 90

• Facilitator

3 Application

• Apply the five selfcoaching skills at work

• Foster motivational work environments that increase engagement

• Self-assessment

• Work engagement

• Questionnaires


• Participants

• Direct reports

• 3 months after program completion

• Program manager

4 Impact

• Reduce product waste and rework

• Increase work engagement

• Percent controllable waste and rework generated by the production business unit

• Work engagement

• Organizational records/databases


• Business unit manager

• Direct reports

• 8 months after program compared to preprogram

• 6 months

• Program manager



• Target ROI 15%

• Comments: ROI = (Net Program Benefits ÷ Program Costs) x 100

< Prev   CONTENTS   Next >