PURPOSE OF EVALUATION

This study focused primarily on the production business unit's performance, as measured by the four participating immediate managers' reaction, learning, and application of the five skills taught during the program. Also evaluated were the work engagement levels of the production managers' 32 line employees compared to the control group's 31 line employees assigned to the maintenance business unit. Moreover, the study evaluated the impact and return on investment (ROI) of the initiative in terms of the production unit's controllable waste and rework. The program was evaluated to provide information for PolyWrighton decision makers considering whether to extend the blended learning program to the remaining 13 business units, to make improvements to the program, or to abandon the program altogether.


EVALUATION METHODOLOGY

The ROI Methodology was used to determine five levels of value, including participant reaction, learning, and on-the-job skills application. Also measured were business impact, intangible benefits, and return on investment in the production unit. That is, the evaluation focused primarily on the unit's performance, as measured by the four immediate managers' reaction, learning, and application of the five skills taught during the program. Using the extensively studied and validated Utrecht Work Engagement Scale (UWES), the evaluation also compared the work engagement of the production managers' 32 direct reports to the 31 direct reports assigned to the maintenance unit. The impact and ROI of the program were evaluated in terms of the production unit's controllable waste and rework.

Planning the initiative and its evaluation required a thorough needs assessment to ensure it aligned with organizational priorities. The results of the needs assessment are shown in Figure 7-1.

For the production unit, aligning the initiative with organizational needs meant increasing employee work engagement and reducing controllable product waste and rework. The production unit's work engagement was compared to the maintenance unit's work engagement using the UWES. In a quasi-experimental research design format, work engagement comparisons were generated by taking repeated UWES measurements of both production and maintenance units. Product quality was measured in terms of costs associated with the production unit's monthly percentages of controllable product waste and rework. Trend analysis and participant and management estimates were used to isolate the effects of the initiative. The fully loaded costs of the program were included in the ROI calculation to ensure the monetary benefits were not overstated. PolyWrighton provided standard values for converting controllable waste and rework data into monetary values. At PolyWrighton, rework and waste result in an additional $35,000 for every 1 percent of product rework and $245,000 for every 1 percent of waste per product produced. Data not converted to monetary values, including work engagement, were listed as intangible benefits.

The participating managers' on-the-job application, learning, and reaction data were also collected. The on-the-job application of participant skills was measured using immediate manager self-assessment surveys and UWES data collected from their direct reports after the program. During six of the seven sessions, learning was measured by the participants' summarizing how they practiced the previous session's content on-the-job, and by their completed assignments and skill development journal entries. Learning was also measured using a preand post-program skill assessment inventory, and by UWES data collected from the direct reports during the course of the program. Participant reaction data pertaining to program content relevance and importance, as well as the participants' planned implementation actions, were collected at the end of each of the seven learning sessions.

Figure 7-1. Business Alignment and Forecasting

Level

Needs Assessment

Program Objective

Measurement and Evaluation

5

Payoff Needs

Avoid costs associated with

controllable waste and rework

ROI Objectives

Target return on investment of 15%

ROI

Calculate ROI

4

Business Needs Reduce controllable waste and rework

Increase work engagement

Impact Objectives Monthly percentages of controllable product waste and rework decline

Increase work engagement

Impact

Percentages of controllable product Waste and rework at 8 months after

completion of the program compared to the same measurements taken before the program

UWES of direct reports at 6 months

3

Job Performance Needs

Immediate manager effectiveness in the

areas of leadership, setting and maintaining standards, and developing and

motivating employees

Application Objectives

Effectively and continuously apply the five self-

coaching skills at work

Effectively create and sustain motivational work environments that increase engagement

Application

Participant self-assessment at 3 months after completion of the program

UWES of direct reports at 3 months

2

Learning Needs

Increase success skills of immediate managers

in the areas of leadership, setting and maintaining standards, and developing and motivating employees

Learning Objectives

Immediate managers learn to effectively apply

the five self-coaching skills of self-managing, reflecting, acting consciously, collaborating, and evolving

Learn how to foster motivational work environments that increase engagement

Learning

Session content summaries, participant assignments, and skill development journal entries during the program

Pre-/post-self-assessment profile Utrecht Work Engagement Scale (UWES) of

direct reports during the program

1

Preference Needs

Learning that is relevant and important to

successful job performance

Reaction Objectives Program content receives favorable rating of 4 out of 5 in relevance and

importance

80% of participants identify planned actions

Reaction

Reaction and planned action questionnaires at the end of each session of the program

 
< Prev   CONTENTS   Next >