EVALUATION RESULTS
Level 1, Reaction Results
The four participating immediate managers from the production unit completed reaction questionnaires at the end of each of the seven learning sessions. The areas surveyed were content relevance to the job, content importance to job success, intent to use the material on the job, facilitator effectiveness, material effectiveness, likelihood the participant would recommend the program to others, and overall satisfaction. Participants rated these items on a scale of 1 (strongly disagree) to 5 (strongly agree). They also indicated planned actions. Evaluation targets were set for a mean rating of 4 out of 5 points for content relevance and importance, and at least 80 percent of participants' identifying planned actions. The targets set for content relevance and importance, and for planned actions were met. Content relevance and importance scored mean ratings of 4.16 and 4.07, respectively. Also, all participants developed plans to apply what they learned during the program.
Level 2, Learning Results
Production unit managers were taught the self-coaching skills of self-managing, reflecting, acting consciously, collaborating, and evolving during seven learning sessions over 90 days. Facilitator observations, participant self-assessments, and UWES measurements of the participant's direct reports were used to evaluate learning.
The preand post-program self-assessment profile asked participants to rate how consistently they practiced 25 specific work behaviors anchored to the five skills. Each behavior was rated on a scale of 1 (never) to 5 (always). Administered at Day 1 of the learning initiative, the preprogram self-assessment mean score for all 25 behaviors was 105.0 points. The Day 90 self-assessment mean score was 107.6 points. An increase of 2.6 points for the mean, while statistically inconclusive, suggests the possibility that at least a modest amount of learning occurred during the program. The project team deemed this acceptable, since the learning was designed to become participant self-directed and to continue after the formal program ended. This evolution from facilitated instruction to self-coaching required that learning be evaluated as it was practiced on-the-job. With this in mind, work engagement levels would help evaluate participant skill acquisition during the program.
Taken in the context of the organizational setting, work engagement readings of the production test group and maintenance control group line employees provided additional insight into the participants' on-the-job skill development. A highly disruptive plant fire that occurred toward the end of the learning program produced a significant gap between these two groups; a slight drop in production work engagement compared to a much steeper decline in maintenance. A thorough assessment of the situation found that the fire should have had no more of an adverse impact on maintenance (control group) compared to production (test group). According to the senior human resource manager, the lead training manager, and the business unit and line managers at PolyWrighton, both groups experienced the same extended period of excessive overtime and intense physical labor. While work engagement was more negatively affected in maintenance compared to production, no apparent reason could be identified. This may suggest that the production immediate managers had been better equipped to deal with the plant fire and its consequences than their maintenance business unit counterparts. In other words, it is possible that the five skills helped keep the production unit from being as negatively affected.
