ADDRESSING THE NEED: ONE FAILURE
Because succession planning was the driving factor in developing English language skills in high-potential affiliate personnel, the solution had to be cost-effective, minimize time away from work, and accommodate students at various levels of English capability. After repeated requests from overseas affiliates for support of this activity, several global ESL training companies were located and reviewed. Ultimately, a global contract meant to leverage global buying efficiencies was negotiated with Berlitz language services. However, after the first year the approach was abandoned for the following reasons:
O Despite global pricing, many affiliates were able to find local classroom training at lower prices—usually from a local university.
O Classroom training sponsored within the company's local offices presented a challenge in scheduling.
O To keep costs reasonable, classes had a wide range of learners at various skill levels. Beginner, intermediate, and advanced learners sat in the same classroom, which created a lot of challenges.
The affiliates refused to participate in the Berlitz program after the first year, so the Berlitz contract was allowed to expire without renewal. An alternate solution needed to be located—one that addressed the problems experienced with the Berlitz program and one that the markets would embrace. An online ESL training system was found to be the potential solution. At the time, GlobalEnglish (GE) was one year old and had been started by a venture capital company. GE's premise was simple:
O English is the global language of business.
O All resources would be put into developing interactive learning tools to support just one language—English.
O There were 11 levels of business English covered in the GE program.
O There were different course tracks for different English language skills, such as grammar, writing, speaking, and listening.
O The approach would allow a learner with level 10 skills in writing to take the level 10 writing track. However, if the learner had level five English grammar skills, the learner would be placed in the level five grammar track. This allowed for unprecedented customization not possible in a group classroom setting.
O Instructions on using the system in the first five levels are provided in local languages until the learner had sufficient English language skills to follow both lesson instructions and the lesson content completely in English.
A pilot program for GE was commenced for 50 people in several affiliates. The ROI Methodology was not applied to the pilot. What was used was a simple measure of Level 1–3 results. Specifically, learners provided feedback on their experience using the GE system (Level 1). Their improvement (or lack thereof) in English test scores from their original placement to when they completed the pilot were measured through assessment in the GE system (Level 2). Simulated application exercises, also within the GE system, were measured from the benchmark placement process at the beginning of the pilot to the learner's final performance at the end of the pilot (Level 3). Based on the results of the pilot, which were favorable, the GE system was adopted.
The L&P department, the program sponsor, was tasked with organizing the global rollout of the program. Since then, a limited ROI analysis has been added to the original pilot measures and is conducted in the fourth quarter every year. The original ROI method simply asked the learners to provide an estimate of how much time was saved due to their new English language skills. Once an “hour” value was established, conducting a ROI was relatively easy. On average, the company has seen an 800 percent ROI for each of the five years the program has been employed. However, many stakeholders were skeptical about the results of those earlier ROI analyses. Therefore, after five years, elements of the ROI Methodology were also added. For the first time, learners and their managers were asked: In addition to how much time was saved due to improved English language skills, how confident are learners and their managers in that estimate and how certain are they that the GE program was the reason for the results? This approach created a more defensible and credible ROI analysis. The balance of this study focuses on this improved process. The year the ROI Methodology was adopted there were 426 users on the system, an all-time high. Therefore, a more credible and rigorous ROI analysis was critical to measuring the value of the program.
Level 1, Reaction Objectives
1. Determine learner satisfaction with the GE learning methodologies using a 5-point scale from “dissatisfied” to “extremely satisfied.”
2. Determine learner self-satisfaction with progress improving English skills using a 5-point scale from “dissatisfied” to “extremely satisfied.” This last point can be correlated to actual Level 2 and 3 assessment results to see if learner perception matches reality.
Level 2, Learning Objectives
1. Objective test scores for knowledge based on placement assessment and progress assessment in order to progress through the 11 levels. An assessment score of 80 percent or higher is required to move to the next level in a particular skill set.
2. Learning objectives will focus on knowledge of vocabulary and rules of grammar.
Level 3, Application Objectives
1. Objective test scores for skill application based on placement assessment and progress assessment in order to progress through the 11 levels. An assessment score of 70 percent or higher is required to move to the next level in a particular skill set.
2. Application objectives, which focus on reading, writing, listening, and speaking skills as applied to specific, real job tasks, as differentiated from simulated tasks or assignments for reading, writing, listening, and speaking skills within GE. It is important to note that, with regard to reading, writing, listening, and speaking skills—whether applied to a simulated task (as within a course lesson) or when applied to a real job task or activity—the methods used for evaluation will be quite similar. The difference in this case is that the simulated Level 3 situations within GE can be automatically assessed by the administration module within the GE system. The real-world job application will need to be evaluated by qualified assessors on the job.
3. Assessment by the learners and their managers (or qualified assessors) using an on-the-job checklist of 12 business situations will be conducted. Ratings will be ranked on a measure of five levels of improvement.
Levels 4 and 5, Business Impact and ROI Objectives
1. Determine level of importance of English skills to the learner's job and career aspirations.
2. Determine learner's estimate of time saved due to improved English language skills.
3. Determine learner's confidence estimate of how much time was saved due to improved English skills.
4. Determine learner's percent estimate of GE's contribution to the improvement in the learner's English language skills.