DATA COLLECTION PLAN

Table 10-1 provides a detailed look at the data collection plan. It is important to note that Level 2 and 3 data are actually collected through placement and progress assessments within the GE program administrative module and are not extraneous data collection tools. As an integral part of the program, these assessments meet the recognized generally accepted standards and criteria of ESL assessment methodology.

Table 10-1. Data Collection Plan

CONVERTING DATA TO MONETARY VALUES

Based on the data collected, the Level 1 through 3 objectives contribute to the development of a positive case for the use of GE in delivering ESL learning. Translating that learning into a monetary value comes from the Level 4 and 5 assessments, which ask learners how much time they and their managers believe was saved due to improved English language skills. To add to the credibility of this estimate, the respondents' level of confidence in that estimate was captured and ultimately combined with the respondents' level of confidence in GE's contribution to that result to develop a very conservative and credible estimate of the time saved due to new ESL skills. Once a time saved value is determined (for instance, one hour per week), a financial value will be associated with that time (such as a fully loaded labor cost) and an ROI cost determined by taking into account the annual estimate of the value of the time saved, less the annual value of the time spent studying, divided by the annual cost per learner for the program. This approach will provide a very solid ROI case for the program.

ISOLATING THE IMPACT OF TRAINING

A chain of evidence is an essential component of demonstrating the impact and ROI of a training solution used as a performance improvement intervention. Specifically, conducting Level 1, 2, 3, 4, and 5 evaluations. Level 5 evaluation is critical to developing a supportable and credible foundation for the overall ROI analysis.

It is important to note that the learners and their managers are scattered around the globe. With 426 users (and associated supervisors) around the world, the most efficient method to gather learner reaction (Level 1), business impact (Level 4), and ROI (Level 5) was in a single survey. The critical element is that the survey had to be crafted in such a way that learner (and manager) reactions and expectations are aligned with a perception of the metrics that are relevant to the measure of success used by these learners, supervisors, and their organization. By linking Level 1, 4, and 5 data questions into a single survey, collating the data from potentially more than 850 people (426 learners and the same number of managers) from around the globe, the data collection became a much more manageable task. It is also important to keep in mind that in addition to this annual survey, which captures Levels 1, 4, and 5, the learners are constantly taking Level 2 and 3 assessments as they progress through the program.

Another factor in planning to conduct a global ROI study for an ESL program was the quality of English of both the beginning learners and the supervisors. Just because the learner is learning English language skills does not mean all the skills to answer a survey were mastered or that the learner's supervisor(s) have mastered enough English as well (in order to respond to the survey). In addition, there are no resources to translate the survey into all the languages that might be required. As a result, it was important to keep the survey as short as possible, keep the form of English as simple as possible, and leave enough time for respondents to complete the survey. These factors were treated as prerequisite issues and potential constraints that had an impact on the design of the evaluation tools.

Consistent with the ROI Methodology Guiding Principle 5—“At least one method must be used to isolate the effects of the solution”—to isolate the contribution of training to improve job performance as a result of better English language skills, a number of the ROI Methodology techniques were employed. Two of the nine methods of data isolation were used in this ROI analysis.

Guiding Principle 7 states that estimates of improvements should be adjusted for the potential error of the estimate. To accomplish this, a questionnaire to gather data from learners was developed. The three key questions in order to capture data for the ROI analysis were:

1. How much time has been saved due to improved English language skills?

hours have been saved each week.

2. On a percentage basis, what is the confidence level concerning the number of hours saved each week? %

3. On a percentage basis, what is the confidence level that GlobalEnglish is the reason this time has been saved? %

Again, consistent with Guiding Principle 7, a second questionnaire was developed to gather data from the learners' supervisors. The key questions for the supervisors were:

1. If one were to assume that improved English language skills allow an employee to work more effectively due to the ability to read and respond to English language emails, telephone calls, teleconferences, and meetings with greater skill and confidence, what would an estimate be of how many minutes or hours per week the learner(s) save(s) due to a perceived improvement in English language skills?

2. On a percentage basis, what is the confidence level concerning the number of hours saved each week? %

3. On a percentage basis, what is the confidence level that GlobalEnglish is the reason this time has been saved? %

With the data that resulted from these questions, a unit of measure, that is, time saved, was established, converted into a dollar value, and ultimately used to generate a benefit-cost ratio (BCR) and an ROI. This is an approach that, while not terribly sophisticated, is simple to measure, very defensible to management, and fits within the constraints of conducting a global ROI analysis.