Forecasting Methods

A more analytical approach to trend-line analysis is the use of forecasting methods that predict a change in performance variables based on the correlation of other variables. This approach represents a mathematical interpretation of the trend-line analysis when other variables enter the situation at the time the learning program is implemented. The primary advantage of forecasting is that it can predict performance in business measures with some level of accuracy, if appropriate data and models are available. While there are no absolutes with any technique, using an appropriate level of statistical analysis can provide credible, reliable results.

A major disadvantage with forecasting occurs when several variables enter the process. The complexity multiplies, and the use of sophisticated statistical packages for multiple-variable analyses is necessary. Even then, a good fit of the data to the model may not be possible. Many organizations have not developed mathematical relationships for output variables as a function of one or more inputs. Without them, the forecasting method is difficult to use.

Expert Estimation

An easily implemented method to isolate the effect of a learning program is to obtain information directly from experts who understand the business performance measures. The experts could be any number of individuals. For most learning programs, the participants are the experts. After all, their performance is in question and the measure is reflecting their individual performance. They may know more about the relationships between the different factors, including the impact of the learning, than any other individual.

Because of the importance of estimations from participants, much of the discussion in this section relates to how to collect this information directly from them. The same methods would be used to collect data from others. The effectiveness of the approach rests on the assumption that participants are capable of determining how much of a performance improvement is related to the learning program. Because their actions have produced the improvement, participants may have an accurate perception of the issue. Although an estimate, this value will typically have credibility with management because participants are at the center of the change or improvement.

When using this technique, several assumptions are made:

O A learning program has been conducted with a variety of different activities, exercises, and learning opportunities all focused on improving performance.

O Business measures have been identified prior to the program and have been monitored following the program. Data monitoring has revealed an improvement in the business measure.

O There is a need to link the learning program to the specific amount of performance improvement and develop the monetary effect of the improvement. This information forms the basis for calculating the actual ROI.

O The participants are capable of providing knowledgeable input on the cause-andeffect relationship between the different factors, including learning and the output measure. With these assumptions, the participants can pinpoint the actual results linked to the program and provide the data necessary to develop the ROI. This can be accomplished by using a focus group or a questionnaire.

Focus Group Approach

The focus group works extremely well for this challenge if the group size is relatively small—in the eight to 12 person range. If much larger, the group should be divided into multiple groups. Focus groups provide the opportunity for members to share information equally, avoiding domination by any one individual. The process taps the input, creativity, and reactions of the entire group.

When conducting a focus group, the following steps are recommended to arrive at the most credible value for learning program impact:

1. Explain the task.

2. Discuss the rules.

3. Explain the importance of the process.

4. Select the first measure and show the improvement.

5. Identify the different factors that have contributed to the performance.

6. Identify other factors that have contributed to the performance.

7. Discuss the linkage.

8. Repeat the process for each factor

9. Allocate the improvement.

10. Provide a confidence estimate.

11. Ask the participants to multiply the two percentages.

Example

Participants who do not provide information are excluded from the analysis. Table 4-1 illustrates this approach with an example of one participant's estimations. The participant allocates 50 percent of the improvement to the learning program. The confidence percentage is a reflection of the error in the estimate. A 70 percent confidence level reduces the estimate to an adjusted percentage of 35 percent (50% x 70% = 35%). In essence, this error adjustment assumes the lowest percentage in an error range. If a person is 70 percent confident in their estimate, that means they are 30 percent uncertain (a 30% error). Given this level of uncertainty, the margin of error is 50% x 30% = 15%. With a margin of error of +/– 15 percent, the range of improvement is 35 to 65 percent. To be conservative, the lowest end of the range, 35 percent, is reported as improvement. Participants who do not provide information are excluded from the analysis.

Table 4-1. Example of a Participant's Estimation

Factor That Influenced Improvement

Percentage of Improvement

Percentage of Confidence Expressed

Adjusted Percentage of Improvement

Technology-Based Program

50%

70%

35%

Advertisements

10%

80%

8%

Market Growth

10%

50%

5%

Revision to Incentive Plan

30%

90%

18%

Total

100%

The use of expert estimations provides a credible way to isolate the effects of technology-based learning when other methods will not work. It is often regarded as the low-cost solution to the problem because it takes only a few focus groups and a small amount of time to arrive at this conclusion.

 
< Prev   CONTENTS   Next >