GENERATING HIGH RESPONSE RATES

An often asked question when considering the data collection process is, “How many responses do you need to receive to make the data valid and useable?” The answer is, all of it! Guiding Principle 6 states that if no improvement data are available for a population or from a specific source, it is assumed that no improvement has occurred. While it is unlikely that 100 percent of potential respondents will provide data, it is important to collect as many responses as possible. Inference may not be possible to nonrespondents, so if 20 participants are involved and data are only provided by 10, results are only reported for the 10 and all the analysis and ROI is based on the 10 responses. This conservative standard ensures that credible results are reported.

If we report for nonrespondents, then we inflate the results on an assumption for which we have no basis. However, because we also adhere to Guiding Principle 10, costs of the solution or program should be fully loaded for ROI analysis, we will account for the cost of learning for all 20 participants. So, the key is to develop a strategy to obtain responses from as many potential respondents as possible.

Table 3-9 lists a variety of action items to take to ensure an appropriate response rate. Start by providing advanced communication about the evaluation. Clearly communicating the reason for the evaluation ensures that participants understand that the evaluation is not about them, it is about improving the program. Identify those people who will see the results of the evaluation and assure them that they will receive a summary of it. If you are using a questionnaire as a data collection instrument, keep it as brief as possible by asking only those questions that are important to the evaluation. If possible, have a third party collect and analyze the data so participants feel comfortable that their responses will be held in confidence and anonymity will remain.

IDENTIFYING THE SOURCE

Selecting the source of the data is critical in ensuring accurate data are collected. Sometimes it is necessary to obtain data from multiple sources. A fundamental question should be answered when deciding on the source of the data: Who (or what system) knows best about the measures being taken?

The primary source of data for Levels 1, 2, and 3 is the participants. Who knows best about their perception of the program, what they learned, and how they are applying what they learned? Although at Level 3, it may also be important to collect data from other sources, such as the manager, to validate or complement the findings.

Table 3-9. Increasing Response Rates

Source: Phillips, P.P., J.J. Phillips, and B. Aaron. (2013). Survey Basics. Alexandria, VA: ASTD Press.

Performance Records

Given the variety of sources for the data, the most credible source is the organization or internal performance records. These records reflect performance in a work unit, department, division, region, or organization. Performance records can include all types of measures that are usually readily available throughout the organization. This is the preferred method of data collection for Level 4 evaluation, since it usually reflects business impact data.

Participants

Participants are the most widely used source of data for ROI analysis. They are always asked about their reaction to the program and the extent to which learning has occurred. Participants are often the primary source of data for Levels 3 and 4 evaluation. They are the ones who know what they do with what they learned and what happened that may have prevented them from applying what they learned. In addition, they are the ones who have insight to what impact their actions have on the business.

While some perceive participants as a biased option, if they understand the purpose of the evaluation and that the evaluation is not about them—it is about the program—participants can remove their personal feelings from their answers and provide objective data.

Participants' Managers

Managers of the participants are another important source. In many cases, they have observed the participants as they attempt to use the knowledge and skills. Those managers, who are actively engaged in a learning process, will often serve as support to the participant to ensure that application does occur. Data from managers often balance the participants' perspectives. In collecting data from the managers, keep in mind any potential bias that may occur from this source of information.

Participants' Peers and Direct Reports

When evaluating at Level 3, participants' peers and direct reports are a good source of data. The 360-feedback evaluation provides one of the most balanced views of performance because it considers the perspective of the participants, their managers, their peers, and their direct reports. While gathering input from peers and direct reports can increase the cost of the evaluation, their perspective may add a level of objectivity to the process.

Senior Managers and Executives

Senior managers and executives may also provide valuable data, especially when collecting Level 4 data. Their input, however, may be somewhat limited if they are removed from the actual application of the knowledge and skills applied. However, senior managers and executives may play a key role in the data collection process when implementing a high-profile program where they have a significant investment.

Other Sources

Internal and external experts and databases provide a good source of data when converting business impact measures to monetary value. The ideal situation is to collect monetary value for the business impact measures from the internal experts or databases outside the organization's records.

DETERMINING THE TIMING OF DATA COLLECTION

The last consideration in the data collection process is timing. Typically, Level 1 data are collected at the completion of the program, and Level 2 data are collected during or at the completion of the program.

Level 3 and 4 data collection occurs after the application is routine—the time in which new behaviors are internalized or the actions are completed. The goal is to collect data as soon as possible, so participants can connect the application to the program. Typically, Level 3 data collection occurs three weeks to two months after the program is complete. Some programs, where skills or actions are applied immediately upon conclusion of the program, should be measured in a matter of days. With Level 4 data, timing may be different than from for Level 3 evaluation, depending on data availability, the stakeholder requirements, and opportunity for the measure to improve. The issue is this: What is the delay or lag time between application and the corresponding impact? Sometimes there is no delay; at other times it may be several months. Usually Level 4 data is collected from three weeks to four months.

While the ROI calculation is an annual benefit, it is unlikely that you will wait a full year to capture Level 4 data. Senior executives usually want to see results sooner rather than later. If the program was introduced to solve a problem (such as unsatisfactory sales revenue), executives and senior managers want the data soon. Otherwise, the decision will be made without the data. It's ideal to collect the Level 4 measures either at the time of Level 3 data collection or soon after, when impact has occurred. Then, those measures should be converted to monetary benefits and included in the ROI calculation.

Sound data collection strategy is imperative for achieving credible results. Ensuring that the most appropriate methods, sources, and timing are employed in the data collection process will yield results that are reliable and useful to stakeholders. However, it is through the analysis that the real story of learning success is told. Analysis begins with isolating the effects of the program on improvement in business measures.

FINAL THOUGHTS

This chapter introduced the concept of achieving business alignment, which is important for any program, particularly those with significant business impact. It also discussed the importance of evaluation planning (to maintain alignment throughout the evaluation) and data collection. The various methods of data collection were outlined. Using these methods of data collection, you will be able to collect the most credible and timely data and can begin with the data analysis, which is discussed in the next chapter.