A variety of data collection techniques can be used to collect the right data from the right source at the right time. How data are collected depends upon a variety of factors, including accuracy, time, cost, and utility.


The data collection technique that will provide the most accurate results is desired when selecting a data collection method. However, accuracy will have to balance with the cost of data collection. Usually the higher the accuracy, the higher the costs. Never spend more on data collection than the cost of the program. A guideline to keep in mind is that the full cost of an ROI study should not exceed 5 to 10 percent of the fully loaded cost of the learning program. All evaluation costs are included in the denominator of the ROI equation, which means expensive data collection reduces the ROI percentage. It's usually a trade-off.

Figure 3-3. Completed ROI Analysis Plan

Program: Product Upgrade With Mobile Learning



Data Items (Usually Level 4)

Methods for Isolating the Effects of the Program/ Process

Methods of Converting Data to Monetary Values

Cost Categories

Intangible Benefits

Communication Targets for Final Report

Other Influences/ Issues During Application


• Monthly sales per associate

• Control group analysis

• Participant estimates

• (Both


• Direct conversion using standard profit contribution

• Needs assessment

• Design

• Content development

• Mobile device

• Participants' salaries plus benefits (time)

• Cost of coordination and administration (time)

• Project management (time)

• Evaluation

• Customer engagement and satisfaction

• Job satisfaction of sales associates

• Stress reduction

• Reputation

• Program participants

• Sales managers

• Product manager

• Senior executives, regional and headquarters

• Learning coordinators, designers, and managers

• All sales associates

• No communication with control group

• Time to first sale

• Control group analysis

• Participant estimates

• (Both


• N/A

Validity and Reliability

A basic way to look at validity is to ask, “Are you measuring what you intend to measure?” Content validity can be determined using sophisticated modeling approaches; however, the most basic approach to determining the validity of the questions asked is to refer to objectives. Well-written objectives represent the measures to take. Consider the use of subject matter experts, along with additional resources, such as literature reviews and previous case studies to judge validity.

While validity is concerned with ensuring you are measuring the right measures, reliability is concerned with whether the responses are consistent. The most basic test of reliability is repeatability. This is the ability to obtain the same data from several measurements of the same group collected in the same way. A basic example of repeatability is to administer the questionnaire to the same individual repeatedly over a period of time. If the individual responds the same way to the questions every time, there is minimum error, meaning there is high reliability. If, however, the individual has different responses, there would be high error, meaning low reliability.

Time and Cost

When selecting data collection methods, several issues should be considered with regard to time and cost. The time required to complete the instrument is one consideration. Also, consider the time required for managers to complete the instrument if they are involved, or the time in assisting participants through the data collection process. All expenditures for data collection—including time to develop and test the questionnaire, time for the completion of data collection instruments, and the printing costs—are costs to the program. Also, consider the amount of disruption that the data collection will cause employees; interviews and focus groups typically require the greatest disruption, yet provide some of the best data. Balance the accuracy of the data needed to make a decision about the program with what it will cost to obtain that data.


A final consideration when selecting a data collection method is utility. How useful will the data be, given the type of data collected through the process? Data collected through a questionnaire can be easily coded and put into a database and analyzed. Data collected through focus groups and interviews, however, call for a more challenging approach to analysis. While information can be collected through dialogue and summarized in the report, a more comprehensive analysis should be conducted. This requires developing themes for the data collected and coding those themes. This type of analysis can be quite time-consuming and in some cases frustrating if the data are not collected, compiled, and recorded in a structured way.

Another issue with regard to utility has to do with the use of the data. Avoid asking a lot of questions simply because you can, and instead consider whether you really need to ask a question in order to obtain the data to make decisions about the learning program. Remember, data collected and reported leads to business decisions, regardless of whether the programs are offered through a corporate, government, nonprofit, community, or faith-based organization. How can you best allocate the resources for programs to develop people or improve processes? With these issues in mind, if you can't act on the data, don't ask the question.

< Prev   CONTENTS   Next >