METHODS FOR COLLECTING DATA

Given the considerations covered in the previous section, a variety of methods and instruments are available to collect data at the different levels of evaluation. Some techniques are more suited toward some levels of evaluation than others; but in many cases, the approaches to data collection can cut across all levels of evaluation. Table 3-8 lists different data collection methods used to collect data at different levels. The most often used are questionnaires, interviews, focus groups, action plans, and performance records.

Table 3-8. Data Collection Methods

Method

1

Type of Data Level 2 3

4

Surveys

+

+

+

Questionnaires

+

+

+

+

Observation

+

+

+

Interviews

+

+

+

Focus Groups

+

+

+

Tests/Quizzes

+

Demonstrations

+

Simulations

+

Action Planning/Improvement Plans

+

+

Performance Contracting

+

+

Performance Monitoring

+


Questionnaires and Surveys

Surveys and questionnaires are the most often used data collection technique when conducting an ROI evaluation. Surveys can collect perception data (such as reaction), and precise data (such as the amount of sales). Questionnaires and surveys are inexpensive, easy to administer, and depending on the length, take very little of respondents' time. Questionnaires can be sent via mail, memo, email, or distributed online (posted on an intranet or via one of any number of survey tools available on the Internet).

Questionnaires also provide versatility in the types of data that can be collected. They are used to collect data at all levels of evaluation: the demographics of participants (Level 0), reaction to the learning program (Level 1), knowledge gained during the program (Level 2), how participants applied that knowledge (Level 3), and the impact of the application (Level 4). You can also ask participants to indicate how much a particular measure is worth, how much that measure has improved, other variables that may have influenced improvements in that measure, and the extent of the influence of those variables.

Questions can be open-ended, closed, or forced-choice. Likert-scale questions are common in questionnaires, as are frequency scales, ordinal scales, and other types of scales, including paired comparison and comparative scales. Periodically, an adjective checklist on a questionnaire gives participants the opportunity to reinforce their perception of the program.

While questionnaires can be quite lengthy and include any number of questions, the best are concise and reflect only questions that allowfor the collection of needed data. Results from brief questionnaires are powerful when describing the impact of a learning program, as well as its monetary benefits.

Interviews

Interviews are an ideal method of data collection when details must be probed from a select number of participants. Interviews allow for gaining more in-depth data than questionnaires, action plans, and focus groups. However, it is important to consider costs and utility, particularly when considering evaluation at Levels 1 and 2. Guiding Principle 2 states, “When evaluating at a higher level, the previous level does not have to be comprehensive.” For example, if you plan to evaluate the program to Level 3, it would not be cost effective to use interviews to collect Level 2 learning data.

Interviews can be structured or unstructured. Unstructured interviews allow for greater depth of dialog between the evaluator and the participant. Structured interviews work much like a questionnaire, except that there is a rapport between the evaluator and the participant. The respondent has the opportunity to elaborate on responses and the evaluator can ask follow-up questions for clarification.

Interviews can be conducted in person, over the telephone, or online. Interviews conducted in person have the greatest advantage, because the person conducting the interview can show the participant items that can help clarify questions and response options. In-person interviews also allow for observation of body language that may indicate that the participant is uncomfortable with the question, anxious because of time commitments, or not interested in the interview process. Unlike the situation with an email or web-based questionnaire where the disinterested participant can simply throw away the questionnaire or press the delete key, in an interview setting, the evaluator can change strategies in hopes of motivating participants. Interviews are used when the evaluator needs to ask complex questions or the list of response choices is so long that it becomes confusing if administered through a questionnaire. In-person interviews are often conducted when the information collected is viewed as confidential or when the participant would feel uncomfortable providing this information on paper or over the telephone.

While interviews provide the most in-depth data, they are also the most expensive. Scheduling interviews can be a challenge with busy managers, professionals, and sales staff. If possible, consider using a professional interviewer, who is skilled at interviewing as well as at using the ROI Methodology. The interviewing process can be daunting, especially when asking questions related to Level 4 business impact measures, isolation, and data conversion. A third-party interviewer skilled in these techniques can ensure that the data obtained are accurate and credible when presented to stakeholders during the reporting phase.

Focus Groups

Focus groups are a good approach to collect information from a group of people when dialogue among the group is important. Focus groups work best when the topic is important to the participants. High-quality focus groups produce discussions that address the topics you want to know about. The key to successful focus groups is to keep focused. Serious planning is necessary to design the protocol for a focus group. The conversations that transpire are constructed conversations focusing on a key issue of interest.

Action Plans and Performance Contracts

In many cases, action plans are incorporated into the program. They are used to collect Level 3 and Level 4 data. Prior to the learning program, participants identify specific business measures they need to improve as a result of the program. Through the process they, along with their program leader, identify specific actions to take or behaviors that they will change to target improvement in those measures.

Performance Records

Performance records are organizational records. Data found in performance records represent standard data used throughout the organization in reporting success for a variety of functions; using performance records as a method of data collection can save time and money. Sales records and quality data are generally easy to obtain. However, not all measures in which there is interest are readily available in the record. It would be a wise investment of your time to learn what data are currently housed within the organization and can be utilized or referenced for the program.

 
< Prev   CONTENTS   Next >