EVALUATION METHODOLOGY

In order to validate the relevance and profitability of the training program offered to Future-Tel's sales representatives and solutions specialists, TalentPlus suggested conducting an impact study.

The Phillips ROI Methodology was selected due to its proven and documented use. This approach produces six types of data: reaction, learning, application, business impact, ROI, and intangible benefits. In addition, the methodology uses a systematic logic model to collect, analyze, and report data. Along the way, the process is guided by 12 conservative standards.

The levels of data captured by the ROI Methodology are listed in Table 8-1.

Table 8-1. Levels of Data

Level 1: Reaction

• Measures the participants' reactions to training.

• Data captured at the end of the session.

Level 2: Learning

• Measures the participants' level of retention.

• Data captured at the end of the session and validated in the application survey.

Level 3: Application

• Assesses the frequency of use of what was learned.

• Data captured via an online survey, 60 to 90 days after the training.

Level 4: Impact

• Measures variations on KPIs (revenues, margins, etc.).

• Data captured via an online survey and through client database, 120 to 180 days after the training.

Level 5: ROI

• Measures the actual ROI by comparing the net monetary contribution of the training program against its fully loaded costs.


General Description of Approach

Figure 8-2 shows that the ROI Methodology was integrated into the entire training project.

Before starting the design of the training content, a meeting took place between TalentPlus CRP's and Future-Tel's stakeholders in order to align (phase 1) the client business objectives and the evaluation objectives. Following the alignment meeting, a data collection plan and an ROI analysis plan were produced and then validated by Future-Tel. Table 8-2 describes the assessment levels.

Figure 8-2. Evaluation Methodology

Table 8-2. Desired Outcomes

Level 5

Desired ROI

What kind of ROI is required from this project?

Level 4

Desired business impact

To achieve the ROI target, what business impact is required?

Level 3

Behavior outcomes

To get business impact, what kind of behaviors need to change?

Level 2

Learning outcomes

To change those behaviors, what must people learn?

Level 1

Reaction outcomes

To drive learning, what kind of reactions must we provoke?

During the skill-building phase, 24 training sessions (one day) took place. At the end of each training session, the participants, the most credible source of data, had to complete two questionnaires to assess Levels 1 and 2. Sixty days after the completion of data collection at Levels 1 and 2, each sales professional who attended the one-day training session had to complete a Level 3 survey. The collected data allowed assessment of what extent the salespeople were using the knowledge acquired during their one-day training session.

In addition, after 120 days, the sales professionals were given a business impact questionnaire (Level 4) to verify if the changes in terms of attitude and behavior were maintained. This questionnaire also allowed for collection of information on predefined key performance indices (KPI).

At the end of the program, Future-Tel granted TalentPlus access to its business results database. Performance indicators such as average revenue per closed sale, number of opportunities closed with their dollar amount, and sales cycle duration were observed.

To calculate a conservative and precise ROI, one KPI was selected: revenue increase. It was decided to measure the difference between the total sales revenues won by Future-Tel nine months before the training sessions and nine months after the training program.

In order to isolate the effects of the program, the participants were asked to estimate in percentage the actual impact of the training on their sales results as well as their level of confidence in their evaluation. This provides the actual contribution of the program.

The revenue increase measured was then multiplied by the actual contribution to assess the portion of revenue increase that could be associated with the program.

Finally, in order to be as conservative as possible, the ROI was measured by comparing the costs of the program to the net profit generated through the revenue increase. (The net profit considered in this study was chosen based on the lowest net margins observed in Future-Tel's annual reports over the last three years.)

 
< Prev   CONTENTS   Next >