Measuring the Success of Learning Through Technology

PrefaceTwo books in one Suggestions Target audience Acknowledgments I The ROI Methodology. A Credible Approach to Evaluating Your Learning Through Technology ProgramsLearning Through Technology: Trends and IssuesTHE EVOLUTION OF LEARNING TECHNIQUES Mobile LearningGame-Based LearningBring Your Own Devices (BYOD)Open Educational Resources (OER)Massive Open Online Course (MOOC)Flipped ClassroomCONCERNS ABOUT TECHNOLOGY-BASED LEARNING The Need for Business ResultsThe Executive ViewResults Are MissingReasons for Lack of DataFINAL THOUGHTS Measuring ROI: The BasicsROI DEFINED THE ROI METHODOLOGY Evaluation FrameworkROI Process ModelOperating Standards and PhilosophyCase Applications and PracticeImplementationROI PROCESS Evaluation PlanningData CollectionData AnalysisReportingBENEFITS OF ROI Justify SpendingImprove the Learning ProgramSet PrioritiesEliminate Unsuccessful ProgramsGain SupportFINAL THOUGHTS Evaluation Planning and Data CollectionACHIEVING BUSINESS ALIGNMENT Clarifying Stakeholder NeedsDetermining Program ObjectivesEvaluating the ProgramDEVELOPING THE EVALUATION PLANS CONSIDERATIONS FOR COLLECTING DATA AccuracyValidity and ReliabilityTime and CostUtilityMETHODS FOR COLLECTING DATA Questionnaires and SurveysInterviewsFocus GroupsAction Plans and Performance ContractsPerformance RecordsGENERATING HIGH RESPONSE RATES IDENTIFYING THE SOURCE Performance RecordsParticipantsParticipants' ManagersParticipants' Peers and Direct ReportsSenior Managers and ExecutivesOther SourcesDETERMINING THE TIMING OF DATA COLLECTION FINAL THOUGHTS Data AnalysisISOLATING THE EFFECTS OF LEARNING Other Factors Are Always PresentWithout Isolation, There Is Only Evidence, Not ProofOther Factors Have Protective OwnersTo Do It Right Is Not Always EasyTECHNIQUES TO ISOLATE THE EFFECTS OF LEARNING Comparison Group AnalysisAssumptionsIssues and Opportunities With Comparison GroupsTrend-Line AnalysisAdvantages and Disadvantages of Trend-Line AnalysisForecasting MethodsExpert EstimationFocus Group ApproachQuestionnaire ApproachSELECTING ISOLATION TECHNIQUES TYPES OF DATA Hard Data vs. Soft DataTangible vs. Intangible DataDATA CONVERSION METHODS Standard ValuesHistorical Costs CalculationInternal and External ExpertsExternal DatabasesLinking With Other MeasuresEstimationsFIVE STEPS TO DATA CONVERSION FULLY LOADED COSTS Initial Analysis and AssessmentDevelopment of SolutionsAcquisition CostsApplication and Implementation CostsMaintenance and MonitoringSupport and OverheadEvaluation and ReportingROI CALCULATION INTANGIBLE BENEFITS FINAL THOUGHTS Reporting and Using the ResultsTHE IMPORTANCE OF REPORTING RESULTS IDENTIFY THE NEEDS IDENTIFY THE AUDIENCE SELECT THE MEDIA MeetingsInternal PublicationsElectronic MediaBrochuresCase StudiesFormal ReportsDEVELOP THE REPORT Detailed ReportsNeed for the ProgramNeed for the EvaluationEvaluation MethodologyResultsConclusions and Next StepsAppendicesExecutive SummaryGeneral Audience ReportsSingle-Page ReportsMacro-Level ScorecardsCOMMUNICATE RESULTS AND EVALUATE THE RESULTS OF COMMUNICATION DELIVERING BAD NEWS USING THE DATA FINAL THOUGHTS Designing for ResultsCOMMUNICATING WITH RESULTS IN MIND AnnouncementsBrochuresMemosWorkbooksCHANGING THE ROLE OF PARTICIPANTS Why This Is NecessaryDefining the RoleDocumenting the RolesCREATING EXPECTATIONS Identifying Measures Before the ProgramInvolving the ManagersMessages From ExecutivesDESIGN FOR RELEVANCE DESIGN FOR RESULTS ActivitiesSkill PracticesSimulationsProblemsCase StudiesBUILT-IN APPLICATION TOOLS Action PlansImprovement Plans and GuidesApplication Tools/TemplatesPerformance ContractJob AidsUSE TRANSFER TOOLS INVOLVING THE PARTICIPANT'S MANAGER The Most Influential GroupPreprogram ActivitiesDuring-the-Program ActivitiesPost-Program ActivitiesReinforcement ToolsFINAL THOUGHTS II Evaluation in Action. Case Studies Describing the Evaluation of Learning Through Technology ProgramsMeasuring ROI in Work Engagement: A Blended Learning SolutionBACKGROUND THE NEED PROGRAM OBJECTIVES A Summary of the Five SkillsBasis for Linking Skills to PerformancePURPOSE OF EVALUATION EVALUATION METHODOLOGY Categories/Levels of DataData Collection StrategyROI Analysis StrategyIsolation TechniquesData Conversion TechniquesCost SummaryEVALUATION RESULTS Level 1, Reaction ResultsLevel 2, Learning ResultsLevel 3, Application ResultsBarriers to ApplicationEnablers to ApplicationLevel 4, Business Impact ResultsIsolationData Conversion to MoneyProgram CostsLevel 5, ROIIntangible BenefitsCOMMUNICATION STRATEGY Results ReportingStakeholder ResponseProgram ImprovementLESSONS LEARNED Process LearningOrganizational ResponseQUESTIONS FOR DISCUSSION ABOUT THE AUTHORS Measuring ROI in Sales Training: A Game-Based ProgramBACKGROUND Program DescriptionEVALUATION METHODOLOGY General Description of ApproachDATA COLLECTION STRATEGY ROI Analysis StrategyOBSERVATIONS AND CAPTURED DATA Reaction (Level 1) DataLearning (Level 2) DataApplication (Level 3) DataBusiness Impact (Level 4) DataIsolating the Effects of the ProgramConversion MethodologyFully Loaded CostROI CALCULATION INTANGIBLE BENEFITS LESSONS LEARNED A High Retention Level Yields a High Impact on SalesA Strong Correlation Exists Between Actual Usage and Contribution to SalesSeveral Teams Have Taken Ownership of the Content ElementsThe Data Show a Clear Chain of ImpactThe Absence of Barriers Correlates With Greater UseTeamwork: Mission AccomplishedThe Content Was Not Only Useful but Very RelevantApplication of Content Yields Business ResultsQUESTIONS FOR DISCUSSION RESOURCES ABOUT THE AUTHORS Measuring ROI in a Sales Program: A Web-Based Learning SolutionPERFORMANCE AND BUSINESS NEEDS THE E-LEARNING SOLUTION EVALUATION PLAN AND OBJECTIVES Program CostE-LEARNING PROGRAM RESULTS COMMUNICATION OF RESULTS LESSONS LEARNED QUESTIONS FOR DISCUSSION ABOUT THE AUTHOR Measuring ROI in an English-asa-Second-Language Program: An Online Learning SolutionBACKGROUND THE TRAINING NEED KEY STAKEHOLDERS ADDRESSING THE NEED: ONE FAILURE MEASURING RESULTS Evaluation ObjectivesLevel 1, Reaction ObjectivesLevel 2, Learning ObjectivesLevel 3, Application ObjectivesLevels 4 and 5, Business Impact and ROI ObjectivesDATA COLLECTION PLAN CONVERTING DATA TO MONETARY VALUES ISOLATING THE IMPACT OF TRAINING EVALUATION RESULTS Determining the BenefitsCalculating the CostsCOMMUNICATION PLAN CONCLUSIONS DISCUSSION QUESTIONS REFERENCES ABOUT THE AUTHOR Measuring ROI in an Upgrade Selling Program: A Mobile Learning SolutionBACKGROUND Organizational ProfileSituationSolutionRationale for ROIOBJECTIVES PLANNING Data Collection PlanROI Analysis PlanRESULTS Isolating the Effects of the ProgramImpact ResultsComparing Data to MoneyCostIntangible BenefitsCOMMUNICATING RESULTS RECOMMENDATIONS LESSONS LEARNED QUESTIONS FOR DISCUSSION ABOUT THE AUTHORS Measuring ROI in a Sales Program: An E-Learning SolutionBACKGROUND DESIGNING AND IMPLEMENTING THE SOLUTIONS The Technology Learning Competency (TLC) ProgramMEASUREMENT METHODS AND DATA ANALYSIS ROI Model and ProcessThe Data Collection Plan and ROI Analysis PlanLevel 1 and 2 DataLevel 3 and 4 DataIsolating the Effects of the ProgramDesign and Implementation CostsLevel 3 and 4 ResultsROI ResultsIntangible BenefitsLESSONS LEARNED REPORTING TO STAKEHOLDER GROUPS QUESTIONS FOR DISCUSSION ABOUT THE AUTHOR About the ROI InstituteBUILD CAPABILITY IN THE ROI METHODOLOGY ROI CERTIFICATION Index
Next >