Understanding how international experiences engage employability: a game-based analytics approach

Dolly Predovic and John L. Dennis

Graduate employability is a key issue for higher education institutions. Industry recruiting strategies have evolved in recent years and the focus has shifted from graduates who have sound academic knowledge to graduates who can also demonstrate how they apply knowledge and other transferable skills in the workplace.

International experiences matter for employers but only if graduates can transform skills acquired into behaviours that are observable and translatable into value-adding workplace performance. We used game-based analytics to gain insight into hidden behaviours associated with skills that are valued most by employers. In doing so, this gave us an opportunity to think more creatively about employability development through international experiences.

In order to understand whether international experiences enhance graduate employability, it is necessary to reduce conceptual ambiguity and define employ- ability. In fact, the operationalisation of employability from a theoretical concept to a measurable index is not a small undertaking. This chapter represents a tentative answer: game-based analytics. Most literature concentrates on the perception of different stakeholders on the development of employability skills, and our study tries to capture how well students can transform these skills into behaviours. We adopt a theoretical concept of employability under the proces- sual perspective of ability to apply knowledge and skill, and we measured it by analysing behaviours with game-based analytics.

Employability definition1

The most widely investigated definition of employability is linked to a possession perspective, based on the assumption that employability is defined by skills and personal attributes that make graduates more likely to gain employment and successfully keep it (Yorke, 2006). Holmes, starting with his seminal work in 2001 (Holmes, 2001), challenged the possessive perspective on employability and based on skills, and built a “graduate identity” approach with a conceptual distinction between three explanations of graduate employability: skills “possession,” social/ cultural capital “position,” and the “process” graduates use to present their claim on being a graduate worthy of employment (Holmes, 2013).

Several recent perspectives on employability are consistent with Holmes’s idea of employability as a process. For example, Reid (2016) argues that employability must account for the social, political, and personal context of the recent graduates, while Jackson’s (2016) concept of pre-professional identity is the result of a sense-making process where a “student makes sense of his/her intended profession through multiple memberships and differing levels of engagement with various communities.” Similarly, Finch, Peacock, Levallet, & Foster’s (2016) idea of an integrated dynamic capabilities view where a graduate’s intellectual, personality, meta-skills, and job-specific resources are developed over time to give the graduate a competitive advantage and employability.

Measuring employability

Operationalising employability and finding an adequate assessment tool have been big challenges, and, generally speaking, employability assessments fall into three main categories: self-assessment, quizzes, and serious games (Employment Ontario, 2015).

Self-assessments have strong limitations, such as scoring accuracy and “content accuracy” (Panadero, Brown, & Strijbos, 2016), as well as social desirability bias, and, in fact, Kormos and Gifford (2014) find that 79% of the variance in the relationship between self-reported and objective behaviour remains unexplained.

Quizzes allow one to judge the quiz-taker’s ability to demonstrate the skills being analysed (Darling-Hammond, 2014). Online, there are many such quizzes (mettl.com, centraltest.com, testofy.com) but very often they are simply poorly disguised self-assessment questionnaires (Employment Ontario, 2015).

Gaming is a new trend in psychometric testing and has been defined as “the use of game design elements in non-game contexts” (Deterding, Dixon, Khaled, & Nacke, 2011), and most employability related academic research in the field of gaming focuses on learning and developing employability skills through games - for example, the European Modes project (Haselberger et al., 2012).

Game-based learning and assessment

The largest body of research on game-based learning (GBL) investigates the learning potential of games (Boyle et ah, 2016). Numerous studies analyse the impact of serious gaming on the development of employability skills: communication skills (Reinders, 2014; Romero, Usart, & Ott, 2015), critical thinking (Carolyn Yang & Chang, 2013), problem solving (Sung, Hwang, & Yen, 2015), conflict resolution (Ramon & Cristobal, 2015), decision making (Savard, 2015), cultural skills (Romero et ah, 2015), and leadership (De Freitas & Routledge, 2013; kin & kin, 2014).

Our focus is instead on game-based assessment, which can be achieved in three ways: game scoring, external assessment, or embedded assessment (Ifenthaler, Eseryel, & Ge, 2012). Game scoring focuses on the targets achieved during the game and is important for the player’s motivation, which is a critical component of skill development and assessment (Keller, 1987). External assessments are not part of the game environment and are “real,” through interviews, questionnaires, or essays (Chin, Dukes, & Gamson, 2009). Embedded assessments, or stealth assessments, are part of the game play and do not interrupt the game. Rich data about the player’s behaviour while playing is the basis for the assessment of the skills. Implementing assessment features in a digital game-based environment is done only in a rather early stage of development because it is a very time consuming to step into the design process, and it needs to be tested in order for it to be reliable (Chin et al„ 2009).

The KNACK

The KNACK suite of tests are stealth assessments that have been tested extensively and have been proven to have, both, very high reliability and validity indicators (Gray, Jerde, Prabhakaran, & Carroll, 2016). The United States Agency for Youth Development (USAID) has chosen the KNACK as being in the top 3% of the measurement tools they analysed (Galloway, Lippman, Burke, Diener, & Gates, 2017). Essentially, the KNACK, as a predictive analytic tool, helps employers find the right fit for employees by assessing the underlying processes that guide behaviour, thoughts, and emotions (basically one’s psychology) and mapping that performance onto extremely well-known, well-tested, and scientifically sound measures (Galloway et al„ 2017).

The KNACK as game-based talent analytics has been found to be a reliable and quantifiable predictor of workplace performance. Players’ “micro-behaviours (e.g., the position and timing of screen gestures, user actions in relation to the state of the game, and so on) are logged at the millisecond level with such data density that we are able to recreate a given game session as the player made it happen.” (Gray et al„ 2016). From this data, within-game behavioural markers are generated that represent things such as how quickly a player processes information or how efficiently they attend to and see social cues, like facial emotional expressions, and then these markers are built upon to validate higher-level psychological constructs, such as intelligence or a growth mind-set. From these mappings, to numerous constructs, predictions to real-world outcomes are then generated for each individual player.

International experiences influence employability

The link between international mobility and graduate employability has been investigated from multiple perspectives: those of universities, employers, academics, and students (Crossman & Clarke, 2010; European Commission, 2014), students who have participated in learning abroad and alumni (Dwyer, 2004; Farrugia & Sanger, 2017; Mohajeri Norris & Gillespie, 2009; Potts, 2015) employers (Archer & Davison, 2012), employers and universities (Diamond, Walkley,

Forbes, Hughes, & Sheen, 2011), and signalling effect (i.e. students with international experiences are more likely to be called for an interview) (Petzold, 2017).

Jones argues that the benefits of internationalisation on employability, either through graduate mobility or through the internationalisation of the curriculum at home, are still not entirely understood by universities, employers, and even students (Jones, 2012, 2013, 2016). Discrepant perspectives on the value of international experiences among students, graduates, career development professionals, and employers are confirmed by Kinash, Crane, Judd, & Knight (2016).

Trooboff, Vande Berg, & Rayman’s (2008) seminal paper finds that human resource professionals and non-senior management, contrary to common belief, place significant value on studying abroad. The main reason is that over 15% of the respondents have studied abroad themselves and by virtue of their own experience are positively disposed. Furthermore, among the different types of study abroad analysed in the research, findings show that employers have a strong preference for internships. More recently, the employers perspective on international study versus international internships in 31 European countries were analysed by Van Mol (2017) and this research confirms that employers seem to value internships abroad more than study abroad; however, this did vary across the countries in his study. For example, more than 40% of employers from Cyprus, Turkey, Luxembourg, Latvia, and Italy value international internships, while fewer than 10% of employers from Hungary, Croatia, Norway, Sweden, and the UK value international internships.

Present study

We conducted a study to determine how different international experiences affect employability. Considering our previous discussions, while we know that the KNACK measures employability, what we don’t know is how the different international experiences translate into different KNACK scores and, therefore, into different measures of employability.

Data from 414 graduate students from 28 Italian universities was used and the study was conducted in conjunction with a project for a major multinational consulting company. The project’s goal was to select 100 graduating students to be invited for a three-day talent program in the company’s headquarters. The project was not a recruiting process for the consulting company, but a project aimed at identifying what tomorrow’s top employable graduates should look like. Between November 2016 and February 2017, 28 Italian universities were visited. In order to participate in the selection process, students were asked to submit their resumes and motivation letters and to complete two KNACK games.

In all, 1973 resumes, motivations letters, and KNACK scores were received, and 414 candidates passed the first selection round and represent the sample used for the analysis. Of the sample group, 63% were male and the age distribution showed most participants (about 80%) were 23-25 years old. About two thirds of the participants studied economics, business, or management, while the remainder were enrolled in engineering (18%), managerial engineering (12%), and other fields (6%). Their previous experiences ranged from domestic internships (62%), domestic casual work (32%), international internships (23%), and international casual work (6%). Fifty-nine per cent had participated in study abroad prior to their participation.

Employability measures

Employability is measured by how students perform on 33 KNACKs, which result from playing the two KNACK assessment games, Meta Maze and Dash Dashi. The 33 KNACKs (see Table 7.1) are each measured on a scale, from 0 to 100, and they can be divided into five groups:

• • Engagement: howyou engage with the world and demonstrate professionalism
• • Impact: how you make an impact on people and organisations
• • Learning: how you learn new information and skills, and your motivation to learn
• • Relationships: how you relate to other people and yourself
• • Thinking: how you perform knowledge work and solve problems

Results

We conducted an analysis to assess whether KNACK scores differed as a function of demographic variables, (i.e., age and gender), internship experience (domestic or international), casual work experience (domestic or international), and study abroad experience.

Table 7.1 Factors in employability and the KNACKs associated with them

By using an exploratory factor analysis, it was possible to explore the structure of the 33 KNACKS and determine if they grouped together in a coherent fashion in relation to our independent variables - that is, international and domestic internships, study abroad, international and domestic casual work, gender and age.

Our analysis grouped the KNACKs into two main factors, revealing the underlying relationships between the 33 KNACKs. Factor 1 we describe as a social/ effort factor that relates to employability behaviours defined by engagement (how one engages with the world) and relationships (how one relates to other people). Factor 2 we describe, instead, as a more cognitive factor that relates to employability behaviours defined by how one learns new information and motivation to learn.

What emerges from our analysis is that only international internships significantly impact the ability to successfully apply cognitive skills - like quick thinking, learning agility, data fluency, and creative insight - into workplace behaviours - that is, Factor 2. From our data, we argue that international internships are associated with higher-order capabilities; specifically, an enhanced power of learning (Rospigliosi, Greener, & Bourner, 2011), which is related to the highest cognitive domain in Bloom’s taxonomy (Bloom, 1956) and is also exactly what employers seem to value the most (Finch, Hamilton, Baldwin, & Zehner, 2013). None of our independent variables significantly impacted the social/effort employability behaviours - that is, Factor 1.

The effect of international experiences on graduate employability has been extensively investigated with varied outcomes. Among intrapersonal competencies developed by study abroad, previous research has demonstrated that students self-rate as being more flexible, adaptable, and self-aware and as having developed better intercultural skills, while being more curious and having more confidence, while interpersonal competencies, such as communication, teamwork, and leadership are not rated as higher post study abroad (Farrugia & Sanger, 2017). Consistent with these results, Trooboff et al. (2008), found that while employers value team work more highly than any other skill, they believe that this skill is least likely to be enhanced through study abroad. Our research demonstrates that perhaps this previous research might not have taken into account those cognitive abilities, such as quick thinking, learning agility, data fluency, and creative insight.

All stakeholders (employers, academics, and students) seem to agree that international experiences do, in general, enhance learning, the acquisition of competencies, and the development of critical soft skills (Crossman & Clarke, 2010). Jones (2013) offers a very comprehensive review of literature on the influence of key transferable employability skills on international experience, divided between self-sufficiency/self-efficacy skills and people skills, and concludes that “it seems evident that transferable skills and capabilities are developed through international mobility, equally it may be the case that international mobility programs appeal to students who already possess, or have an advantage in developing, these skills.”(Jones, 2013, p. 8) In fact, one limitation of our research findings is that since we don’t have before and after snapshots of employability skills, we don’t know whether those students who scored higher on employability skills post international internship had those very skills before they engaged in their abroad employment experience.

Conclusions

International experience matters for employers, but only if graduates can transform skills acquired into behaviours that are observable and translatable into value-adding workplace performance.² Unexpectedly, our research finds that an international experience translates into behaviours involving the highest order cognitive skills (e.g., quick thinking, learning agility, and creative insight).

Game-based analytics allow us to gain insight into the hidden behaviours associated with those skills that employers value most and offer us an opportunity to think more creatively about employability skills development through internationalisation. Our current research goes a step further by demonstrating behaviours that have not been typically found to be the expected outcome of international experiences - that is, higher-order cognitive skills.

According to Cavanagh, Burston, Southcombe, & Bartram (2015), students rate high-order skills as the most difficult to develop and to relate to work contexts. International internships might help with just that. Perhaps, those skills develop “under the radar” - such that students don’t really know that they have developed them, and gaming analytics like the KNACK can help identify these hidden skill acquisitions.

There are also interesting prospects for internationalisation at home. Although collaborative online international learning is involved in an increasing number of programs, more could be done to actively simulate international workplace environments in virtual classrooms (Schech, Kelton, Carati, & Kingsmill, 2017). This would enable educators to offer such experiences to the entirety of the student body, not just to the mobile minority. Designing “international” internship activities into the curricula at home could yield unexpected and exciting findings.

This study underlies the importance of looking at employability from a behavioural perspective and looking at international experiences not from a social perspective but rather from a cognitive perspective. Our study could lead, therefore, to a paradigm shift where self-report data must be evaluated in conjunction with behavioural data, and where, for international experiences, the role of cognitive skills is evaluated in conjunction with social skills.

Notes

• 1. Other discussions about the definition or concept of employability can be found on pages 12, 25, 25, 60, 62,117, 164.
• 2. This appears to be so; see impact on career development of Japanese graduates with international study experiences on page 82.