Suggestions for Future Research and Recommendations for Practice
After outlining two SJT perspectives (context-dependent vs. general domain knowledge), we end this chapter by highlighting some important avenues for future research. In the preceding sections, we briefly touched on some of those future research directions.
A first vital issue is to gain a better understanding of the circumstances in which each perspective produces the best criterion-related and construct-related validity evidence. For example, when designing an SJT for entry-level admission purposes, evidence accumulated throughout this chapter is unsupportive of contextualizing such an SJT and instead supports streamlining the SJT, thereby making it more context-independent (e.g., Kell et al., 2014). Interestingly, development costs are reduced while at the same time the test’s criterion-related and construct-related validity are not jeopardized. As a potential disadvantage, however, applicants might perceive the generic SJT to be less job-related because the relation to the job becomes somewhat less obvious (as manifested in the more generic wording of the item stems and response options). Similarly, at this time we do not know how contextualized and domain-general SJTs compare to one another in terms of fakabil- ity, subgroup differences and coachability. In such comparative evaluations, it is important to take the method-construct distinction into account (Arthur & Villado, 2008). That is, when tests are compared on their content, the test format should be kept constant. By directly contrasting contextualized with domain-general SJTs, it becomes possible to provide HR practice with the empirical evidence it needs to confirm the legitimacy of these issues. Krumm and colleagues (2015) carried out an example of such a study. They distinguished between two conditions: one in which a traditional SJT was used and another condition in which the situation description was removed from the items of the same SJT. So, respondents received only the item options in that condition. These conditions were implemented across three SJTs: a teamwork SJT, an integrity SJT and an aviation SJT. The results showed that the provision of context had less impact than expected. That is, it did not matter for about 50-70% of the items whether situation descriptions were included in terms of the number of correct solutions per item. In addition, respondents’ expertise level, item length, item difficulty and response instruction did not moderate the results.
A second area of research deals with examining the effectiveness of the different SJTs for specific practical purposes. As a backdrop to this, we recommend using domain-general SJT items for entry-level selection. Conversely, context-specific SJTs seem particularly useful when applicants have already acquired the requisite fine-grained procedural (general) and declarative (job-specific) knowledge. Contextualized SJT items can then home in on such context-dependent knowledge. These items are particularly useful for advanced-level selection and certification applications. When selecting for specialized functions, declarative knowledge is an essential component in addition to procedural knowledge for effective job performance. Initial research is supportive of these recommendations because in advanced-level selection, administering a contextualized SJT was found to capture both procedural and declarative knowledge (Lievens & Patterson, 2011). Future studies should focus on further elucidating the additional value of increasing the contextualization of SJTs in advanced-level selection as compared to domain-general SJTs.
Training applications represents another SJT purpose that is relevant to our distinction and in urgent need of research. For training purposes, we also recommend using contextualized SJTs. SJTs might be specifically adapted to use as tools in training needs analysis (assessment of pre-training knowledge), as actual training content materials or as a training outcome assessment instrument. In particular, contextualized SJTs might be useful as training content materials in scenario-based training in which scripted work situations allow trainees to practice critical job-related skills in a safe environment (Fritzsche, Stagl, Salas & Burke, 2006). So far, virtually no research is available on the efficacy of using SJTs in training. Therefore, we need studies that explore to what extent increasing the response and/or stimulus fidelity of SJTs improves the training’s effectiveness.
Fourth, future research might benefit from making a clearer distinction between these two SJT types. Many existing SJTs contain both generic and contextualized items
(Krumm et al., 2015). This might impact construct measurement. In particular, keeping the contextualization of SJT items at the same level (as required by the criterion specificity to be predicted) might lead to a better measurement model underlying SJTs as some of the item heterogeneity that has been posited to lead to poor factor analytical results in SJTs is removed. Generally, we believe that SJT research should not receive a ‘free pass’ on the construct measurement issue and should continue to undertake efforts to improve construct measurement in SJTs.
Efforts on a clearer distinction between these two SJT types might also address when and how test-takers make use of the context provided. That is, we should also be concerned with the underlying thought processes when solving SJTs. Leeds (2012) suggests that solving an SJT is a two-step process in which test-takers first scrutinize response alternatives in an absolute (‘How effective is this option?’ ‘Does it make sense?’) as well as in a relative sense (‘Is this option better than that one?’). In a second process, test-takers take the contextual information as presented in the situation description into account. So, one may assume that even contextualized SJTs are only ‘used’ for context-specific judgements if test-takers’ primary perusal of response options is inconclusive as to how to respond. Interestingly, Rockstuhl, Ang, Ng, Lievens and van Dyne (2015) revealed that the judgements made by test-takers on the basis of the situation descriptions (i.e., their construal of the situation) were equally or even more predictive of job-related criteria in an international context as compared with the judgements made on the basis of response alternatives alone. Thus, how test-takers construe and use the context provided could also be an important part of the information captured with SJTs. An avenue for future research may be to comparatively examine the cognitive underpinnings described by Leeds (2012) and Rockstuhl and colleagues (2015) (e.g., through eye-tracking or verbal protocol analysis) and also to assess their relevance in contextualized and generic SJT items.
A final interesting aspect of context-independent SJTs that deserves more research deals with their claim that they can be used cross-culturally. This assumption is based on the notion that such SJTs were designed to measure general procedural knowledge of the costs and benefits of engaging in specific trait-relevant behaviour. Conversely, contextualized SJTs are more dependent on the context and culture for which they were developed, and therefore cross-cultural transportability might be a problem (Lievens, 2006). Like cognitive ability and personality tests, general domain knowledge SJTs are developed to have generalizability across a wide variety of situations. Therefore, they could potentially be implemented more easily across different cultures. That said, we also caution that ITPs might be valued differently across cultures. For example, individualistic cultures might value expressions of extraversion in a specific situation, whereas collectivistic cultures might value these expressions less in that same situation and instead value expressions of other traits such as agreeableness more. Accordingly, empirical evidence is needed to determine the extent to which domain-general SJTs can be successfully implemented across different cultures.
