General Domain Knowledge SJTs: Implications and Trends
The conceptualization of SJTs as measures of relatively context-independent knowledge has fundamental implications for SJT design. If SJTs aim to tap into general domain knowledge, it seems to make less sense to invest in elaborate, contextualized situation descriptions. Instead, this perspective conceptually guides research efforts to streamline SJTs. This can be done in at least two ways. One approach is to make use of singleresponse SJTs in which test-takers are asked to rate the effectiveness of a single critical action (Crook et al., 2011). As described above, traditional contextualized SJT items usually have multiple-response options. Test developers have to gather a large number of response options from SMEs in the test construction phase. Next, response options have to be investigated and checked for redundancy and SME agreement, before ending up with a pool of suitable response options for the final SJT. Single-response SJTs are proposed to reduce this laborious and time-intensive process because the edited critical incidents (i.e., retaining only the situation description and a single critical action) can directly serve as the response options, thereby rendering the need for generating large amounts of response options superfluous. SMEs also simply rate the effectiveness of edited critical incidents. When applicants complete the SJT, they have to provide an effectiveness rating for each item, which is compared to the one generated by the SME for scoring purposes. Thus, each item of a single-response SJT consists of a couple of sentences describing one critical brief incident, with candidates being asked to rate the effectiveness of this incident. Crook and colleagues (2011) created such single-response SJTs (see also Motowidlo et al., 2009; Motowidlo et al., 2013). In two studies, Crook et al. (2011) found singleresponse SJTs to be significantly correlated with performance (r = 0.22-0.33), and showed that job knowledge as measured by one of their SJTs showed 4% incremental variance on SJT scores above personality. These preliminary findings are in line with McDaniel and colleagues’ (2007) meta-analytic evidence of traditional SJTs, suggesting that single-response SJTs do not appear to pay a ‘predictive power reduction price’ for their streamlined development.
A similar approach would be to eliminate the situation stem altogether and ask test-takers to rate the effectiveness of several courses of action from a multiple-choice response option set (Kell, Martin & Motowidlo, 2011) Kell et al. (2014) devised such a test consisting of 40 brief descriptions of courses of action - in this case physicians interacting with patients. An example of one such (effective) description is: ‘When a 10 year old with a broken arm needed surgery, the anesthetist introduced herself to the parents and then knelt down to the child’s eye level to introduce herself to the child’. The statements were developed from critical incidents. Test-takers have to score each item’s effectiveness. Thus, their test is similar in format to single-response SJTs (with the exception that only the actions were retained and the situations were dropped from the items) and was designed to measure prosocial knowledge (i.e., helping behaviour). Prosocial knowledge as measured with this instrument correlated 0.20 with clinical skill on a standardized patient examination (SPE). Furthermore, prosocial knowledge scores were positively associated with students’ clinical performance scores from their primary care rotations (r = 0.22), but non-significantly correlated to students’ clinical performance scores in the specialties (r = -0.04). So, this study suggests that general-domain knowledge seems to be more important in the early phases of one’s career before specialization takes place, and declines in importance in the later phases when specialized skills become more and more essential.