Unobtrusive Measurement of Team Cognition: A Review and Event-Based Approach to Measurement Design

Salar Khaleghzadegan, Sadaf Kazi, and Michael A. Rosen

CONTENTS

Introduction..............................................................................................................95

Team Cognition: Concept and Measurement...........................................................97

Key Concepts in Team Cognition.......................................................................97

Traditional Methods of Team Cognition Measurement......................................98

Emerging Methods for Assessing Team Performance.............................................99

Team Physiological Dynamics............................................................................99

Location and Activity Sensing.....................................................................101

Linguistic Features of Communication.............................................................102

Paralinguistic Features of Communication.......................................................103

Event-Based Methods for Unobtrusive Measurement of Team Cognition............105

Conclusion and Future Directions.........................................................................108

Acknowledgments..................................................................................................108

References..............................................................................................................108

INTRODUCTION

In today’s complex and dynamic work environment, organizations look to teams to address problems effectively and efficiently. Team-based work structures drive safety, innovation, productivity, and other important organizational outcomes across industries. One key dimension of teamwork at the forefront of understanding team performance across these criteria and work settings is collective or team cognition, a multi-level phenomenon underlying how teams process information, make decisions, plan, learn, and adapt. There are many perspectives on these general phenomenon ranging from those that focus primarily on how individual cognition (i.e., knowledge or cognitive processing) is shared or distributed across space, time, and team members (e.g., DeChurch & Mesmer-Magnus, 2010), to those taking a broader perspective to understand cognitive processing as a property of the sociotechnical system as a whole, inclusive of humans, technological agents, information systems, and artifacts in the physical environment (Hollan, Hutchins, & Kirsh, 2000; Zhang & Norman, 1994). These person-focused and whole-system-focused approaches have been integrated in different ways (e.g., Fiore et al., 2010) over the years. This chapter focuses primarily on unobtrusive measurement of team process, and therefore is most closely aligned with the view of team social interaction processes constituting a critically important aspect of distributed cognition (Cooke, Gorman, Myers, & Duran, 2013). However, these methods can be extended to understand broader aspects of distributed cognition such as artifact use in team settings (e.g., Li, Yao, Pan, Johannaman et ah, 2016).

While team cognition is important for all teams, it is especially vital for teams that operate in dynamic and complex work settings that contain high degrees of stress, workload, and severity of consequences for performance lapses. The measurement of team cognition is therefore critical to both researchers seeking to build and refine better theories of team cognition, and to practitioners seeking to develop interventions to compose, train, or support team cognition in vivo. Over the years, a variety of methods have been developed and refined to measure different aspects of team cognition (Wildman, Salas, & Scott, 2014). However, new technologies and analytic tools provide the opportunity to create a new generation of team cognition measurement methods. Advances in wearable and environmental sensors, natural language processing, video analysis, and high dimensional time series analysis promise to deliver scalable and meaningful data collection around social interactions in field and lab settings (Yarkoni, 2012). Potential advantages of these emerging technologies include less labor-intensive data acquisition over longer durations of time, ultimately allowing for the measurement of team cognition in actual task environments rather than simulated conditions. While there are clearly open methodological issues to resolve (Chaffin et al., 2017), technical capabilities and evidence are maturing quickly.

In this chapter, we review the state of the science in applying unobtrusive measurement approaches to team cognition. To that end, we address three goals. First, we briefly review key concepts and traditional measurement strategies for team cognition. Second, we present a framework of unobtrusive measurement features (i.e., patterns in communication, physiological, and activity monitoring data) which can be used as indictors of team cognitive processing. Third, we detail a method for designing unobtrusive team cognition measurement systems within training and assessment simulations. This approach builds from an event-based approach to training (EBAT) methods of concurrent simulation and measurement system development (Fowlkes et al., 1997). Using this approach, training objectives and desired competencies drive the design of scenario events delivered through simulation. Observation of targeted responses demonstrate the presence or absence of competencies tested in the scenario and feedback can help improve performance for that desired competency.

TEAM COGNITION: CONCEPT AND MEASUREMENT

Key Concepts in Team Cognition

Team cognition encompasses a variety of concepts, including team mental models (TMM), transactive memory systems (TMS), and team situation awareness (TSA). These different terms conceptualize team cognition as inputs (e.g., TMMs), processes (e.g., interactive team cognition; Cooke et al., 2013), and emergent states (e.g., TSA; Gorman, Cooke, & Winner, 2006). TMMs describe the content and structures of individual team member’s knowledge (Mohammed, Ferzandi, & Hamilton, 2010). Expanding previous research on TMMs, temporal TMMs incorporate time- related dimensions of teamwork which had been lacking in previous TMM frameworks because of the critical role that time can play on taskwork and performance (Mohammed, Hamilton, Tesler, Mancuso, & McNeese, 2015). These knowledge structures may relate to teamwork (e.g., roles and responsibilities, team interdependencies, model of interaction, etc.) or task work (e.g., nature of team task, tools and technology, etc.; Cooke, Kiekel, & Helm, 2001; Mathieu, Heffner, Goodwin, Salas, & Cannon-Bowers, 2000; Mohammed et ah, 2010). The concept of TMS also concerns knowledge, but focuses on its distribution between team members, and on processes through which knowledge is organized (Wegner, 1987).

As opposed to the knowledge-based perspective of team cognition, Cooke and colleagues (2008, 2013) adopt a more dynamic view of team cognition that focuses on processes through which team members interact. TSA is also a dynamic construct of team cognition because it refers to the collection of an individual team member’s perception and comprehension of the current state of the environment and projection of its state in the future (Bolstad & Endsley, 2003; Endsley, 1988). The use of scaled world simulations in research has been instrumental in conducting empirical research in concepts like TSA because of the ability to study real-world problems in realistic settings within a controlled lab environment (McNeese, McNeese, Endsley, Reep, & Forster, 2017). Many of the scaled world simulations that are used today in studies of different aspects of teamwork constructs stem from the NeoCITIES simulation, which is an interactive simulation environment for emergency crisis management (Hamilton et al., 2010; Hellar & McNeese, 2010). These simulations are used today in various team environments to study different variables like TSA, including their use in surgical teams (Hazlehurst, McMullen, & Gorman, 2007) and cybersecurity teams (Mancuso & McNeese, 2013).

Concepts in team cognition differ in their unit level of analysis (e.g., measurement of individual- vs. team-level data), as well as how frequently team cognition is measured. Some concepts within team cognition consider team cognition as the aggregate (e.g., sum, similarity, dispersion, fit) of the knowledge of individual team members at any given point in time. For example, TMMs are typically measured by assessing knowledge structures of individual team members at a single time point, and then comparing similarities between members’ knowledge structures and accuracy level of knowledge structures (Mohammed et al., 2010). Similar, or shared, mental models can indicate that team members have similar descriptions, predictions, and explanation of events (Cannon-Bowers, Salas, & Converse, 1993). Accuracy of knowledge structures is assessed by comparing TMMs with objective evaluations of team performance (Edwards, Day, Arthur, & Bell, 2006). The research connecting shared mental models to team performance is mixed. Whereas some studies have found task work accuracy to be associated with better team performance (Cooke et al., 2001), others have found that teamwork similarity was more important than task work similarity for team performance (Webber, Chen, Payne, Marsh, & Zaccaro, 2000).

Measures relying on simple aggregations of individual team member cognitions may not be appropriate to make inferences about team cognition because they assume that interaction between team members is linear and that all members contribute equally across tasks (Webber et ah, 2000). Therefore, Cooke and colleagues propose a more dynamic view of team cognition that is focused on evaluating team interactions across multiple points in time. This view considers team cognition as an emergent phenomenon that is adaptive to and shaped by interactions between team members as they accomplish task goals. Measures based in the interactive team cognition view focus on holistic, team-level communication that encapsulates dynamic cognitive processing as the locus of measurement.

Traditional Methods of Team Cognition Measurement

Similar to other domains of research in psychology, team cognition has been traditionally measured through self-reported knowledge and perceptions (e.g., surveys, interviews) and behavioral observations. Interviews and focus groups are used to elicit the structure, content, and organization of knowledge. In addition, these methods are also valuable in assessing recollections of interactive team processes. There is great diversity in methods to assess self-reported individual knowledge and perceptions about the team. These include surveys (e.g., transactive memory; Lewis, 2003), knowledge tests (Smith-Jentsch, Mathieu, & Kraiger, 2005), relatedness ratings (Fisher, Bell, Dierdorff, & Belohlav, 2012), card sorting (Smith-Jentsch, Campbell, Milanovich, & Reynolds, 2001), concept mapping (Marks, Sabella, Burke, & Zaccaro, 2002), and in-task probes (Bolstad & Endsley, 2003). Audio, video, and task logs and observations are valuable sources of capturing real-time team interactive processes. Data from all these sources are most commonly analyzed through aggregation, although researchers also use content analysis, analysis of patterns (e.g., recurrence, dominance, etc.), computational scaling, and holistic consensus (Wildman et al., 2014).

Decisions about methods of data collection should be theoretically guided by the team cognition construct under consideration and the overarching research question. However, it is equally important to be cognizant about the feasibility of data collection from the desired sample. Wildman et al. (2014) propose a variety of criteria to guide researchers in choosing appropriate methods of collecting data on team cognition. These include determinations about knowledge vs. interaction processes contained in the data, the underlying dynamic structure of the phenomenon, etc. In addition, Wildman and colleagues also evaluated methods of measuring team cognition in terms of their resource requirements for data collection and storage and the level of obtrusiveness to participant work. Audio and video observations may require low researcher burden during data collection and are relatively unobtrusive, but require significant resources during storage, processing, and analysis. Methods relying on self-report may be well suited to capturing knowledge about teamwork structures. However, they may be obtrusive and interrupt work, thus changing the nature of work.

EMERGING METHODS FOR ASSESSING TEAM PERFORMANCE

In this section, we provide an overview of emerging methods used to study group and team performance that could be applied to the measurement of team cognition. These methods are unobtrusive in nature and can complement the traditional methods reviewed in the previous section. These methods include team physiological dynamics, location and activity sensing, and linguistic and paralinguistic features of communication. In Table 5.1, we present a brief overview of data streams, data measures, and representative findings of these methods. We expand on these measures below.

Team Physiological Dynamics

Team physiological dynamics (TPD) refers to the continuous assessment of patterns of physiological arousal, synchrony, and organization of individual team members during team performance episodes (Lewis, 2003). TPD has been studied on diverse populations, including submarine operators (Gorman et al., 2016), healthcare teams (Stevens, Galloway, Halpin, & Willemsen-Dunlap, 2016), and flight deck pilot crews (Toppi et ah, 2016), in addition to college students (Chanel, Kivikangas, & Ravaja, 2012). This body of research has enabled the study of important variables in team cognition.

Gorman and colleagues (Gorman et ah, 2016) investigated neurophysiological and communication patterns of submarine operators in a simulation. Teams of novice and experienced submarine operator teams were measured prior to the task during briefing, during the task, and after the task during debriefing. Neurophysiological activity was measured through neurodynamic entropy (a measure of the change in neurophysiological distribution of a team) with a measure of low entropy representing a smaller change in team neurophysiological distribution (i.e. relatively fixed team mental state) and a measure of high entropy representing a higher change in team neurophysiological distribution (i.e. more flexible team mental state). In addition, communication content was analyzed using latent semantic analysis, a computational tool that measures the degree to which communication is synchronous or asynchronous based on the level of similarity between words that co-occur in dialog (Landauer, Foltz, & Laham, 1998; Dong, 2005). Teams of experienced submarine operators were more flexible during the task than novice teams, as shown by higher neurodynamic entropy in experienced teams during the scenario. This flexibility allows teams to function effectively and efficiently without becoming too rigid or deterministic in their operation (Stevens & Galloway, 2014). In addition, changes in communication patterns occurred before changes in neurophysiological patterns in experienced teams. This suggests that experience level and communication patterns can influence TPD.

TABLE 5.1

Unobtrusive Methods for Measuring Team Cognition

Physiology

Activity Tracing

Data Streams

  • • Cardiac system
  • • Electrodermal system (EDA)
  • • Facial electromyography
  • • Electroencephalography (EEG)
  • • Wearable and environmental sensors
  • • Accelerometer
  • • Use of the byproducts of interaction captured through information systems used for collaboration (e.g„ email, paging)

Data Measures

  • • Interbeat interval, beats per minute, respiratory sinus arrhythmia, etc.
  • • Tonic/phasic activity
  • • Degree of physiological synchrony in a team
  • • Degree of stability in patterns of brain activation across team members over time
  • • Patterns of interaction
  • • Level of activity
  • • Physical movements

Representative

Findings

  • • Experienced submarine operator teams show more diverse neural physiological organization during team performance which may indicate greater cognitive flexibility in the team (Gorman et al., 2016)
  • • Higher level of cardiovascular arousal across team resulted in worse team performance (Walker, Muth, Switzer, & Rosopa, 2013)
  • • Higher perception of mental exertion in nurses associated with greater time spent in high activity non-service areas (Rosen et al„ 2018)
  • • Close proximity of team members is associated with fewer emails exchanged between them (Olgum et al„ 2019)

Linguistic Communication

Paralinguistic Communication

  • • Lexical analysis/dictionary-based methods
  • • Supervised learning
  • • Generative language modeling
  • • Vocal Features
  • • Communication flow
  • • Gesture and posture
  • • Facial expression and gaze behavior
  • • Frequency of word category use
  • • Linguistic style matching (LSM)
  • • Speech act/functional coding counts
  • • Anticipation ratio
  • • Semantic content and similarity
  • • Speech energy
  • • Pitch
  • • Rate/tempo
  • • Speech dominance of members
  • • Frequency of gesture and posture associated with emotional states
  • • Degree of postural sway
  • • Synchrony in gaze behavior
  • • LSM predicts team cohesion and perfonnance outcomes (Gonzales, Hancock. & Pennebaker. 2010)
  • • High-performing teams have a higher anticipation ratio (Gontar etal.,2017)
  • • Egalitarian turn taking positively predicts team task outcomes or collective intelligence (Woolley et al.. 2010)
  • • Synchrony in postural sway negatively predicts team cohesion (Strang et al., 2014)

TPD has also been used as an indicator of cognitive readiness. Walker et al. (2013) measured cardiovascular activity when participants were engaged in a process control simulation. The simulation involved a team of two participants monitoring five tanks. Each participant was responsible for simultaneously monitoring levels of three parameters on two tanks; the fifth tank was jointly monitored. Walker and colleagues manipulated the difficulty level of the task by changing the frequency with which individual and team tanks had to be monitored. Results showed that higher levels of cardiovascular arousal across the team resulted in worse team performance, and that team cardiovascular activity accounted for 10% of variance in team error on the process control task. Overall, however, team performance was better predicted by individual team members’ cardiovascular activity.

The study of TPD offers a new window into team cognition. TPD can be used to study variables across the IMO framework of studying teams (Kazi et al., 2019). Developments in relatively unobtrusive sensors capable of tracking physiological states can help accelerate study in TPD. This is a promising time to establish guidelines in the use of sensor technology in capturing TPD. These guidelines can help direct not only appropriate data collection and analysis methods, but also guide the linkage between team cognition and team member physiology.

Location and Activity Sensing

Radiofrequency identification (RFID) and infrared sensors are used in creating low-cost sensors to capture location and activity data. Rosen et al. (2018) investigated nursing workload in the intensive care unit by using sensors to capture location, activity, and speech patterns, and focus groups and perceptions of physical and mental exertion. Data from the sensors, focus groups, and surveys was interpreted in the context of patient census during each shift and patient acuity. Rosen and colleagues found that perceptions of mental and physical workload were differentially affected by individual and unit-level activity. Higher perceptions of mental exertion were predicted by noise levels in patient rooms, greater time spent in high- activity non-service areas, high-intensity speech that occurred close together with periods of sparser speech activity, and a high number of patients on insulin drip. On the other hand, higher perceived physical exertion was predicted by high levels of noise in areas containing patient supplies, unpredictable physical movements that occurred in close succession, greater time spent in nursing areas and more time spent speaking outside work areas, a negative interaction between noise in service areas and higher activity in patient rooms, and mean speech level at nursing stations and average patient load.

Proximity sensors can also be used to investigate team communication and interaction. Olguin and colleagues (2009) investigated patterns of face-to-face and email communication for 22 office workers though activity tracking sensors and email interactions. They found that being in close proximity with team members was associated with fewer emails exchanged with those members. Interestingly, a study of interaction of individuals in an organization found that changing the office to an open layout resulted in a significant reduction in face-to-face interaction and an increase in electronic communication (Bernstein & Turban, 2018).

Team communication is an essential process for all types of organizations (Marlow, Lacerenza, Paoletti, Burke, & Salas, 2018). The words that are used to communicate information comprise of the linguistic aspects of communication. The most common method for measuring linguistic communication is using self-report and content analysis of communication. Content analysis has traditionally been done by the manual coding of transcripts in order to categorize different measures of communication (Brauner, Boos, & Kolbe, 2018). In the following section, we discuss emerging methods of linguistic analysis that are done unobtrusively and do not depend on burdensome manual coding of interaction. These methods include dictionary-based lexical analysis, supervised learning, and generative (or unsupervised learning) methods (Grimmer & Stewart, 2013).

Lexical or dictionary-based methods require the use of pre-defined lists of words and phrases that are associated with known constructs of interest. The degree to which a specific construct is present in a team’s communication depends on the rate at w'hich these key words or phrases appear in the team’s communication. Tausczik and Pennebaker (2010) review' Linguistic Inquiry and Word Count (LIWC), a well- validated example of this approach. LIWC is a text analysis tool that counts words and phrases in psychologically meaningful ways in order to provide cues to different constructs such as thought processes, intentions, motivations, and emotional states (Tausczik & Pennebaker, 2010). While LIWC has been a popular tool among researchers interested in this domain, there are a multitude of novel tools available, especially for sentiment analysis which quantifies the emotional valence and intensity in communication content (Gilbert & Hutto, 2014). Dictionary-based methods such as LIWC provide measures of domain-independent language use including verb tense or pronoun use. The main advantage to such a technique is its broad applicability across teams and tasks done by teams. One disadvantage of this technique is that it does not capture any domain-specific or technical content that is used in language. A laboratory study of linguistic style matching (LSM), which is the degree to which members in a team mimic one another’s use of function words or speech rate, positively predicts cooperation, group cohesion, and team performance (Gonzales, Hancock, & Pennebaker, 2010). In addition, lexical analysis has shown predictive validity for teams in complex, real-world tasks. The patterns of general language use, such as the use of first person plural, was found to be correlated w'ith error rates amongst commercial aircrews (Sexton & Helmreich, 2000). Similarly, there is evidence that both general patterns of language use and task domain-specific communication are related to performance outcomes for spaceflight teams (Fischer, McDonnell, & Orasanu, 2007).

Supervised learning methods use a range of machine learning techniques to automate the coding of text (Evans & Aceves, 2016). These machine learning algorithms initially require the input of a training set of documents that have been previously coded by humans as w'ell as a validation process using an additional set of documents that were not initially included in the training set. Supervised learning methods are most appropriate when there is a well-defined coding scheme and it is feasible to code a subset, although not the complete set, of documents. The advantage of these supervised learning methods is their potential to reduce the burden that is normally felt during manual coding of interaction.

Generative (or unsupervised learning) methods are latent variable modeling techniques that are applied in natural language processing (NLP) and machine learning tasks. These approaches work using previously unclassified documents. As an individual generates a speech or a document, they have in mind certain ideas, such as the subject that they are writing or speaking about. The idea(s) that a person has in mind may influence the likelihood of words chosen to help describe that idea (Landauer et al., 1998). Generative modeling identifies a set of topics that describe a set of documents, such as latent variables produced from a set of documents or speech acts. One such generative modeling technique is latent semantic analysis (LSA), which infers expected relationships between contextual usages of words in a discourse. Team performance outcomes was predicted with a reasonable degree of accuracy (correlation of r = .63) using LSA algorithms (Gorman, Foltz, Kiekel, Martin, & Cooke, 2003; Martin & Foltz, 2004). In addition, LSA revealed higher semantic similarity among high-performing teams (Gorman et al., 2016).

These methods each have the potential to contribute to team interaction measurement systems in different ways. Dictionary-based methods are popular because they are highly generalizable and relatively easy to use, but fail to capture context-specific characteristics of speech. Supervised learning methods need pre-existing categories, large training sets of coded documents, and a complex validation process, but they can have the potential to expand the throughput of a human coding team. Generative models such as topic modeling and LSA do not require initial coding like supervised learning methods, but are atheoretical and model diagnostics remain underdeveloped (Chuang, Gupta, Manning, & Heer, 2013). There is a rich community of researchers actively working on further developing these methods, so they are bound to grow in power and ease of use.

Paralinguistic Features of Communication

Paralinguistic aspects of communication are currently not frequently researched within work team settings. However, there is an extensive literature that team researchers can draw' from in order to better measure, understand, and improve team communication (Gatica-Perez, 2009). Measuring paralinguistic features of communication include approaches for capturing vocal features, communication flow, gesture and posture, and facial expressions and gaze behavior.

Vocal features of potential interest to investigators that measure team communication include pitch, tempo, and energy (Vinciarelli, Pantic, & Bourlard, 2009). These communication features are validated as indicators of personality, perceptions of speaker competence, and persuasiveness (Schuller et al., 2015). Prosody, which includes different attributes of speech delivery such as stress and intonation, and voice quality distinguish between the most and least dominant group members (Charfuelan, Schroder, & Steiner, 2010). Principal components analysis (PCA) and support vector machines (SVM) have been used to identify vocal features associated with other team factors such as the speaker role (Charfuelan & Schroder, 2011). SVM and naive classifiers were used alongside human annotation and coding in order to estimate cohesion using nonverbal communication behavior (Hung & Gatica-Perez, 2010). Maximum overlap of speaking rate was found to be significantly higher in cohesive teams, where there was active participation among team members. Furthermore, nonverbal cues like prosody, speaking activity, and variation in energy (i.e., loudness heard by ear) and visual nonverbal features like head activity predict emergent leadership (Sanchez-Cortes, Aran, Mast, & Gatica-Perez, 2012).

Communication flow includes the temporal dynamics of communication acts. This is possibly the most extensively researched paralinguistic aspect of communication within work team settings. Communication flow involves the overall assessments of turn-taking behavior (e.g., speech dominance of individual members compared to the egalitarian sharing of turns; Woolley, Chabris, Pentland, Hashmi, & Malone, 2010), the degree of stability in turn taking behavior (Gontar, Fischer, & Bengler, 2017), and the occurrence of specific patterns of interaction (Tschan, 2002). Unobtrusive measures of communication flow can be derived from a variety of data streams, including audio recordings, automated speech detection systems (which identify segments of paralinguistic communication from microphones but do not actually record the content of communication in order to protect privacy in work settings), and activity traces (e.g., email, paging, phone, chats, and access logs in shared information systems). An example of measuring communication flow is an experimental paradigm that used a push-button-to-talk technique which allowed for tracking conversational flow (Gorman, Hessler, Amazeen, Cooke, & Shope, 2012). Gorman and colleagues showed that information from the communication device, which looked at who was selecting to talk to whom and when, provided important information about team interaction. Another study also examined different aspects of communication flow such as pauses between individual turns, pauses between floor exchanges, turn lengths, and overlapping speech, among others, using supervised machine learning techniques (Hung & Gatica-Perez, 2010). There were a number of significant findings for identifying highly cohesive teams. For example, it was found that total pause time represents how actively attentive members of the team are to one another. In addition, total overlap in communication was expected to be negatively correlated with cohesion. However, it turns out that overlap is actually a feature of highly cohesive teams because it indicates good rapport when team members are able to finish one another’s sentences.

Gesture and posture includes the communication of emphasis and intent, as well as the emotional states of team members (Vinciarelli et ah, 2009). Data for gesture and posture data can be generated through multiple methods, such as the analysis of video data or equipping individuals with sensors to capture position, movement, and muscle activation. There have been some studies which link gesture and posture within teams to specific outcomes or other constructs of interest. First, synchrony in postural sway negatively predicts team cohesion (Strang, Funke, Russell, Dukes, & Middendorf, 2014). In addition, it was found that authoritarian leaders are likely to move their arms more frequently than considerate leaders, while considerate leaders are more likely to imitate posture changes and head nods of their team members (Feese, Arnrich, Troster, Meyer, & Jonas, 2012). Mimicry is generally considered an indicator of group cohesion. These paralinguistic features of communication provide another opportunity to expand team communication measurement toolbox.

Facial expression and gaze behavior are powerful for their potential to communicate the attentional focus and emotional states within the team. Facial expressions can be coded using video data or measures of facial muscle activity captured by electromyography (EMG). Gaze behavior can be captured through eye-tracking systems or inferred from the position of the head using video data. Interestingly, synchrony of facial expressions between members in a team measured via video analysis or facial muscle activity is positively predictive of team cohesion (Mpnster, Hakonsson, Eskildsen, & Wallot, 2016) and task performance outcomes (Chikersal, Tomprou, Kim, Woolley, & Dabbish, 2017). In addition, teams with low levels of synchrony in facial muscle activity were found to be more likely to adapt their strategies across different performance episodes (Mpnster et al., 2016). In a study of airline crews, synchrony in gaze behavior, as an indicator of shared attentional focus, was predictive of performance outcomes (Gontar & Mulligan, 2016). In another study of student air traffic control teams, investigation of individual and team situation awareness (SA) suggested that eye tracking had the potential to be used to measure individual and team SA by using the co-occurrence of information seeking and acquisition to predict specific aspects of simulation (Hauland, 2008).

EVENT-BASED METHODS FOR UNOBTRUSIVE MEASUREMENT OF TEAM COGNITION

One of the main benefits of the emerging methods detailed above is high-volume, high-frequency data collection. However, this comes with challenges in analysis and interpretation, as this data is information rich, but often missing the context needed to generate understanding and meaningful insights into team performance (Rosen, Dietz, & Kazi, 2018). In field settings, this can be addressed to some degree by looking at patterns across sensor streams, or by targeting the use of traditional observational or self-report measures at identified critical or otherwise sampled time periods (Beal, 2015). However, the controlled but realistic environment of simulations provides more options for rigorous assessment. In this section, we briefly introduce event-based methods, discuss their value as an approach to unobtrusive measures, and provide example applications.

Simulation has been used extensively to both train and assess teams across a range of industries and research communities (Salas, Rosen, Held, & Weissmuller, 2009). The event-based approach to assessment is based on the premise that training or measurement objectives can drive the design scenario events that represent opportunities for team members to demonstrate and raters or trainers to observe behaviors targeted for assessment and development (Fowlkes, Dwyer, Oser, & Salas, 1998). A scenario event can be anything under the control of the scenario designers including shifting task demands, manipulation of information resources, or changes to team composition (Rosen et al., 2008). From a test development perspective, events represent the items that team members respond to with their interactions. Traditionally, event-based measurement links behavioral checklists to critical task situations experienced by the team as a way to focus rater attention on key moments. We argue here that creating critical events within scenarios also provides an anchor for the development of unobtrusive measures of team cognition by adding needed context to interpret the large volume of data generated by these sensor systems. As the field of unobtrusive measurement advances, there likely will be some indicators or markers of the level or quality of different team cognition constructs that are valid across team and task types. It is also likely that there will be unique indicators within different contextual constraints. This is similar to the notion of team and task specific and team and task generic competencies (Cannon-Bowers, Tannenbaum, Salas, & Volpe, 1995). Event-based methods provides an organizing framework for thinking about how different features of unobtrusive measurement may relate to team cognition constructs under different conditions. In Figure 5.1, we provide a brief overview of this framework and follow with more detailed examples below.

We introduce example event types and associated unobtrusive markers below. These are based on empirical findings reviewed above, but speculative in their connection to specific team cognition constructs as these relationships need to be further tested. First, the leadership role structure within the team can be manipulated as a scenario event and provide opportunities to measure TMM development. For example, scenarios can include conditions where formal leadership roles move from one team member to another (e.g., moving from one phase of work to another; induction of a patient led by an anesthesia provider in a surgical procedure to the actual procedure led by a surgeon), or where a novel situation is presented to the team without a clear leadership structure, yet an emergent leader is necessary. While equality of turn taking appears to be a general marker of effective team information processing (Woolley et al., 2010), teams exhibiting a shift in speech dominance such that the new or emergent leader has a higher proportion of speaking time may have more clearly recognized and adapted to the situation (Chaffin et ah, 2017). This pattern of behavior could be indicative of better TMM of role structure within the team. Second, task workload demands within a simulation are frequently manipulated as critical events. Various makers of individual physiology are used to assess workload and the degree of synchrony in team physiological dynamics in response to shifts in task workload demand may be a valid marker of TSA, indicating shared perceptions of and similar reactions to new demands. Third, team member turnover or the addition of new team members (real or confederates) provides another opportunity to observe the team’s development of a shared mental model of communication norms by capturing linguistic markers of style and their convergence or lack thereof. Fourth, disruptions to communication structures within the teams is common in military simulations (e.g., a team where all members could directly communicate shifts to a hub and spoke communication structure with a simulated communication channel outage). Measures of team member activity following this type of event (e.g., changes to use of available synchronous or asynchronous communication channels) is another opportunity to measure the TSA component of team adaptation (Burke, Stagl, Salas, Pierce, & Kendall, 2006).

In sum, the control over the environment afford by simulation-based assessment provides opportunities to focus unobtrusive measures, align them with traditional observational approaches, and create needed context for interpretation. As the evidence base grows, the repository of team and task general and team and task specific unobtrusive markers can be refined and serve as a valuable resource for practitioners and researchers.

Organizing Framework for Event-Based Unobtrusive Measurement of Team Cognition

FIGURE 5.1 Organizing Framework for Event-Based Unobtrusive Measurement of Team Cognition.

CONCLUSION AND FUTURE DIRECTIONS

Team cognition plays a crucial role in team effectiveness and performance (DeChurch & Mesmer-Magnus, 2010). Decades of team cognition research has led to a multitude of measurement strategies that have enriched our understanding of different team cognition constructs (Wildman et al., 2014). The emergence of new technological tools such as wearables and other unobtrusive measurement tools provide us the opportunity to elevate our understanding of the science behind team cognition. These technological advances make it feasible for researchers to analyze data streams larger than ever before due to reduction in the burden for capturing this data. In addition, these tools can help provide more context to understanding the quality of cognition within teams, especially when used in conjunction with traditional measures. The four different categories of data streams reviewed in this chapter (i.e., physiological activity, linguistic and paralinguistic communication, and location and activity sensing) are not exhaustive but rather representative of the type of promising research that is developed in this arena.

For researchers and practitioners hoping to assess team cognition with the measurement tools discussed here, there are several considerations to keep in mind. First, it is important to recognize the novelty of many of the obtrusive measurement techniques. While there is a great deal of enthusiasm for current and future discoveries made possible by these innovations, these techniques need to be further developed with an emphasis on improving reliability. Second, organizations that seek to measure team cognition should use unobtrusive measurement techniques alongside more traditional measurement techniques that have been validated. Third, further studies of unobtrusive measurement in team cognition are needed in order to validate these methods for team cognition constructs. Improvements in team and cognitive task analysis that are better linked to unobtrusive methods should help by allowing us to know specific markers to look for in event-based approaches. This is undoubtedly an exciting time for researchers and practitioners as the presence of innovations has sparked a new era of discovery that will only strengthen our understanding in the coming years.

ACKNOWLEDGMENTS

This work was partially funded by a grant from the National Aeronautics and Space Administration (Grant # NNX17AB55G; PI: Rosen).

REFERENCES

Beal, D. J. (2015). ESM 2.0: State of the art and future potential of experience sampling methods in organizational research. Annual Review of Organizational Psychology and Organizational Behavior, 2(1), 383-407.

Bernstein, E. S., & Turban, S. (2018). The impact of the “open” workspace on human collaboration. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1753), 20170239.

Bolstad, C. A., & Endsley, M. R. (2003). Measuring shared and team situation awareness in the army’s future objective force. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 47, pp. 369-373). Los Angeles, CA: Sage Publications. Brauner, E., Boos, M., & Kolbe, M. (2018). The Cambridge handbook of group interaction analysis. Cambridge: Cambridge University Press.

Burke, C. S., Stagl, К. C., Salas, E„ Pierce, L., & Kendall, D. (2006). Understanding team adaptation: A conceptual analysis and model. Journal of Applied Psychology, 9/(6), 1189.

Cannon-Bowers, J. A., & Salas, E., & Converse, S. (1993). Shared mental models in expert team decision making. In N. J. Castellan (Eds.), Individual and group decision making: Current issues (p. 221). Hilldale, NJ: Erlbaum.

Cannon-Bowers, J. A., Tannenbaum, S. I„ Salas, E., & Volpe, С. E. (1995). Defining competencies and establishing team training requirements. Team Effectiveness and Decision Making in Organizations, 333, 380.

Chaffin. D., Heidi, R., Hollenbeck, J. R.. Howe, M.. Yu. A.. Voorhees, C., & Calantone, R. (2017). The promise and perils of wearable sensors in organizational research. Organizational Research Methods, 20(1), 3-31.

Chanel, G., Kivikangas, J. M, & Ravaja, N. (2012). Physiological compliance for social gaming analysis: Cooperative versus competitive playw-.” Interacting with Computers, 24(A), 306-316.

Charfuelan, M., & Schroder, M. (2011). Investigating the prosody and voice quality of social signals in scenario meetings. In International conference on affective computing and intelligent interaction (pp. 46-56). London: Springer.

Charfuelan, M., Schroder, M., & Steiner, I. (2010). Prosody and voice quality of vocal social signals: The case of dominance in scenario meetings. In Eleventh annual conference of the international speech communication association. Makuhari, Japan. Retrieved from https://www.isca-speech.org/archive/interspeech_2010/il0_2558.html

Chikersal, P, Tomprou, M.. Kim. Y. J.. Woolley. A. W„ & Dabbish, L. (2017). Deep structures of collaboration: Physiological correlates of collective intelligence and group satisfaction. In CSCW(pp. 873-888). Portland. OR: ACM.

Chuang, J., Gupta, S., Manning, C., & Heer, J. (2013). Topic model diagnostics: Assessing domain relevance via topical alignment. In Proceedings of the 30th International conference on machine learning (pp. 612-620). Atlanta, GA: JMLR-W & CP.

Cooke, N. J., Gorman, J. C., Myers, C. W., & Duran, J. L. (2013). Interactive team cognition. Cognitive Science, 37(2), 255-285.

Cooke, N. J., Gorman, J. C., & Rowe, L. J. (2008). An ecological perspective on team cognition. In Team effectiveness in complex organizations (pp. 191-216). Abingdon: Routledge.

Cooke, N. J., Kiekel, P. A., & Helm, E. E. (2001). Measuring team knowledge during skill acquisition of a complex task. International Journal of Cognitive Ergonomics, 5(3), 297-315.

DeChurch, L. A., & Mesmer-Magnus, J. R. (2010). The cognitive underpinnings of effective teamwork: A meta-analysis. Journal of Applied Psychology, 95(1), 32.

Dong, A. (2005). The latent semantic approach to studying design team communication. Design Studies, 26(5), 445-461.

Edwards, B. D„ Day. E. A., Arthur Jr., W.. & Bell, S. T. (2006). Relationships among team ability composition, team mental models, and team performance. Journal of Applied Psychology, 9/(3), 727.

Endsley, M. R. (1988). Design and evaluation for situation awareness enhancement. In Proceedings of the human factors society annual meeting (Vol. 32, pp. 97-101). Los Angeles, CA: Sage Publications.

Evans, J. A., & Aceves, P. (2016). Machine translation: Mining text for social theory. Annual Review of Sociology, 42.

Feese, S., Arnrich, B., Troster, G., Meyer, B., & Jonas, K. (2012). Quantifying behavioral mimicry by automatic detection of nonverbal cues from body motion. In Privacy, Security, Risk and Trust (PASSAT), 2012 International Conference on and 2012 International Confernece on Social Computing (SocialCom) (pp. 520-525). Piscataway, NJ: IEEE.

Fiore, S. M., Rosen, M. A., Smith-Jentsch, K. A., Salas, E., Letsky, M, & Warner, N. (2010). Toward an understanding of macrocognition in teams: Predicting processes in complex collaborative contexts. Human Factors, 52(2), 203-224.

Fischer, U., McDonnell, L., & Orasanu, J. (2007). Linguistic correlates of team performance: Toward a tool for monitoring team functioning during space missions. Aviation, Space, and Environmental Medicine, 78(5), B95.

Fisher, D. M., Bell. S. T, Dierdorff, E. C„ & Belohlav, J. A. (2012). Facet personality and surface-level diversity as team mental model antecedents: Implications for implicit coordination. Journal of Applied Psychology, 97(4), 825.

Fowlkes, J., Dwyer, D. J., Oser, R. L., & Salas, E. (1998). Event-based approach to training (EBAT). The International Journal of Aviation Psychology, 8(3), 209-221.

Gatica-Perez, D. (2009). Automatic nonverbal analysis of social interaction in small groups: A review'. Image and Vision Computing, 27(12), 1775-1787.

Gilbert, C. J., & Hutto, E. (2014). Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth International Conference on Weblogs and Social Media (ICWSM-I4). Retrieved April 20, 2016 from http://comp.social.gatech. edu/papers/icw'sml4.vader.hutto.pdf

Gontar, P., Fischer, U., & Bengler, K. (2017). Methods to evaluate pilots’ cockpit communication: Cross-recurrence analyses vs. speech act—based analyses. Journal of Cognitive Engineering and Decision Making, 11(4), 337-352.

Gontar, R, & Mulligan, J. B. (2016). Cross recurrence analysis as a measure of pilots’ visual behaviour. In A. Droog, M. Schwartz, & R. Schmidt (Eds.), Proceedings of the 32nd conference of the European association for aviation psychology. Groningen, NL.

Gonzales, A. L., Hancock, J. T, & Pennebaker, J. W. (2010). Language style matching as a predictor of social dynamics in small groups. Communication Research, 37(1), 3-19.

Gorman, J. C., Cooke, N. J., & Winner, J. L. (2006). Measuring team situation aw'areness in decentralized command and control environments. Ergonomics, 49(12-13), 1312-1325.

Gorman. J. C.. Foltz. P. W.. Kiekel, P. A.. Martin. M. J.. & Cooke. N. J. (2003). Evaluation of latent semantic analysis-based measures of team communications content. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 47, pp. 424-428). Los Angeles, CA: Sage Publications.

Gorman. J. C„ Hessler, E. E.. Amazeen. P. G.. Cooke, N. J.. & Shope. S. M. (2012). Dynamical analysis in real time: Detecting perturbations to team communication. Ergonomics, 55(8), 825-839.

Gorman, J. C., Martin, M. J., Dunbar, T. A., Stevens, R. H., Galloway, T. L., Amazeen, P. G., & Likens, A. D. (2016). Cross-level effects between neurophysiology and communication during team training. Human Factors, 5S(1), 181-199.

Grimmer, J., & Stewart, В. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 2/(3), 267-297.

Hamilton, K., Mancuso, V., Minotra, D., Hoult, R., Mohammed, S., Parr, A.,. . . McNeese, M. (2010). Using the NeoCITIES 3.1 simulation to study and measure team cognition. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 54, no. 4, pp. 433-437). Los Angeles, CA: Sage Publications.

Hauland, G. (2008). Measuring individual and team situation aw'areness during planning tasks in training of en route air traffic control. The International Journal of Aviation Psychology, 18(3), 290-304.

Hazlehurst, B., McMullen, С. K., & Gorman, P. N. (2007). Distributed cognition in the heart room: How situation awareness arises from coordinated communications during cardiac surgery. Journal of Biomedical Informatics, 40(5), 539-551.

Hellar, D. B., & McNeese, M. (2010). NeoCITIES: A simulated command and control task environment for experimental research. In Proceedings of the human factors and ergonomicssocietyannualmeeting (Vol. 54, no. 13, pp. 1027-1031). Los Angeles, CA: Sage Publications.

Hollan, J., Hutchins, E., & Kirsh, D. (2000). Distributed cognition: Toward a new foundation for human-computer interaction research. ACM Transactions on Computer-Human Interaction (TOCHI), 7(2), 174-196.

Hung, H., & Gatica-Perez, D. (2010). Estimating cohesion in small groups using audio-visual nonverbal behavior. IEEE Transactions on Multimedia, 12(6), 563-575.

Kazi, S., Khaleghzadegan, S., Dinh, J. V., Shelhamer, M. J., Sapirstein, A., Goeddel, L. A.,

. . . Rosen, M. A. (2019). Team physiological dynamics: A critical review. Human Factors, 0018720819874160.

Landauer, T. K., Foltz, P. W„ & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2-3), 259-284.

Lewis, K. (2003). Measuring transactive memory systems in the field: Scale development and validation. Journal of Applied Psychology, 88(4), 587.

Li. X., Yao, D.. Pan. X., Johannaman, J., Yang, J., Webman. R.,... Burd, R. S. (2016). Activity recognition for medical teamwork based on passive RFID. In 2016 IEEE international conference on RFID (RFID) (pp. 1-9). Orlando, FL: IEEE.

Mancuso, V. F., & McNeese, M. (2013). TeamNETS: Scaled world simulation for distributed cyber teams. In International conference on human-computer interaction (pp. 509- 513). Berlin, Heidelberg: Springer.

Marks. M. A.. Sabella. M. J., Burke, C. S., & Zaccaro, S. J. (2002). The impact of crosstraining on team effectiveness. Journal of Applied Psychology, 87( 1), 3.

Marlow, S. L., Lacerenza, C. N.. Paoletti, J., Burke, C. S., & Salas, E. (2018). Does team communication represent a one-size-fits-all approach?: A meta-analysis of team communication and performance. Organizational Behavior and Human Decision Processes, 144, 145-170.

Martin, M. J., & Foltz, P. W. (2004). Automated team discourse annotation and performance prediction using LSA. In Proceedings of HLT-NAACL 2004: Short papers (pp. 97-100). Boston, MA: Association for Computational Linguistics.

Mathieu, J. E.. Heffner, T. S.. Goodwin, G. F., Salas, E., & Cannon-Bowers. J. A. (2000). The influence of shared mental models on team process and performance. Journal of Applied Psychology, 85(2), 273.

Mohammed, S., Ferzandi, L., & Hamilton, K. (2010). Metaphor no more: A 15-year review of the team mental model construct. Journal of Management, 36(4), 876-910.

Mohammed, S., Hamilton, K., Tesler, R., Mancuso, V., & McNeese, M. (2015). Time for temporal team mental models: Expanding beyond “what” and “how” to incorporate “when”. European Journal of Work and Organizational Psychology, 24(5), 693-709.

Monster, D., Hakonsson, D. D., Eskildsen, J. K., & Wallot, S. (2016). Physiological evidence of interpersonal dynamics in a cooperative production task. Physiology & Behavior, 156, 24-34.

McNeese, M., McNeese, N. J., Endsley, T., Reep, J., & Forster, P. (2017). Simulating team cognition in complex systems: Practical considerations for researchers. In Advances in neuroergonomics and cognitive engineering (pp. 255-267). Cham: Springer.

Olguln, D. O., Waber. B. N.. Kim, T., Mohan, A.. Ага, K., & Pentland, A. (2009). Sensible organizations: Technology and methodology for automatically measuring organizational behavior. IEEE Transactions on Systems, Man, and Cybernetics, Part В (Cybernetics), 39(1), 43-55.

Rosen, M. A., Dietz, A. S., & Kazi, S. (2018). Beyond coding interaction. In The Cambridge handbook of group interaction analysis (Cambridge handbooks in psychology, pp. 142-162). Cambridge: Cambridge University Press. https://doi. org/10.1017/9781316286302.009.

Rosen, M. A., Dietz, A. S., Lee, N.. Wang, I.-J., Markowitz, J., Wyskiel, R. M.....Gurses,

A. P. (2018). Sensor-based measurement of critical care nursing workload: Unobtrusive measures of nursing activity complement traditional task and patient level indicators of workload to predict perceived exertion. PLoS One, /3(10), e0204819.

Rosen, M. A., Salas, E., Wilson, K. A., King, H. B., Salisbury, M., Augenstein, J. S., . . . Birnbach, D. J. (2008). Measuring team performance in simulation-based training: Adopting best practices for healthcare. Simulation in Healthcare, 3(1), 33-41.

Salas, E., Rosen, M. A., Held, J. D., & Weissmuller, J. J. (2009). Performance measurement in simulation-based training: A review and best practices. Simulation & Gaming, 40(3), 328-376.

Sanchez-Cortes, D„ Aran, O., Mast, M. S., & Gatica-Perez, D. (2012). A nonverbal behavior approach to identify emergent leaders in small groups. IEEE Transactions on Multimedia, 14(3), 816-832.

Schuller, B., Steidl, S., Batliner, A.. Noth. E., Vinciarelli, A., Burkhardt. F.,. . . Bocklet. T. (2015). A survey on perceived speaker traits: Personality, likability, pathology, and the first challenge. Computer Speech & Language, 29(1), 100-131.

Sexton, J. B., & Helmreich, R. L. (2000). Analyzing cockpit communications: The links between language, performance, error, and workload. Human Performance in Extreme Environments, 5(1), 63-68.

Smith-Jentsch. K. A.. Campbell, G. E.. Milanovich, D. M„ & Reynolds, A. M. (2001). Measuring teamwork mental models to support training needs assessment, development, and evaluation: Two empirical studies. Journal of Organizational Behavior: The International Journal of Industrial, Occupational and Organizational Psychology and Behavior, 22(2), 179-194.

Smith-Jentsch, K. A., Mathieu, J. E., & Kraiger, K. (2005). Investigating linear and interactive effects of shared mental models on safety and efficiency in a field setting. Journal of Applied Psychology, 90(3), 523.

Stevens, R. H., & Galloway, T. L. (2014). Toward a quantitative description of the neurodynamic organizations of teams. Social Neuroscience, 9(2), 160-173.

Stevens, R. H., Galloway, T. L., Halpin, D.. & Willemsen-Dunlap, A. (2016). Healthcare teams neurodynamically reorganize when resolving uncertainty. Entropy, /3(12), 427.

Strang, A. J., Funke. G. J., Russell. S. M., Dukes. A. W.. & Middendorf, M. S. (2014). Physio- behavioral coupling in a cooperative team task: Contributors and relations. Journal of Experimental Psychology: Human Perception and Performance, 40(1), 145.

Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24-54.

Toppi, J.. Borghini, G„ Petti, M.. He, E. J., Giusti. V. D„ He, В.....Babiloni, F. (2016).

Investigating cooperative behavior in ecological settings: An EEG hyperscanning study. PLoS One, 11(4), e0154236.

Tschan, F. (2002). Ideal cycles of communication (or Cognitions) in triads, dyads, and individuals. Small Group Research, 33(6), 615-643.

Vinciarelli, A., Pantic, M., & Bourlard, H. (2009). Social signal processing: Survey of an emerging domain. Image and Vision Computing, 27(12), 1743-1759.

Walker, A. D.. Muth. E. R., Switzer III. F. S., & Rosopa, P. J. (2013). Predicting team performance in a dynamic environment: A team psychophysiological approach to measuring cognitive readiness. Journal of Cognitive Engineering and Decision Making, 7(1), 69-82.

Webber, S. S., Chen, G., Payne, S. C., Marsh, S. M., & Zaccaro, S. J. (2000). Enhancing team mental model measurement with performance appraisal practices. Organizational Research Methods, 3(4), 307-322.

Wegner, D. M. (1987). Transactive memory: A contemporary analysis of the group mind. In Theories of group behavior (pp. 185-208). London: Springer.

Wildman, J. L., Salas, E., & Scott, С. P. R. (2014). Measuring cognition in teams: A crossdomain review. Hitman Factors, 56(5), 911-941.

Woolley, A. W.. Chabris, C. F., Pentland, A., Hashmi. N.. & Malone. T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004), 686-688.

Yarkoni, T. (2012). Psychoinformatics: New horizons at the interface of the psychological and computing sciences. Current Directions in Psychological Science, 21(6), 391-397.

Zhang, J., & Norman, D. A. (1994). Representations in distributed cognitive tasks. Cognitive Science, 18(1), 87-122.

 
Source
< Prev   CONTENTS   Source   Next >