Section IV Models and Measures of Cognitive Work

Measuring Team Cognition in Collaborative Systems Integrative and Interdisciplinary Perspectives

Susan Mohammed, Katherine Hamilton,

Vincent Mancuso, Rachel Tesler, and Michael D. McNeese

INTRODUCTION

Increasingly, work is carried out by teams consisting of diverse members who have never interacted previously, but are required to perform complex, dynamic tasks in novel and time-pressurized environments (e.g., Tannenbaum, Mathieu, Salas, & Cohen, 2012). Given the difficulty of this context, the issue of how to enhance team collaboration and performance is increasingly salient in organizational and military settings. Indeed much work has been done in cognitive systems engineering and human-centered design (Lee & Kirlik, 2013) to reflect the active integration of cognition, teamwork, technologies, and context to enable building tangible, resilient, and reliable systems that meet both individual and team demands. Although already a considerable challenge in colocated teams, getting members on the same cognitive page is even more problematic when team members are geographically separated (e.g., Gilson, Maynard, Young, Vartiainen, & Hakonen, 2015).

Addressing these research challenges, our interdisciplinary research team investigated antecedents and consequences of multiple types of team cognition toward the goal of improving team coordination and performance in distributed decision-making teams. Contemporary work in cognitive systems engineering interweaves and is often coupled to naturalistic decision making and team cognition (e.g., see Glantz, this volume, Chapter 15). The living laboratory framework (McNeese, 1996; McNeese, Perusich, & Rentsch, 2000) provides a foundation for integrating and aggregating multiple methodological approaches to gain holistic knowledge for designing cognitive systems that are practically effective and useful. The purpose of this chapter is to provide demonstration and use of the living laboratory framework especially as relevant for problems involving team cognition where (1) theory, modeling, and measurement; and (2) scaled-world simulation are of high priority. Within the living laboratory framework using NeoCITIES as a scaled-world simulation (McNeese, Mancuso, McNeese, Endsley, & Forster, 2013), we conducted four programmatic experiments investigating the influence of several interventions on team cognition and subsequent team outcomes. This work was supported by the Office of Naval Research and had the following three overall research objectives:

• 1. To integrate and empirically investigate multiple types of team cognition (information sharing, team SA, and TMMs) that occur in complex systems to understand their differential effects on team outcomes.
• 2. To infuse a temporal focus (deadlines, pacing, and sequencing) into the study of team cognition and team outcomes, to more accurately reflect the context of organizational and military teams.
• 3. To investigate the effectiveness of various interventions (storytelling, team reflexivity, and individual reflexivity) designed to improve team cognition and collaboration in distributed teams.

These research objectives contributed to the team literature in several ways. First, learning how various forms of team cognition integratively influence team outcomes has been identified as a critical research need (e.g., Mohammed, Ferzandi, & Hamilton, 2010; Salas & Wildman, 2009). As such, we empirically investigated information sharing, SA, and TMMs in the same set of studies, allowing for a deeper understanding of their differential effects. Second, as team cognition measures have been deemed temporally deficient (Mohammed, Tesler, & Hamilton, 2012) and time has been identified as “perhaps the most neglected critical issue” in team research (Kozlowski & Bell, 2003, p. 364), we infused time into assessments of information sharing, shared SA, and TMMs. In doing so, we answered numerous calls to sharpen the temporal lens used in conducting team studies (e.g., Ancona, Goodman, Lawrence, & Tushman, 2001; Mohammed, Hamilton, & Lim, 2009).

Third, we were the first (to our knowledge) to empirically investigate how the powerful benefits of ad hoc stories could be leveraged as a team-training tool to allow team members to achieve higher levels of shared understanding, and thereby perform more effectively. Although several authors have proposed that storytelling can improve team learning and performance (e.g., Bartel & Garud, 2009; Denning, 2001; Fiore, McDaniel, & Jentsch, 2009), their claims have gone untested because much of the work on storytelling is conceptual (e.g., Denning, 2001), and empirical work has been largely focused at the individual level of analysis (e.g., Ang & Rao, 2008). In combination with storytelling, we also examined reflexivity (a team’s reflection on its performance and future strategies) as a tool to enhance team cognition and collaboration among members.

Fourth, through the integration of team cognition, temporality, and storytelling, we expanded these literatures into previously unexplored avenues identified as potentially theoretically fruitful (e.g., Fiore et al., 2009), but not investigated empirically. For example, we answered previous research calls to broaden the list of empirically verified antecedents of TMMs (e.g., Mohammed et al., 2010) by examining the effects of storytelling on team cognition. Fifth, beyond these theoretical and empirical contributions, NeoCITIES methodologically offered a novel, but ideal, interface to investigate multiple aspects of team cognition because it is easily adaptable and designed to test team cognition and collaborative-system processes.

In this chapter, we review the four studies we conducted as an exemplar of a living laboratory framework integrating cognitive systems, communication technologies, and human-in-the-loop simulation in a virtual setting (McNeese et al., 2013). We begin by providing a brief introduction to the team cognition literature and the need for a temporal focus as well as storytelling and reflexivity as interventions. Next, we describe the methodology of the four experiments, including the NeoCITIES simulation and team cognition measures. After providing an overview of study results, we end with lessons learned, highlighting the benefits of adopting an integrative and interdisciplinary perspective.