Reflections on Team Simulations—Part II: Contemporary Progressions
Michael D. McNeese, Nathaniel J. McNeese, Lisa A. Delise, Joan R. Rentsch, and Clifford E. Brown
Technological Innovations Underlying Teamwork.............................................50
Contextual Perturbations of Teamwork...............................................................53
Current Examples of Distributed Team Cognition...................................................55
Emergency Crisis Management..........................................................................56
Uninhabited Air Vehicles.....................................................................................59
Remote Medicine on the Battlefield....................................................................61
An Overview of Team Cognition Research—Contemporary Perspectives.............62
Challenges and Issues for Simulation.................................................................64
Simulations in Team Cognition Research................................................................64
Ecological Validity of the Simulations................................................................67
Importance of Technology for Simulations.........................................................68
Ecological Affordances in Simulation.................................................................68
Methodologies for Measuring Team and Individual Constructs.........................69
Sustained Areas of Research within Team Simulations...........................................70
Individual and Team Situation Awareness..........................................................70
Trends—Strengths and Limitations....................................................................73
A Conceptual Framework for Distributed Team Cognition and Simulation...........................................................................................................74
Integrative Living Laboratory Extended Framework for Futuristic
Application of Framework to a Specific Case.....................................................77
Scaled World Simulation................................................................................80
One purpose of the Handbook of Distributed Team Cognition is to examine the crossroads of looking back and looking ahead while reviewing contemporary research and practice in how teams address currently complex and challenging problems. Part of these crossroads is the contributions of team simulations in terms of concepts, value, and worth to the scientific community. Hence the previous chapter and this chapter provide a holistic view—from a particular reflection—into how team simulation has transpired and worked through the confines of a given research group (more recently referred to as the MINDS Group).1
Consideration of teamwork in contemporary culture may produce different meanings than what was presented in the previous chapter, which focused more on historical research practices. In today’s world teams are very much prevalent in many kinds of work domains and fields of practice including medical and health, transportation, logistics, maritime operations, information and cybersecurity, offshore operations, manufacturing and production, and aviation, to name a few, and very much in evidence in a variety of organizations, businesses, government, and industry. Although this is not necessarily different from the previous practice of teamwork, there are some distinct changes in teamwork that are notable that lead to a current understanding of distributed team cognition, and why research has evolved given some of these differences. There are two areas that especially stand out as contributors to change in teamwork: (1) technological innovation and (2) contextual complexity.
Technological Innovations Underlying Teamwork
Technological innovation is rampant across contemporary society wherein computational power and digital capability is omnipresent. Information technology developments have coupled power and capability to produce new socio-media tools, cognitive architecture platforms, group interfaces, and collaborative support applications that directly facilitate how people think and act within teams. Many of these innovations are now' provided directly through the internet or through the use of smartphones where apps can be downloaded and quickly used for real work considerations. Computers are allowing a greater diversity of teams to come together from great distances all across the world with much ease. As pointed out in our previous chapter, historically, many traditional teams were limited in that they were typically only in the same place at the same time. The ability to distribute teams at remote distances and have synchronous and asynchronous interaction was limited (primarily to use of phones). The ability to fully integrate members at a distance at the same time was pretty much unavailable. This created intellectual barriers as diverse and multiple perspectives on team problem solving and decision making were subject to who “was in the given context at the time.” Not only was distributed work limited but distributed information was usually not available to teams for use in their work. Information such as text, graphics, video and photographic feeds, and other forms of media and data were limited to just what was available in the local context. As mentioned in the last chapter, research focused on team cognition wherein information and collaborative technology was predicted for the future but not really developed to any extent.
Insightful researchers such as A. Rodney Wellens (1993) simulated team decision making at distance for purposes of comparing different forms of electronic media (text-based messages in computer setups, telephony, and cable-generated video to simulate video teleconferencing). This projected what the future would become 30 years later, which is where we are today, except the innovation of today is far beyond the expectations of 30 years ago. The power of the internet alone and what it has meant in terms of distributed teamwork and distributed information access has been a tremendous boost in changing how people work together. The presence of such apps as Facebook, Twitter, and Instagram have changed the nature of communication, collaboration, coordination, and perception in unique ways. Therein, whereas in the past much of the research focused on team cognition, the research of today is clearly resonant with distributed team cognition. In turn, team simulators must adapt and be representative of real-world technological changes in today’s society rather than just show ad hoc add-ons.
In addition to technology appropriating instant collaboration via digital software applications (e.g., Slack, Zoom, Google Docs) the processes of teamwork/coopera-tive work over the years have been inspired through various theories and approaches resulting in the design and use of various tools/supports (see Schmidt & Bannon, 2013 for review). Whereas in the past these computing technologies, appropriated for team use (e.g., team decision aids), were awkward for the user, today they are seamless, integrative, and have been designed from more user-centric and awareness perspectives (Gross, Stary, & Totter, 2005; Tenenberg, Roth. & Socha, 2016). Our first foray into understanding computer-supported cooperative work emphasized the various approaches that could be taken and how the resultant designs of groupware where representative of those approaches, but in particular how a technology-only perspective was wholly inadequate (McNeese, Zaff, & Brown, 1992). Since then the role of affect, emotion, and beliefs (Hudlicka, 2003), context and field of practice, and user experience have become much more paramount in the design of systems.
Additionally, the collaborative technologies of today emphasize an important concept within distributed cognition—information sharing—whereby distributed teams can not only access reams of information but they can work through distributed shared objects, interactive group interfaces (McNeese, Theodorou, Ferzandi, Jefferson, & Ge, 2002), all with built-in documented updates and traces of who contributed what when and at what time. This creates distributed cognition with high temporal flow present. Many perspectives, much diversity, and multiple levels of intelligence can be easily rendered within these kinds of collaborative suites. However, that is not the total picture, as distributed cognition also encapsulates new formulations of perception combined with information sharing through the use of advanced technologies which have been referred to as virtual presence. Being able to perceive situations remotely is a way to enhance individual and team situation awareness across multiple events. A simple application of this is the vast network of cameras that one can now access online to “see” situations remotely.
During the 2013 Boston Marathon bombing distributed information sharing was of utmost importance to track situations and perform search and rescue activities. This provides a perceptual anchor for communications and joint coordination of activity. The networks of cameras provided source material for use of face recognition technologies which also facilitated identification and tracking of subjects. This is just a simple example but shows how the power of many different views from remote locales can contribute to sensemaking, surveillance, and problem solving. The obvious use of this is in the area of police cognition to “see” a situation unfolding from different temporal and visuospatial perspectives. However, it should be noted that the distributed cognition in this kind of event can also go wrong. Although crowd sourcing of faces and cogent pictures were actively being distributed across sites on the internet (e.g., Reddit), this produced numerous false alerts which can then lead police down the wrong rabbit holes. This is example where distributed cognition was active and engaged but ironically it did not necessarily lead to all the correct outputs. There were also ethical issues at play as people were falsely accused in the haste of trying to identify subjects. This signals that the collective can be more problematic than a simple team and that one must be careful given results obtained. The work our group did in the mid-1980s on information sharing using small and large group displays (McNeese & Brown, 1986) has certainly come a long way in more than 30 years but the premises underlying information sharing, interdependent cognitive processes, and team situation awareness are still intact.
The world of virtual environments has skyrocketed providing teams with a unique interface that can bring people together even though they are remotely located. Virtual environments are created by using multimodal, high bandwidth technologies (such as 3-D interactive graphics, 3-D sound space, interactive touch with virtual controls, and dynamic integration of sensory perception). One of the most rewarding experiences the senior author had during the 1980s at the USAF Harry G. Armstrong Medical Research Laboratory was flying the Super Cockpit that Dr. Thomas Furness had designed and operationalized. Dr. Furness is widely known as the “grandfather of virtual and augmented reality” (Clapway, n.d.). The Super Cockpit was basically an entire virtual cockpit inside of a very large helmet, which was wired with sensors for information flow and interactive touch-feedback. The immersive experience emulated being inside an advanced fighter cockpit. This was a unique kind of simulation as one could use the apparatus to fly and perform a mission, at least in experimental fashion. Technically, the Super Cockpit was originally designed for a single-pilot immersion, however concepts were hypothesized wherein multiple people could engage in a virtual world. While that was primarily a vision of the 1980s, virtual environments are now feasible for teamwork and have been used commercially through such applications as the Second Life® virtual world. Virtual environments and augmented tools have since taken off with new commercially available technologies (e.g., Oculus Rift, Google Glass, HTC Vive) which are predicated on Furness’ early developments and experiments. Virtual and augmented reality are one example of where digital information technology intersects team processes producing distributed team cognition.
The software application of a decade ago termed Second Life is an example of where brilliant 3-D virtual environments could create a medium for team interaction through the use of avatars representing individuals within a constructed world. The avatar that represents “you” can communicate with other avatars in the virtual community, take joint actions based on the affordances that can be picked up in the world, and just “hang out” and engage in social interaction with others. The world also provided certain commercial services embedded in the world, which utilized online financial economies. This unique collaborative technology provided the means for a community of actors to engage in distributed team cognition. The environment (graphics, auditory space, and information services) could be constructed according to the purpose of the interaction. For example, if one was desiring to create a Second Life virtual course that studied ancient Rome, the interface could be designed to emulate the ancient ruins with specific places where avatars could meet and interact and discuss various elements of culture, philosophy, and architecture. Each student would have an avatar that represented them in the class. Although this sounds futuristic, experiments in Second Life have looked at how this kind of environment facilitates team problem solving (see McNeese, Pfaff, Santoro, & McNeese, 2008), therein testing it as a means of producing new forms of teamwork. The Second Life virtual world was intriguing and it required monthly payments to subscribe to it (or to “rent an island” in the vernacular). However, sustaining it through the economic model and figuring out exactly what advantages it had for more traditional team environments has been challenging. Therefore, it did not exactly catch on as a research tool.
Contextual Perturbations of Teamwork
Technology has advanced distributed team cognition research in creative and prodigious ways, but teamwork enacted within the contemporary culture also has been interpreted more from a contextual-situated lens. Put simply, the role and influence of the environment has become increasingly salient in understanding individual and team activities. The predecessor book that lent credence to this handbook, New Trends in Cooperative Activities (McNeese, Salas, & Endsley, 2001), represents a kind of juncture between traditional work in team cognition and a newer vision of cognitive-social-contextual interactions. This newer view has been referred to as situated cognition (Brown, Collins, & Duguid, 1989; Greeno, 1998; Young & McNeese, 1995) or distributed cognition (Hollan, Hutchins. & Kirsh, 2000; Hutchins, 1995; Salomon, 1993) in that cognition is always in reference to and situated within a specific place, a specific context, and a given environment. This could be inclusive of physical characteristics of a place, the social milieu that surrounds and leads to team cognition, the cultural backdrop that makes sense out of actions, and the technology that supports cognition. Some researchers use situated and distributed cognition interchangeably, but it has been our experience that situated cognition often refers to tasks, events, episodes, and situations in which humans are coupled with action and embedded in the social milieu that affords intentionality through action. Distributed cognition highlights social activities as being highly distributed and represented with information, objects, space, and people. Although these differences are subtle, they can be meaningful to certain research theories and levels of understanding.
Recent views emphasize the predominance of ecological psychology through concepts such as affordances the environment supplies that agents act upon (always in relation to one another) and the idea of direct perception where designs (Norman, 2013) make it likely that information is directly picked up from the environment an agent is acting upon (Gibson, 1979). Ecological approaches emphasize the availability of the dynamic environment where movement through space and time are important for experiencing situations and problems, whether it be at the individual or the group level. Dynamics like these can be specified by information distributed across a context and lead to enhanced awareness of activity and situations occurring in that environment. Designs for team simulations started to utilize Gibsonian concepts to make the simulation more reflective of real-world behavior to the extent possible, contributing to the ecological validity, specification of salient information, and the fidelity participants should experience.
Ecological psychology and situated cognition perspectives (Norman, 2013) that place an emphasis on understanding real-world behavior (in the context it occurs) have concomitantly contributed to a change in methodological approaches in team cognition. Traditional cognitive psychology approaches typically are coupled with quantitative methods in a highly controlled experimental laboratory whereas eco-logical/situated approaches emphasize more qualitative research methods (e.g., ethnography, case study, contextual inquiry, scenario-based design) although they may also utilize quantitative study. Qualitative methods often place reliance upon contextual perturbations that can change the way an individual or team perceives a problem state.
Likewise, qualitative methods may capture how a team prioritizes contingencies that are emergent owing to dynamic events or dynamic movement. Qualitative methods can also reveal impacts and importance of the individual differences within the team that may go unnoticed in quantitative methods. During our use of the qualitative method, concept mapping (Zaff, McNeese, & Snyder, 1993) knowledge integration could be shown to develop between the individual and team-level performance which showed the interdependence among team members.
One aspect of knowledge that is important is how beliefs may translate/evolve from individual differences and get integrated into a team mental model and how the context/culture influences this process. Belief formation may arise through (1) interaction with others and (2) repeated experiences with specific, similar contexts. Although there are other sources for beliefs such as biases, dogmatic convictions, and influence of a trusted source, beliefs arising out of social interaction and experience may highlight the role of individual differences. It may be the case that team conflict is present in a situation as a function of different team members holding to their belief systems obtained through experience which may be oppositional to other team members’ beliefs. Conflicting beliefs can be a real barrier for team cognition and prevent formation of shared mental models. Although team members show unique individual differences and cultural biases, at the same time, their combined set of team processes, requisite knowledge, and skills/abilities offer the opportunity to congeal effective, shared mental models, apropos for the situation. Qualitative methods such as ethnography or concept mapping can provide, capture, and analyze the processes underlying how individuals put their thoughts together to formulate team cognition (or not, in the case of when teams fail to reach common ground). Convergence can be motivated by the need for interdependence to solve a problem at hand (e.g., when different roles on the team produce strands of information that need to be tied together at the right point in time to act on a problem).
The qualitative approach may also help reveal the basis for how cultural variation imprints on individuals and small team behavior. With the global economy being the norm for businesses and for military actions being initiated across nations and governments, team composition will continue to be predicated on determining cultural meanings across team members. For recent research on this refer to Endsley (2016).
Because technology has enabled team participants to be present in different team orchestrations—either at the same time or at different times—they may be required to switch among the contexts that are appropriate to the team to which they are contributing. Context switching is indicative of complex cognition-to-context awareness and is a more difficult task than just having to keep track of one context. Later in the chapter, the use of this type of context switching arrangement is elaborated in one of the simulations built to study distributed team cognition in army command and control tasks (Fan, McNeese, Sun, Hanratty, Allender, & Yen, 2009).
The senior author has always been sensitive to ecological and environmental influences on cognition, because the primary approach in contemporary team research resonates around situated cognition, naturalistic decision making (Klein, Orasanu, Calderwood, & Zsambok, 1993), and real-world constraints on emergent problems (McNeese & Forster, 2017).
CURRENT EXAMPLES OF DISTRIBUTED TEAM COGNITION
As a review of contemporary distributed team cognition is distilled in this chapter within the purview of team simulation, it is useful to consider three real-world examples of how team cognition is highly interdependent on distributed information, contextual perturbation, spatial-temporal integration, and the help/support afforded by information and collaborative technologies. The examples focus on emergency crisis management, uninhabited air vehicle operation, and practice of remote medicine on the battlefield.
The first example revolves around an actual crisis event the senior author experienced at an engineering organization within the confines of a military air force base. Because the event was terrifying and dramatic there is much familiarity and recalled memory associated with it. The focus for this first example then is emergency crisis management which is also the context that inspired our current NeoCITIES simulation. It is a somewhat limited example because it occurred in the late 1970s when information and collaboration technologies for the most part were not yet present in most organizational environments. However, it provides meaning for contextual variability, sensemaking, and emergent perception of a situation as it unfolds within a collective, distributed environment. And it is instructive to see by contrast how the presence of distributed team cognition would have been different if the event had happened in today’s culture.
Emergency Crisis Management
There is a lot of teamwork that develops and is present at the immediate vicinity surrounding a crisis event. Likewise, multiple distributed teams also are drawn into action with additional resources to be managed (therein a team of teams context is in effect). In crisis events, the information that is known versus unknown often dictates the coordination and appropriateness of a team response, and how that response will be worked out given the particulars of the context. As changes happen more context perturbations will need to be considered and factored into the response. The elements of the crisis that are difficult and challenging then are the level of uncertainty that exists, the degree and level of risk that is impending and therein the necessity for quick, immediate action, and the varying degrees of stress, turmoil, and chaos that have to be managed to the extent possible. When these elements are all present action needs to be taken immediately to mitigate risk but at the same time safety is of utmost concern. The elements must be thoughtfully considered with the constraint of safety as a foremost value and concern for all involved. The scale of crises can range from mild (concerned and involved) to the more extreme (volatile and dangerous). A trashcan fire at a business might be an example of a mild crisis. In this case it is pretty well-defined and the actions to mitigate risk are straightforward. However, if the action is not taken immediately then the event can escalate into to a full-fledged crisis. On the other hand, an example of an extreme crisis might be a mass shooting wherein the situation is volatile and dangerous for many. When a shooting occurs, chaos often ensues and is evident by a lack of understanding of what is happening. Information distributed across multiple sources may be hidden or partially hidden to other people in the context. In such high-risk and stressful situations people may quickly have to come together to figure out what is going on and/or to protect themselves to survive, and action steps may be of high importance to help facilitate mitigation of the risk.
The senior author of this paper was involved in an actual crisis management event which took place at a military installation in the continental United States. This event took place around 40 years ago and it is informative to consider some of the aspects inherent in distributed team cognition as it arises on the fly in the midst of a crisis emerging:
I was working in the job as a designer for an avionics systems engineering group when this event occurred one morning. The context was a large engineering building where various individuals and teams were distributed at work on multiple different projects at various locales throughout the building. The layout of the building was such that it had a central high-bay area where actual systems could be built and tested with the proper equipment. I was at work on a drafting table/desk, deeply focused on completing the details of a drawing, when I heard a woman screaming very loudly about twenty-five feet away. This was an unusual occurrence that is experienced infrequently. This first cue indicated that a crisis might be in the making or it might be some other kind of anomalous event taking place and immediately puts one on high alert. As I investigated what was going on several things happened. In my mind, as I recall, this was a warning of something odd happening therefore out of the ordinary, which immediately increased stress and uncertainty, a state of concern, and an awareness of my surroundings. There was also a sense of uncertainty present. Because of this I decided to investigate the source of screaming and quickly walked towards it. In essence this is an example where events are still uncertain and ill-defined, and investigation hopefully leads to more information and potential action to reconcile what might be a pending crisis.
As I moved towards the source of screaming, my investigation was disrupted by an Office of Security Investigation (OSI) agent holding a weapon. My stress skyrocketed upon seeing this. As he turned towards me (with the pistol pointed directly at me) he asked in an urgent voice, “Who are you and what are you doing here?” I identified myself as a government contractor—my office was in the back—and I came to see “what was going on.” His response still leaves me with a chill 40-some-odd years later, “Your boss was just shot to death upon leaving this building a few minutes ago—the screaming you heard is his girlfriend in the office just adjacent to yours.” My boss was a military officer. My fear and anxiety significantly increased even further to the point of shock upon hearing his next sentence, “You had better secure yourself and prepare for the worst, this may be a terrorist event—we are not sure what is going on—but be prepared for gunfire, bombs, and, if need be, find an exit in any way possible.” Now my heart was pounding and at the same time I was trying hard to think and figure out what to do. A perfect example of how emotions shutdown attention and memory—actually an odd form of attention deficit disorder.
The OSI agent at this point also commanded me to go back to my work area rather than venturing into the high-bay area; I quickly conformed. The high-bay was primarily surrounded by offices and a few labs but it was a very large open space which spanned upwards probably three floors. This area was literally in the process of being secured. As soon as he said to secure yourself in your work area I immediately observed what looked like SWAT teams coming into the high-bay with automatic weapons drawn. German police bomb dogs, and general chaos ensuing as employees where scattering and running into various offices (emotion and stress were increasing across the board).
I had decided if I heard gunfire. I would slam my heavy office chair through the window in my office and run out of the back of the building, assuming the gunfire was not coming from that direction. This was my impromptu escape plan given the information I had. As I look back on this situation the information was rather limited, other sources of information were distributed across the building and were not available— they could not have been seen or heard. This timeframe was before computer connectivity to the internet, before cell phones existed, and old-style telephones were the main form of remote communication but I think the lines were jammed so they were ineffective. Direct communication with others on other teams was nonexistent unless people were together within a given office or lab space. As military police started to swarm upon the building and determine if it was secure or to determine what was occurring, they communicated on their headsets but the rest of us were in the dark. Police I believe were now communicating to the extent possible with different offices/ lab spaces throughout the context while also investigating/securing these spaces. This is an example of chaos and fear in the midst of very little communication. Eerily, I could hear many people screaming now, sobbing and crying, after being told of what had happened. The officer shot was well-liked by many and this created a monumental emotional layer to the context that was emerging.
Command was taken over by military police as they searched the building for potential terrorists, bombs, or other clues. I could hear loud speakers in the high-bay but the sound was muffled so I could not discern what was being said. The building was under lockdown until the middle of the afternoon. However, after about 2.5 hours the military-in-charge were certain that it was not a terrorist event and communicated by word of mouth that people could move freely around the building. This was the first sense of relief and psychological release for me and many others. Prior to this point many of us thought this might be the end. This is when we found out the real story of “what was happening” and when collective awareness increased as employees in different offices began migrating to share or to seek the help of their friends or bosses in other parts of the building, and any additional information that could be found out. This is when communication started to ensue. Of course, this is the time and setting where rumors run rampant—and they did. As the day went on I found out that my boss had been stalked for several weeks by a woman who was his former girlfriend. The woman screaming was his current girlfriend. Apparently, his former girlfriend had a mental disorder and unfortunately had a psychotic episode, killing him on his last day of duty as he left the engineering building. I was the last person to talk to him as I wished him well in his next duty station. Haunting.
If this emergency crisis would have occurred in today’s world, information and communication technologies w'ould have enabled much more sharing of distributed information through smartphone communication and capture of different locales throughout the building (and outside the building). Collaborative picture sharing using different apps on smartphones and the feedback from multiple cameras positioned throughout would have enabled a much greater perception of how the context was perturbed (whether anomalous events were taking place across a distributed environment). Distributed team members would have engaged in texting and actual phone conversations without phone lines going down.
In conclusion, this situation was not what it first appeared to be. The killer—after the shooting occurred—escaped by changing clothes and then going out an alternative exit at the installation and was found in hiding a few days later in a forested area near the installation. This is an example of emergency crisis management wherein the context is explored to come up with answers, information (among the police and agents) was shared to figure out the situational awareness of events, and most of the collective intelligence among the workers was very limited (not distributed). The main team orchestration was through the military police on-site who spread out across the building trying to pick up clues and evidence. The one element which was very extreme was the ensuing stress that left many sick, depressed, and with the need for counseling. In his limited capacity, the senior author of this chapter tried to provide leadership and help to others using his psychology background, but this was a terrible thing to have happen.
As researchers look to make team simulations more ecologically valid, representative of real-world problems and issues, and produce scaled worlds that emulate the cognitive demands within contexts of teamwork, there are natural limitations. As can be determined from this example, the whole arena of stress, volatility, emotional affect, and danger are difficult to create or replicate. In many instances, it would be immoral and unethical to consider inducing such states. Yet, these states contribute to the holistic nature of problems and how teams or collections of people respond and experience them. Certainly, many of the distributed information, and situation awareness, conditional changes, temporal constraints, and workload attributes are replicable with contextual realism and should be present within a scaled world simulation. But it will remain a major challenge to capture these other components. The qualitative method is imperative for assisting understanding of these kind of states and contributes to triangulating the comprehension of complex systems.
Uninhabited Air Vehicles
The second example of contemporary distributed team cognition is situated in a military context. From 1995 to 2000, many of the initial engineering designs of uninhabited air vehicles (UAVs) were transformed from ideas to realistic products for use in actual encounters. There have been many military drones engineered to remotely fly on their own for different purposes (e.g., close air support, reconnaissance, intelligence gathering, and even combat). Yet that flight is only possible if there is a competent and alert crew supporting the UAV flight, performing interleaved functions, and acting to achieve the mission objective. Hence, a more common designation of these aircraft today is “remotely piloted aircraft” because it highlights the necessity of humans and teamwork as part of their flight/ mission. Stated another way UAVs represent semi-autonomous vehicles wherein they contain embedded automatic subsystems for specific purposes. However, their overall mission is controlled and overseen by a crew. Different kinds of UAVs (e.g., Predator, Reeper, Global Hawk) have achieved marked success over the last 20 years and have been a strategic asset in the war on terror in Afghanistan and Pakistan and other regions. Having mentioned the success of UAVs they also can have flaws in specific automation functions as well as human errors within the team controlling them which can result in mistakes and failures. Although this section focuses on military use of drones there have also been accidents in the commercial use of drones on various occasions.
UAVs are an example of distributed team cognition occurring within complex contexts that change and require updated system dynamics for success. First, the ability of a plane to fly remotely as controlled from another location at distance is one example of distributed control. Pilots who fly in the cockpit are now offered the capability to fly planes remotely without the threat of being shot down or of being involved in a crash in a dangerous environment. This provides a tremendous human benefit of saving lives of pilots and crews, and an economic benefit of system reuse and precision-based strikes. But other elements of distributed team cognition are part of a successful orchestration as well.
Second, distributed information associated with flying the plane itself, along with the specific functions needed for the mission of the aircraft, are being processed across many channels and propagated from multiple sources (e.g., the area of information fusion). These sources of information may need to be combined, refined, processed, and amplified for greater meaning, and then distributed to various team members to provide updates on the state of the aircraft, the threat, and the environment as the drone engages in its mission.
Third, because these kinds of information streams are voluminous they must be managed, controlled, and displayed in a manner that is human-centered for the distributed crew at a point in time when they are most relevant (the right information at the right time in the right place for the right human). In operational control, it may be the case that a crew has oversight, management, and control of multiple vehicles simultaneously which makes complexity, risk, workload—and hence human errors—increase dramatically. The crew would typically perform different roles such as external pilot, air vehicle operator, or other innate specialists (e.g., aerial photography specialist) that require interdependence and integration of effort inclusive of team situation awareness. The structure of work within the UAV system is representative of distributed team cognition as there are some tasks that individuals must do independently based on their crew role, while there are other times where they need to switch to shared team tasks (Nisser & Westin, 2006). In performing both individual and team-level tasks, the crew relies to a high degree on multiple facets of automation, satellite communications, radar, sensors, video feeds, etc. to gain information that helps fulfill the mission requirements. Given the dynamics embedded in UAV missions, the crew requires transitions between individual and team tasks, while subsystems may also require manual or automatic control. Another transition issue that complicates matters is the point when the crew needs to be replaced with new crew members due to fatigue. This state requires the necessity of handoffs, which can be problematic in team cognition (Tvaryanas, 2006).
At given points in time, the UAV crew may also require communications with ground personnel or other personnel specialists remotely located to provide additional insights, awareness, distributed workload, or spotting for targeting or contextual knowledge. Hence, the cognition and coordination components can place high demands on the crew that challenge team situation awareness and potentially cause information overload. When multiple UAVs are being controlled simultaneously, this is especially true. Models and team architectures have been proposed and studied in team simulation for helping a crew control multiple robots, such as UAVs, whereupon the idea of backup behavior was found to be critical for model success (Gao, Cummings, & Solovey, 2014).
When you add all of these components together, the UAV represents a very complex, distributed cognitive system that is reliant on human, technological, and contextual factors, all of which are highly interdependent to achieve mission objectives. It provides a contemporary example of distributed team cognition that is nonlinear and operative in layers and degrees where time, precision, and calibration are very critical for successful outcomes.
Because of the high degree of synchronicity among system functions during flight, shared knowledge among crew members, and the command, control, communication, collaboration, and intelligence functions that must all come together through human-system interaction, the UAV context is perfect for consideration of team simulation to test various aspects of distributed team cognition. Indeed, Nancy Cooke and her associates have done just that with their high-fidelity simulation of a UAV crew (Cooke & Shope, 2002), which provides a variety of configurations, experimental control, and a wide range of dependent team performance measures. As modeling and simulation have become more advanced, the role of human-autonomous systems becomes more appealing and potentially available for UAV operations. Inherently, intelligent and integrative models may replace some of the crew member functions to make the UAV system even more complex (McNeese, Demir, Cooke, & Myers, 2018), therein requiring profound insights into human-centered computing. Research studies within this area will be extremely valuable for understanding distributed team cognition.
Remote Medicine on the Battlefield
The third example also related to distributed team cognition is in a military setting. The history of warfare has produced battles in which there have been many losses of human life but left many behind on the battlefield injured without much hope. Those who were injured were assisted by trained medics who cared for them with tools of the trade (mainly medications and certain instruments which could be used appropriately until more in-depth care could be received), and through the expertise they had gained with others who were in similar predicaments. In other cases, soldiers had to make do on the fly in the best possible way. In certain situations, helicopter/ air relief could be called in to lift out the injured and fly them to hospitals out of the warzone.
In today’s world, the promise of technology and information availability provide an innovative way to address battleground situations that involve injury. Using telemedicine a team of medical providers can have remote access to diagnosis and even enact surgical procedures (see Reichenbach et al., 2017). Accessing physicians and specialists for interpretation, guidance, and communication about various injuries is possible wherein imagery, pictures, and radiological imprints can be shared. The ability to do robotic surgery is commonplace in hospitals now but to be able to perform surgery, say 1,000 miles away, at a remote medical installation on the battlefield involves a complex array of distributed information, communication, collaboration, coordination, and control, all integrated in real time with precision and with visual and auditory feedback.
Obviously, having appropriate team processes and personnel in place for this major endeavor is necessary as well. This is an example of distributed team simulation wherein being able to have synchronicity and a high degree of sharing across several environments is of utmost importance. It also represents a team-to-team orchestration as the medical personnel/team at the onsite battleground needs to collaborate with a remote medical group to ensure joint operations ensue with accuracy and with adroit knowledge present. This example emphasized the importance of collaborative information sharing under high-risk conditions where the consequences of a mistake or misjudgment could be catastrophic. In summary, this kind of situation is perfect for team simulations involving remote procedures to enable familiarity, practice, and dealing with emergency events at specific focal points in the individual-to-team performance spectrum.
Three contemporary examples of distributed team cognition have been articulated to inform the reader of the contexts that are complex and dynamic, and that produce risks in teamwork. Within these contexts, many possibilities and constraints make solution paths to problems either feasible or impossible to consider. Being able to conduct research using team simulations that represent scaled worlds is very important for contributing to the state-of-the-art in this area. Likewise, expanding methodological approaches to comprehensively address and triangulate these issues will produce prolific dividends. The chapter now looks at a review of research produced in this area and the trends that contribute to further comprehension.
AN OVERVIEW OF TEAM COGNITION RESEARCHCONTEMPORARY PERSPECTIVES
When one considers the specific team simulation history coupled to work at Wright-Patterson Air Force Base as part of the entire learning constructivism, it has woven together many perspectives, engaged different theories and approaches, and compiled various measurements associated with team cognition/distributed cognition. In fact, one might say it has produced numerous published studies that provide evidence of what constitutes team cognition, what it does, how it comes about, and just as importantly how it fails. Team cognition research hence has been developed and evolved from these distant passages into the form it occupies in today’s world. This history represents only one distinct constructivism in the development and formulation of distributed team cognition, albeit it is related to the extensive work of other top researchers and their associates (Nancy Cooke at Arizona State University, Mica R. Endsley at SA Technologies, Inc., Steve Fiore at the University of Central Florida, Susan Mohammed at The Pennsylvania State University, Joan Rentsch at The University of Tennessee, Ed Salas at Rice University) who have been involved in military teamwork, distributed team cognition, and collaborative technologies for the last 25 years.
Each team simulation embarked upon in the 1980s and into the late 1990s had (1) its own theoretical foundation that drove research hypotheses, as well as (2) its own ecology wherein (3) issues and problems from work settings drove questions about how cognition emerges in teams to harness efficiency and effectiveness. The niche occupied through our team simulation repertoire was definitely determined by the specific needs and requirements underlying the necessity to learn more about distributed team cognition, boundary constraints placed on these requirements, how definitions of team cognition were operationalized within the boundary constraints, and the state of technological innovation possible at the time.
The research foundation enabled a certain “instantiation” of teamwork in terms of hypotheses generated, variables of interest explored, and trajectories that have continued to influence where research headed. Inherently, the biases and focus created and influenced our own understanding of distributed team cognition in many different ways. For example, the work with TRAP was derived from the theoretical perspective of distributed dynamic decision making (Pattipati, Kleinman, & Ephrath, 1983; Wohl, 1981), whereas the work with CITIES emphasized an expansiveness of individual theories of situation awareness (Endsley, 1995), and lent credence to psychological distancing theory and the initial testing of electronically mediated communication (Santoro, 1995). Wellens’ work (1993) incorporated his own theory of team situation awareness as coupled to certain types of interaction modalities and expert systems. The early work on Jasper/Repsaj tested theories related to collective induction and perceptual differentiation that afforded analogical transfer of knowledge in complex problem solving (Bransford, Brown, & Cocking, 2000; McNeese et al., 2002). These strands have come and gone but current research involving the follow-on simulation (NeoCITIES) continues to look at many elements of distributed team cognition and the impact of various forms of information and collaborative technologies.
Given this background and progression, it is now useful to look at how team research has evolved today and to consider contemporary trends, prevalent needs, and new roles for technological coupling. Stated another way, in what follows we provide an overview of what we think we know, given what we have done, and how the future could unfold.
One of the first aspects of team cognition to discuss is how it is framed within certain constraints. In the previous chapter of this volume it was established that the research at Wright-Patterson Air Force Base (that the first author was a part of) focused on the context of either C3 or emergency crisis management, especially within the cognitive elements of decision making, problem solving, planning, and design. Although much of the work utilized experimental quantitative paradigms, it was also the case that qualitative study was used to derive understanding of complex teamwork (McNeese & Ayoub, 2011). As work expanded during the last 20 years there have been other fields of practice where distributed team cognition has become prevalent (e.g., civilian search and rescue, driving performance, human-robotic work in crisis management, medical-health informatics, natural gas processing plants, air traffic control, logistics and supply chains, cyber operations, and UAVs). Furthermore, with the development of many new distributed technologies the nature of teamwork has changed dramatically wherein technology is facilitating autonomous tasks with human and synthetic actors (McNeese et al., 2018), allowing unique distillation of patterns for awareness and activity-based responses (crowd sourcing, machine learning, fuzzy cognitive maps, citizen science), and creating seamless opportunities for information sharing and communication (e.g., Skype, Google Hangouts).
The necessity of complex problems creates demands that one person would have trouble addressing alone. Therein, the development of teams has evolved with stated rationality (i.e., intentionality/goals) wherein a member is assigned a given task where tasks (and also team members) are interdependent in their joint actions, and where people, tasks, and activities are interdependent on and distributed across the environment where work occurs. When high interdependency exists in teamwork then there are many temporal and conceptual contingencies that must be assessed and attended to. Because many contemporary problems involve cognition applied within a given context, teams frequently employ cognitive activities to accomplish their tasks. The old idiom “two heads are better than one” is certainly relevant but one must also be aware of the other idiom “too many cooks spoil the broth.”
In more recent times, there have been movements (especially within military contexts) to do “more for less.” For example, the use of robots or personal agents have been used to replace humans on a given team to improve productivity, practicality, or even accuracy of responses or to even do what humans cannot do (Kruijff et al., 2012). Although this strategy may prove to be successful or end up with failure, teamwork is still prevalent and needed to respond to complexity. Also, it may be the case that workers are trained to attend to multiple roles, or interchange roles given the response needed at a particular time, providing adaptive response to complexity (Marks, Sabella, Burke, & Zaccaro, 2002).
The team cognition research presented in this chapter involves team size with two to six team members, hence it is classified as small group research, rather than large group research. The presence of internet technology has made larger group research more prominent over the last ten years as different apps enable groups of people to compare their unique information to other groups of people. Therefore, people in larger groups are enabled to interact in creative ways.
Overall, the McGrath (1984) circumplex (as mentioned in the previous chapter) provides the kinds of processes relevant for distributed team cognition. Using this framework, research groups have focused on and provided in-depth studies associated with the various processes represented in the circumplex. Our research groups have emphasized processes of cognition that intersect situation awareness, mental model formation, and learning.
Challenges and Issues for Simulation
A review of team cognition literature from 1990 until the present revealed nine simulations that were utilized in at least three published articles that investigate team cognition phenomenon. As opposed to some of the early simulations, all of the more recent simulations for team cognition reviewed here were highly situated within a synthetic task environment (though some higher in fidelity than others) but none were as abstracted as the TRAP simulation discussed in the previous chapter. The simulations represent a wide variety of degrees of fidelity and ecological validity, fields of practice, ecological affordances, and methodological approaches to measuring team cognition. This section begins with a brief description of each simulation and a list of representative published studies that utilized each simulation.
SIMULATIONS IN TEAM COGNITION RESEARCH
Unsurprisingly, given the affiliation of many of team cognition researchers with military research institutions (e.g., Air Force Research Lab, Department of Defense, Office of Naval Research), many of these simulations revolve around flight, defense, and military surveillance. These simulations typically utilized undergraduate university students, although in a few cases Air Force ROTC students with some military background were recruited for participation in the distributed dynamic decisionmaking (DDD) and UAV simulations. The nuclear reactor simulation utilized intact control room teams. Below are brief descriptions of each simulation.
The DDD simulation uses teams of four to defend a base against incoming aircraft attacks, and was developed for the Department of Defense to use in training (Miller, Young, Kleinman, & Serfaty, 1998). DDD utilizes a grid divided into four quadrants with a base in the center. Participants use radar to detect and identify incoming aircraft as friend or foe. Each participant is assigned to a quadrant, which only he/she can monitor, and has the ability to operate four types of vehicles with different capacities for engaging incoming enemy aircraft. Because no team member can monitor, identify, or attack aircraft in other quadrants, the team members must work together to share information and coordinate attacks to protect the base. DDD has been used for measuring team cognition in many studies including Christian, Pearsall, Christian, and Ellis (2014), Ellis (2006), Ellis and Pearsall (2011). Johnson et al. (2006), Moon et al. (2004), Pearsall, Ellis, and Bell (2010), and Porter, Gogus, and Yu (2010) and to measure other team constructs outside of cognition research.
The Falcon F-16 simulation and Longbow/Gunship helicopter (AH-64 Apache) simulations utilize dyads to identify and shoot down enemy aircraft while keeping the plane/helicopter in the air. The Falcon simulation assigns a joystick to one member, w'ho flies the plane and positions the plane for making attacks, and a keyboard to the other member, who can set airspeed, call up weapons systems, and gather additional information. Compared to flight simulators used to train pilots, this was a low fidelity simulation (Mathieu, Heffner, Goodwin, Salas, & Cannon-Bowers, 2000), but was complex compared to other commercially available PC-based simulators of the time. Team members were required to coordinate, as neither could accomplish the mission to shoot down enemy aircraft alone. Among the published papers to use the Falcon simulation to study team cognition are Chen et al. (2002), Mathieu et al. (2000), Mathieu, Heffner, Goodwin, Cannon-Bowers, and Salas (2005), and Volpe, Cannon-Bowers, and Salas (1996).
Longbow and Gunship are both low-fidelity PC-based simulators in which each dyad member utilized a joystick to operate helicopter functions; generally, one member flew the plane and fired weapons while the other operated the weapons systems, used radar for surveillance, and monitored systems. Members could not complete the task on their own as they relied on information from the other and the distributed operation of the helicopter functions to meet the goals of identifying and attacking primary and secondary targets, protecting friendly targets, and avoiding anti-aircraft attacks. Chen, Thomas, and Wallace (2005) and Marks et al. (2002) utilized the Longbow simulation, and Stout, Cannon-Bowers, Salas, and Milanovich (1999) and Stout, Salas, and Carson (1994) used the Gunship simulation.
The Uninhabited Aerial Vehicle-Synthetic Task Environment (UAV-STE or UAV, Cooke & Shope, 2004) engages teams of three members in surveillance and taking reconnaissance photos of key targets and is based on Air Force Predator operations. Team members communicate through a push-to-talk intercom system as they each execute one of three roles: pilot (who flew the vehicle, controlling airspeed, heading, and altitude), navigator (who determined flight paths and oversaw the missions), and photographer (who adjusted the camera, took photos, and monitored camera equipment). As in the previous simulations, no one member can complete a mission alone, as the interdependent roles relied on unique information from all three members in order to be successful. Examples of published articles using the UAV simulation to study team cognition include Cooke, Gorman, Duran, and Taylor (2007), Cooke, Gorman, Myers, and Duran (2013), Cooke, Kiekel, and Helm (2001), Gorman and Cooke (2011), and Gorman, Cooke, and Amazeen (2010).
Finally, in terms of military-based simulations, the Non-Combatant Evacuation Operation: Red Cross Rescue Mission (NEO) is a low fidelity, open-ended team decision-making task with three roles (weapons, intelligence, and environmental) where teams must create a rescue plan using military resources. Similar to the previous simulations, each team member received shared and unique information about the resources available for the impending rescue as well as mission goals (for example, rescue crews should avoid detection). On the other hand, teams were not constrained to specific actions as in the other simulations. Teams interacted either face-to-face or via text-based computer mediated communication to share information and develop a rescue plan. Because key information about assets was distributed across the roles, team members were interdependent on one another and communication was required. Three published articles that used this simulation to measure team mental models/schemas include McComb, Kennedy, Perryman, Warner, and Letsky (2010), Rentsch, Delise, Mello, and Staniewicz (2014), and Rentsch, Delise, Salas, and Letsky (2010).
Beyond the military context, there is additional variety of representativeness in other fields of practice. A high-fidelity gas-cooled nuclear-reactor training facility simulation on site at the nuclear plant has also been used to measure team cognition among intact plant crews. This research was not conducted in a typical research laboratory with undergraduate students, but in a training simulator used to train nuclear power plant crews in an exact replica of their working environment. As a result, the participants in these studies were intact nuclear power plant operation crews. The simulator could be programmed for a variety of situations where crews dealt with alarms, phone calls, and crisis situations. Crews members were typically separated at their stations but had the ability to talk together face-to-face when necessary. Published studies utilizing this simulation include Patrick, James, Ahmed, and Halliday (2006), Stachowski, Kaplan, and Waller (2009), and Waller, Gupta, and Giambatista (2004).
The NeoCITIES simulation (an updated version of the CITIES simulation discussed in depth in the previous chapter) is a complex crisis management simulation with three roles (EMS/fire, police, and hazmat) where teams must allocate resources in a given time period to respond to a variety of city crisis scenarios. Because CITIES was ahead of its time, NeoCITIES is still widely used. It will also be discussed in depth later in this chapter. A few examples of research utilizing the NeoCITIES simulation include Hamilton et al. (2010), Hamilton, Mancuso, Mohammed, Tesler, and McNeese (2017), Mohammed, Hamilton, Tesler, Mancuso, and McNeese (2015), Pfaff (2012), Pfaff and McNeese (2010), and Tesler, Mohammed, Hamilton, Mancuso, and McNeese (2018).
Finally, a simulation based on the game SimCity4 has also been utilized as a synthetic task environment for decision making regarding city planning and strategic management. Teams of three to four members are assigned roles to perform the management of a partially developed city created on the SimCity platform. Roles include city planning officer, financial officer, public w'orks officer, and social welfare officer, where each role has unique information and training about the decisions that can be made within their role’s purview. Therefore, team members are highly dependent on information from one another in order to meet the goal of growing the city’s population. Members are given time to prioritize and plan actions, which are then put into play in the simulation. Members can see in real time how decisions affect the city’s population. Some published studies utilizing the SimCity4 simulation include Randall, Resick, and DeChurch (2011), Resick, Dickson, Mitchelson, Allison, and Clark (2010), Resick, Murase, et al. (2010), and Resick, Murase, Randall, and DeChurch (2014).
Ecological Validity of the Simulations
Ecological validity varies across these simulations, although all of the simulations are situated within fields of practice with high-stakes situations where performance is vital to preserving human lives and property. The military-based simulations require participants to identify and protect against enemy aircraft, to identify and photograph enemy targets, or to plan a rescue mission. The NeoCITIES simulation puts teams in crisis management situations that range from fire and crashes to potential terrorism scenarios. The nuclear reactor control room simulation is obviously focused on maintaining the integrity of containment to protect neighboring communities. Even the SimCity4 simulation includes roles focused on the well-being of the city’s population. Although the simulations are all situated in realistic, high-stakes situations, they vary in the degree to which the simulation represents the real-world context. The nuclear simulation has the most ecological validity because it is an exact replica of the control room at the plant. Simulations where participants are monitoring and flying vehicles such as DDD, Falcon, the Apache simulations, and UAV have some level of ecological validity in that they approximate the same functions that would be required by real-world teams but at a lower level of realism or complexity. In part, this is necessary if research is to be conducted using undergraduate student participants. The SimCity4 simulation allows participants to see a bird’s eye view of the city they are developing, but does not immerse them in the realities of making complex decisions for a community.
However, although the realism of simulations may be questioned, the more important issue for ecological validity is whether the simulations approximate the teamwork conditions required in the real-world scenarios. Although these simulations may not have the realism of more sophisticated simulations, they have been purpose-built (or chosen, in the case of off-the-shelf software simulations like Gunship and SimCity4) to create the same kind of interdependence and requirements for communication, coordination, development of team mental models, and situational awareness that would be required in real-world teams. Mathieu et al. (2005) acknowledged the limitations of these kinds of simulations, asserting,
Similar to the position advocated by Mathieu et al. (2000) and Marks, Zaccaro, and Mathieu (2000; Marks et al., 2002), we do not suggest that the results of this investigation would generalize directly to real-world settings such as air combat situations. Rather, we were interested in testing how teammates’ mental models and team processes and performance relate to one another in general and adopted a test-bed that would enable us to do so while controlling contextual influences.
Indeed, this element of control is still a major benefit of choosing an experimental use for simulations.
Importance of Technology for Simulations
The previous chapter noted that the early days of simulation-based cognition research had an experimental focus with a desire for controlled situations and the ability to measure multiple processes and outcomes. The team cognition simulation work since 1990 retains much of that same focus. The more sophisticated simulations rely heavily on technology, specifically software that is designed to be adaptable to the needs of the research and to allow for different experimental and contextual manipulations. Many of the predominant complex simulations have been designed for flexibility, with the ability, for example, to program multiple scenarios, adjust the starting/ending points of aircraft trajectories, determine how many aircraft appear in a scenario and whether the crafts are friend or foe, or create a variety of crisis simulations and events that need responses. Because simulation specifics can be changed and controlled, simulations allow researchers to test the effects of different workloads, member rotations, unexpected events or contextual changes, use of face-to-face versus computer-mediated communication, or team member familiarity with one another (just to name a few variables). Researchers can then measure team performance, adaptability, situational awareness, communication, information sharing, transactive memory, mental models/schemas, coordination, and other team processes. Some of the simulations have been developed with the purpose of initially investigating particular types of team cognition but have also been adapted to investigate others (for example, research with the DDD simulations often measures transactive memory, but DDD has also been used in the context of measuring team interaction mental models).
Research prior to the 1990s sometimes used technology as a way to test group technologies that were being developed. That is still the case with a simulation like the NEO where the simulation itself is administered in a low-tech manner (task information provided on paper), but several software programs have been used to test the effects of that affordance on team information sharing, knowledge organization and retention, and decision making. As the nature of the workplace continues to change such that team members may be working in different locations, team cognition research continues to be interested in ways to improve sharing of information and to develop shared understanding through technology.
In many of the simulations, technology is used predominantly to deliver the content and environment of the simulation (e.g., cockpit, defense grid, vehicle controls, reactor workstations, information on incident/crisis locations). Additionally, technology is being used to enhance the richness of communication in some of the simulations, by allowing non-collocated team members to talk with one another via audio rather than just sending information to computer workstations. Additionally, technology is leveraged by many researchers to support training on task and teamwork skills and (similar to earlier research) to facilitate the collection of process and behavioral data through audio/visual recordings that are later utilized in qualitative coding.
Ecological Affordances in Simulation
Many of the simulations were designed or chosen with particular ecological affordances in the environment that can support the development of team cognition.
The major affordance in all of the simulations mentioned is the interdependency of the team members in terms of distributed information and varied abilities to perform specific behaviors. Because the tasks are designed in such a way that one team member is unable to know and do everything required for success, the simulations provide fertile ground for developing team cognition. For example, in the UAV simulation, the navigator is the only one who can determine a route to the target, but the pilot must fly it, and the photographer alone can take the required photos. To get the best photos, they must all work together within constraints and still meet the goals in an efficient manner (i.e., responding quickly, not wasting film).
Each simulation provides mechanisms for team members to gather and share their own unique information, either through the simulation software or through audio contact. Audio contact also affords teams the opportunity to elaborate on their information or discuss their strategies, which can enhance situational awareness and team mental models and, ultimately, team decision-making abilities. Affordances also include the ability to visualize information. For example, in the DDD, once the team member with an aircraft in his/her detection grid is able to identify the track, the identification information becomes visible to everyone. And, in the NEO task, teams using the CMAP or eWall visual collaboration software for communication can post and visually organize information to help the team make sense of relationships among task concepts.
Methodologies for Measuring Team and Individual Constructs
Much of the simulation research is ultimately concerned with how team cognition and processes affect team performance, hence most of the simulations discussed above include built-in objective measures of team (and often individual) performance. These quantitative measures include number of targets photographed, enemy aircraft shot down, flights successfully routed, time taken to respond to an event, number of weapons or amount of film used, money spent, or even population growth in a simulated city. Objective measures are often built into the architecture of the simulation software and are tallied without additional effort by the researchers. These are quantitative, objective measures of performance outcomes. They are relatively easy to use, can be collected the same way for every team, and provide an easy method to compare teams across conditions.
However, they do not represent the underlying team cognition of interest. In order to develop a deeper understanding of the “black box” processes of interdependent teams engaged in complex tasks, researchers must utilize qualitative methods of data collection and analysis. The most popular of methods across all of the simulations involves coding behaviors (often communications) evident in recorded audio and video of team simulation performances. Behaviors are qualitatively coded by trained experts based on pre-defined coding schemes relevant to the particular type of team cognition being studied. Coding across the duration of an exercise can allow researchers to explore temporal dimensions of team cognition processes and to determine if behavior changes with experience (across scenarios, trials, or sessions). However, although this method provides a more granular look at team phenomena it, unfortunately, captures coder perception of behaviors, but not the actual cognitions of the team members.
Hence, researchers utilizing these simulations also employ several methods to tap into team member cognitions. One popular method involves eliciting team or task mental models/schemas using a paired comparison approach; participants rate the relatedness of pairs of items (usually key dimensions or characteristics of the task or the teamwork requirements of the simulation) to create a matrix, which is then analyzed for accuracy with a subject matter expert model or for similarity with the models of other team members. However, this method is time-intensive and somewhat tedious (a paired comparison approach with ten items requires participants to make forty-five distinct ratings) and studies typically only utilize this method after the team completes the simulation, as requiring participants to complete multiple matrices during the simulation would detract from fidelity and likely exhaust subjects. Situational awareness on the other hand is often checked at multiple points during a simulation, as it can be elicited with a few questions asking participants about their understanding of the current state of affairs in a given scenario. With growing interest in communication and mental models, simulation research since the 1990s has combined objective performance scores with self-report measures of attitudes and beliefs, qualitative coding of behaviors, and participant-provided situational awareness and mental model similarity/accuracy scores.
SUSTAINED AREAS OF RESEARCH WITHIN TEAM SIMULATIONS
Individual and Team Situation Awareness
Situational awareness (SA) is long considered a fundamental teamwork construct, specifically influencing the development of both similar and accurate team cognition. The awareness that one has regarding their own team and other team member roles is often linked to team effectiveness and performance. Without adequate amounts of situational awareness at both the individual and team level, breakdowns in communication and coordination can happen which then negatively impact team cognition.
Multiple teamwork simulations have sought to specifically investigate and measure both individual and team situational awareness in the context of dynamic tasks. Mancuso, Hamilton, Tesler, Mohammed, and McNeese (2013) utilized the NeoCITIES task simulator to study situational awareness for intrusion detection analysis of cybersecurity teams. More specifically, the research team developed a new system built upon NeoCITIES, termed idsNETS. IdsNETS allowed for a task that mimicked the task of an intrusion detection analyst. Utilizing this simulation, the researchers were able to measure cyber SA, or SA that is occurring during a cyber event. Cyber SA was measured using a built-in version of the Situation Awareness Global Assessment Technique (SAGAT), which is a prominent survey metric to measure SA. In addition to SAGAT, the simulation also has a built in humanperformance scoring model that calculates scores relevant to a player’s response to an event in relation to overall damage. This score essentially provides a metric into understanding if players are appropriately responding to events. If they score high, then it can be assumed that they had higher levels of situational awareness. In fact, the score can be looked at in light of SAG AT to provide a more complete understanding of a player’s SA. In addition to idsNETS, the regular NeoCITIES platform provides capabilities to measure individual and team SA. Using SAGAT, the simulation provides capabilities to elicit individual and team measures of both perceived and behavioral SA. In most of the NeoCITIES references found in this chapter, the measurement of SA was employed during the experiment.
Work by Gorman and colleagues (2005; Gorman, Cooke, & Winner, 2006) has also explored SA through the utilization of a simulation. Their research perspective focuses on how communication and coordination impacts the development of team-level SA. Experiments from this research team using an unmanned air vehicle task environment (UAV-STE) have sought to understand how SA is tied to coordination. Findings show that it is not necessary for every team member to be identically aware of the same information and that there is a significant need for team members to have appropriate levels of coordination in place to successfully develop or enact team SA. More recent work in this same UAV-STE has explored the role of team SA in human-autonomy teaming. The UAV-STE was originally designed for the utilization of three human team members (pilot, navigator, payload operator) to control a UAV to take photographs at specific waypoints. More recent iterations of this simulator platform have replaced the human role of the pilot with a synthetic agent. Recent work (Demir, McNeese, & Cooke, 2017) has found that pushing information (as opposed to pulling) amongst the team was positively associated with team SA. In addition, this study found that human-autonomy teams exhibited less pushing and pulling information than human-human teams. In general, much like humanhuman teams, this study highlights that the anticipation of team member behaviors is important in team SA. In similar work, McNeese and colleagues (2018) found that human-human and human-autonomy teams had statically equivalent levels of team situational awareness. This is an important finding as it shows potential from humanautonomy teams to operate in an effective manner. Traditionally speaking, some researchers think that a human-autonomy team could not exhibit effective levels of performance on multiple teamwork dimensions. This may have some truth to it, but this work shows that it is possible for a human-autonomy team to, at a minimum, develop teamwork manners equal to some human-human teams.
Team schemas have been studied using more complex simulations that tend to require team members to take on expert roles and to acquire substantial information prior to being able to execute the team task. Mancuso et al. (2013) employed an emergency crisis management simulation known as the NeoCITIES 3.1 simulation. The simulation involved scenarios generated from observations of and interviews with individuals on actual crisis management teams. Three-member teams with each member assigned to one of three specific roles where each role has specific resources and functions were to respond to two scenarios containing eighteen events. Successful performance required team members to assess the problem and to communicate and coordinate with each other. Team members worked together via technology designed to simulate a dispatch terminal. This complex simulation required training lasting approximately five minutes followed by a five-minute training scenario as practice. The NeoCITIES scenarios were approximately fifteen minutes each. Teams completed two scenarios which could be scripted to varying levels of difficulty. High performance scores were obtained when team members employed appropriate resources. Scenarios were designed such that teams had to be judicious in allocating resources. Thus, the task was scorable.
Mohammed and Ringseis (2001) used a modified version of the Tower Market Task (Beggs, Brett, & Weingart, 1989) which included four stores (grocery, florist, bakery, and liquor store) and each group member served as a representative of one store. The representatives worked together to make business decisions of mutual interest for the stores (e.g., advertising, building temperature). There was no right answer for this version of the simulation. Participants received common information and unique information regarding the store they were representing that included preferences for the decisions to be made by the group. Once groups of representatives were formed to make four mutual decisions, they had thirty minutes to achieve answers.
Rentsch et al. (2014) and Rentsch et al. (2010) utilized a simulation developed by the United States Navy based on a noncombatant evacuation operation (Biron, Burkman, & Warner, 2008). In the simulation, each of three team members was assigned an expert role. Team members were provided common information regarding the goals of the mission and unique information associated with their expert roles. In order to successfully plan the rescue, team members had to share their unique information. Task plans were scorable. Each role contained substantial information such that team members were given time to review their information packets and then teams were allotted sixty minutes to develop their rescue plan.
The complexity of these simulations presents a double edge. The benefits include increased realism and holding participants’ interest. The complexity, in some cases, requires teams to spend more time on the task which enables team processes to unfold and reveal themselves. The disadvantage of the complexity is the time required for participants to adequately acquire the knowledge needed to complete the task. We contend there is a sweet spot for task complexity that enables tests for fully developed team processes without introducing error associated with team members’ lack of understanding of the task. These tasks can also be designed to be scorable and afford the analysis of process data and qualitative observations.
Samples of simulations employed in the study of information sharing among team members include decision-making dilemmas and hidden profile tasks. Dubrovsky, Kiesler, and Sethna (1991) used a dilemma decision-making simulation consisting of four real-life decisions around issues relevant to college students (e.g., career choices and freshman curriculum). Each decision offered an attractive but risky option versus a less attractive but safer option. Groups were given about fifteen minutes to reach consensus for each decision. There were no correct answers.
In his studies of the information-sampling model, Stasser and his colleagues favored hidden profile simulations in which all group members possessed common information and each member possessed unique information critical to identifying the problem solution. In order to achieve high-quality solutions, groups working with hidden profile tasks discover the correct problem solution only when all members share and discuss all relevant information (i.e., members must pool their information). Stasser and Stewart (1992) constructed a hidden profile using a murder mystery in which task information was presented in a twenty-seven-page booklet that included interviews conducted as part of a murder investigation. Twenty-four clues identifying which of three murder suspects was guilty were embedded in the interviews. The interviews (and therefore the clues) were distributed among group members such that groups would have access to all clues only if each member shared all his or her available clues. Combining the clues presented a clear case identifying one suspect as guilty. Group members were provided time to review the task materials and then they met together to discuss the problem and determine the primary suspect. Stasser, Vaughan, and Stewart (2000) used a similar type of hidden profile task in which group members were to complete a decision-making simulation by evaluating characteristics of three candidates running for studentbody president.
These simulations have several commendable characteristics. First, they are straightforward simulations requiring no specialized knowledge, making them appropriate for student participants and relatively short data collection periods. These features also make them amenable to holding participants’ interest (e.g., Dubrovsky et al., 1991). Second, these simulations tend to contain standardized, static information (e.g., Stasser et al., 2000). Third, they lend themselves to quantitative and qualitative data collections. Fourth, these types of simulations can be designed to offer closed-ended solutions that are scorable or offer preference options that are not scorable. Fifth, such simulations may be designed to include manipulations such as distributing information to create “expert” team members (e.g., Stasser et al., 2000).
One limitation of these simulations is that they tend to be quite static, which is appropriate for testing many hypotheses, but may limit their usefulness in testing hypotheses where dynamic simulations may be more appropriate. It is very important to select a simulation containing the characteristics (e.g., an available correct solution versus no correct solution available; static versus dynamic) that maximize researchers’ ability to adequately test their hypotheses.
We now take a look at some of the trends within distributed team cognition and what they mean.
Trends—Strengths and Limitations
Teamwork simulations have contributed to many of the current insights we have obtained regarding team cognition, information sharing, team situational awareness, and a myriad of other team variables. In reviewing the teamwork simulation research for this chapter, we have identified some general trends that point towards both strengths and limitations of teamwork literature.
One positive trend is the utilization of teamwork simulators that are developed with insight from relevant contexts. In this chapter, we have identified multiple relevant contexts that are linked to simulators, such as command and control, emergency management crisis, and cybersecurity. These are all contexts that are (1) deeply important to societal impact and (2) highly dependent on teamwork to adequately function. In addition, another positive trend that is found in most teamwork simulators is the collection of various types of teamwork-related data. For example, most of these simulators collect data relevant to both the individual and the team on dimensions such as performance, communication, coordination, SA, and workload. Taken together these data provide a more holistic sense of what a team is doing and how effective they are.
Some of the limitations that we identified relate to a lack of qualitative work being used in concert with these simulators. In general, it is necessary for more qualitative work to be conducted in the teamwork literature. It is one thing to say that team X is performing at this level or has this amount of SA, but more is needed beyond that. The research community needs to continue collecting this type of data, but we need to also do a better job of asking participants why and how they think they performed in such a manner. And, we are not just referring to survey work as being adequate to fill the lack of qualitative data. We need interviews, focus groups, concept mapping, etc., to occur in addition to our traditional data collection methods.
Another limitation to most of the current work is a lack of simulators that accurately simulate real consequences of the work environment. For example, a UAV simulator is not simulating real-world pressure and the consequences of real-world failure. Simulations need to better represent things like time pressure, stress, and fatigue. There is some work that attempts to adequately simulate this but much more needs to be done to attempt to make the simulation environment feel more realistic beyond just the task being contextually relevant. Finally, we feel that the work conducted in the lab using a simulator needs to be better utilized and introduced back into the real environment that the simulator is indeed simulating. The research results that are found in the lab must be then implemented or at least considered in a real-world environment with the aim to improve teamwork in that environment. We recommend using the living lab approach to better implement research back into a real context (McNeese, Perusich, & Rentsch, 2000).
A CONCEPTUAL FRAMEWORK FOR DISTRIBUTED
TEAM COGNITION AND SIMULATION
Given all the research conducted on distributed team cognition using differing theories, variables of interest, methods, measurements, contexts, and technologies it is useful to consider a common framework to establish (1) an integrative knowledge of the phenomena under study, (2) a plan to execute specific research requirements and approaches to enable contributions that help formulate integrative knowledge, and (3) a means for specifying and developing what a team simulation should consist of, in order to address the underlying requirements. From the mid-1990s forward to 2016 the senior author’s research group created a framework that satisfies these objectives—termed the Living Laboratory Framework (LLF; McNeese, 1996, 2004; McNeese et al., 2005; McNeese & Forster, 2017).
The framework emerged from transdisciplinary research (much of which was described in the previous chapter) with the goal of merging research with design, humans with technology, and making context a basis for understanding cognition when complex systems are part of a field of practice. The basis for this framework derived from the first author’s work in making sense of socio-cognitive systems from ecological viewpoints (see McNeese, 1992; Young & McNeese, 1995) where problem solving and decision making are prominent in social learning and cooperative teamwork. The foundation in this area came from the IDEAL problem solver model (Bransford & Stein, 1984) which is an early proclamation of problem-based learning.
The heart of the Living Laboratory Framework proposes problem-based learning (PBL; Norman & Schmidt, 1992) as a core value in determining research intentionality. PBL as a social learning experience encompasses constructivism where practice, reflection, and insights are primary constituents of comprehending the phenomena under study. Hence, the framework would be considered a problem-centric approach to transformative research within distributed team cognition. The framework essentially considers the trichotomy of theory à problems B practice as informed by four interrelated research acts. It is self-directed, employs opportunistic problem finding and exploration, emphasizes collaborative knowledge seeking, and is situated for a given context of interest (e.g., a field of practice such as emergency crisis management) (Dochy, Segers, Van den Bossche, & Gijbels, 2003; Hmelo-Silver, 2004; Schmidt & Moust, 2000). The LLF (see McNeese & Forster, 2017) hence is the embodiment of PBL with the intent to broadly address many facets of understanding distributed team cognition through an integration of multiple methodological approaches. The use of the LLF enables a researcher to answer many of the conceptual questions posed in the introductory section that enable meaningful team simulations for the purpose at hand.
The original framework (McNeese, 1996) had a central node emanating directly from the PBL foundations—identify and define problems—whereupon problems could jointly be addressed through theory and practice. The LLF considers theory and practice as underlying problem-based learning. In the original framework problem-based learning is bolstered (surrounded) by four inter-connected activities to explore and develop increased understanding of distributed team cognition: (1) knowledge elicitation, (2) ethnographic study, (3) scaled world simulation, and (4) reconfigurable prototypes. (See Figure 3.1 below.) The model assumes that the input and output of these activities are flexible but that together they buildup the knowledge base to address the problems under consideration (i.e., for a phenomenon under consideration in distributed team cognition).
A basic tenet in these activities is that research is a form of continuous process improvement wherein feedback and feedforward processes are critical for both iteration and extensibility. The LLF can be activated iteratively (and thereby achieve an acceptable level of depth for each activity) to produce the best possible scaled world simulation within the constraints that are active, that addresses the specific research problem at hand.
Although an optimal situation would afford enough time and effort to thoroughly analyze each of these activities in-depth, the reality of practical situations is that only a subset of resources may be available for use to the researcher. In these cases, a researcher must make sound decisions and establish priorities for the kind of information that is going to help facilitate the most beneficial problem finding/solving combination.
These research activities can be informed from a top-down (theory-centered) or bottom-up (practice-centered) approach to begin initiation of the research cycle. A bottom-up approach would generally begin with ethnography or knowledge elicitation activity, whereas a top-down approach would begin with simulation using scaled worlds for establishing experimental-quantitative studies. The activities also highlight a “middle-through” perspective with the output being the production of a design that addresses the issues resident in the problem that was defined as the anchor of the process to begin with. Prototypes are the seeds that are translated into actual designs that would then be tested in the field along with requisite feedback for improvement. The idea of the LLF is to provide dynamic feedback from these activities to supply a holistic understanding of complex phenomena and in turn address issues, tradeoffs, and conflicts within defined problem states.
INTEGRATIVE LIVING LABORATORY EXTENDED
FRAMEWORK FOR FUTURISTIC TECHNOLOGIES (ILEFT)
In 2017 the LLF was updated and extended (see Figure 3.1) from the basic model to a more refined research perspective that additionally emphasized the use of data science techniques to further inform theory-problem-practice. The other new component of the extended Living Lab is that user and researcher experience form an orchestrated mutuality of understanding that affords feedback and feedforward experiences. In many current approaches, only user experience is elicited, but the researcher’s knowledge and experience is just as important to provide progressive learning about phenomena. The researcher is an active participant in the Living Laboratory, not just a passive component gaining knowledge from users. The new perspective provided the means to address user experience and researcher experience in dynamic worlds that were heavily coupled with advanced technological systems (such as artificial intelligence, robotics, and social media).
For this chapter, another update has been added to the LLF, primarily the incorporation of the fifth activity: testing, evaluation, and validation. It is important to provide triangulation among the various activity components to test, evaluate, and validate to the extent possible the veridicality of data. This solidifies findings and helps formulate an integrative, holistic nature of findings using multiple methods. Note that each of the activity components can use different methods or techniques to accomplish the purpose of the activity being addressed (e.g., knowledge elicitation may be accomplished through interviews or concept maps). For the extended framework there is also a continued emphasis on opportunities to employ broad datasets (what is now often referred to as “data science”) whenever possible. This extended
FIGURE 3.1 The Integrative Living Laboratory Extended Framework for Futuristic Technologies.
Source: Adapted from McNeese, 2017.
framework amplifies the idea that online teamwork can produce the availability of large sets of data (referred to as “big data”) that can be utilized with predictive analytics to inform decision making and in particular distill meaningful patterns that are representative of issues, constraints, and tradeoffs in a given problem space.
The application of the LLF has been used many times for various applications and fields of practice to solve problems, develop technologies, and to investigate cognitive phenomena from a multi-dimensional perspective. It has been used for research involving team performance in C3 battle management, emergency crisis management, aviation, cybersecurity, police cognition, and health and medicine (McNeese & Forster, 2017). Some of the technology design applications emanating from the LLF are geo-collaborative information systems, predictive decision aids, SA decision aids, fuzzy cognitive maps, intelligent group interfaces, team-based agent architectures, extensible team simulation architectures, and an electronically mediated communication suite. To help explain the use of the framework, a case of application is provided.
Application of Framework to a Specific Case
This case is beneficial for demonstrating the value of the original LLF and it places emphasis on iteration, extensible production, and continuous process improvement, and shows how the LLF functions as an interdisciplinary, transformative framework. Figure 3.1 is used to show a given temporal progression in using LLF. One can begin about anywhere in the framework and the use of the concepts and activity components is in accordance with the needs and constraints of any given project. However, owing to the framework being heavily coupled to ecological interests, a suggested order of progression typically begins with a focus on a given context or situation of interest that occurs in an overall environment/field of practice. Whether related to language, culture, and/or perception, context provides a researcher with initial boundary constraints that allow the mind to generate requisite details and interpret other sources of information as they incur. Our research over the years has dealt with the field of practice of emergency response/crisis management within the civil sector. This is a continuation from the history of team simulation work described in Chapter 1 of this handbook and directly highlights the extensibility of the CITIES research as a context involving fire, police, and hazmat team operations. At the same time, much of the specific context of interest has involved team resource allocation and this is extensible from both the TRAP and the CITIES research simulations.
As one considers NeoCITIES, research can be looked at through the multiple lenses of theory-problems-practice-technology development. From the standpoint of research interests—and specifically the work at Penn State University over the last 15 years—we have been interested in team cognition specifically related to team mental models and team situation awareness, but also how teams end up sharing and communicating information necessary to solve problems and make decisions. That interest directly segues into technology developments as much of the research crisscrosses electronically mediated communications and artificial intelligence. In fact, this was Wellens’ basic interest as his original research (Wellens, 1986, 1993) examined how team SA and communications functioned when teams were coupled through video, text, and phone interfaces. The notion of distributed cognition is highly relevant in what we study wherein information-communication-collaboration technologies can help facilitate effective sharing of information when teams are distributed across environments. In today’s world where Twitter, texts, and social media run rampant, distributed communication and cognition are omnipresent and taken for granted. Therefore, the research stream considered team cognition using different technologies to see whether team mental models and situation awareness could be as good as or better than face-to-face communication. This is the basic level of our research approach. Using the LLF to investigate different facets of this question has been highly valuable and it has been iterated and elaborated on many levels. The following describes the way NeoCITIES has been developed into maturity during the last few years using the LLF elements in Figure 3.1. The approach taken begins by exploring the context starting with the bottom-up approach. Ethnography and knowledge elicitation are the primary activities employed.
NeoCITIES incorporates many of the basics within the old CITIES paradigm inclusive of the crisis management focus. But as time has marched on it was important to achieve more veridical information to anchor the research to realistic contexts of information in order to add specificity to events, situations, and operational reality. Ethnography can be applied in different ways but over the years our research has engaged with operations centers or been able to observe certain exercises within relevant fields of practice (e.g., police cognition, Glantz, 2017; emergency medical services, Jones & McNeese, 2006; 911 dispatch centers, Terrell, McNeese, & Jefferson, 2004). Problems have been overturned and explored and meaning has been derived through observing the environment where work takes shape. When working with operational exercises, ethnography tends to emerge as observation in the moment wherein observers take notes to document human involvement with activities in a given situation, and how they may come to use the available technologies they have. For command center settings, it may be a planned approach to layer observations with interviews or use of other methods (e.g., concept mapping or cognitive task analysis). Because organizations often do not just open their doors for extended ethnography/interviews, one may have to compact these activities into a day or two and capture what one can within constraining conditions. In fact, there is a lot of pre-ethnographic setup effort that is often needed with getting access to a company or government entity to enable ethnographic analysis. In reference to these kind of constraints versus outcome possibilities. Whitaker, Selvaraj, Brown, and McNeese (1995) point out that,
Collecting observational data on real-world design activities is time-consuming, resource-intensive work. On the other hand, there is no other method guaranteed to provide so detailed a trace of design collaborations. The scale of investment necessary to conduct an observation study should motivate researchers to comprehensively plan their tactics and budget their resources. Our experience affirms the utility of observational approaches in studying design collaboration. Recent advocacy of such methods (especially in academic circles), is well justified but solid “how-to” information and/or experienced observational researchers are less readily obtained.
Ethnographic research is important to lay a foundation for the cognate areas being addressed and most importantly can reveal problems in context that may be hidden when applying other techniques. Yet as stated above, it must be highly planned and done in a pragmatic way. For NeoCITIES, we have had the opportunity to engage through various observations and interviews, uncovering valuable knowledge which informed problem definition, information valence, and most importantly the development of scenario richness and specificity. This helps to ensure the validity of what we are studying and helps to make simulations “scaled worlds” rather than just shallow, experimental toy worlds.
Once the foundation is laid for comprehending a given context within a field of practice, a researcher can begin working with subject matter experts (and even novices) to elicit more knowledge about the domain and perhaps engage in sensemaking (Klein, Moon, & Hoffman, 2006) that helps situate observations captured in the previous elements. Knowledge elicitation consists of finding relevant experts acting in the environment/context of interest, working with them to acquire knowledge through various means, then representing knowledge for future use to aggregate further comprehension and learning. The outcomes of ethnography and knowledge elicitation are strengthened when they receive feedback from each other as this amplifies the interpretive component of research investigation. Knowledge elicitation may take various forms such as survey design, interviews, concept maps, concept grids, or schemas, (see Cooke, 1994, for an in-depth review). Dependent on what is desired as a knowledge format and how much time the elicitation is limited to, different techniques may be necessitated. For many of our projects, the use of concept mapping has been used to advantage and forms the basis of the elicitation process and representation language used to capture expertise (Zaff et al., 1993; McNeese, Zaff, Citera, Brown, & Whitaker, 1995) for different fields of practice. Elicitation of knowledge along with results of ethnography may serve as a bridge to validate prior theoretical knowledge or current results from experimental quantitative data, hence establishing a triangulation of data, methods, and theory (Tiainen & Koivunen, 2006). This is an example of triangulated interpretation that strengthens reliability and validity of what is known and helps to establish research integrity. It is also a bridge in the sense that knowledge elicited may directly fold back into the scaled world simulation either as scenario development, or for informing specifics of the demands of cognitive tasks, or the temporal-procedural aspects of carrying forth an action.
As applied to crisis management research, the senior author’s group has had opportunities to interview novices and experts in this domain and related domains (e.g., intelligence analysis work, battle management operations) that provide information for clarifying the contextual demands, inform theory and experiments, contribute to scenario development and design, and leverage the basis for technological innovation and design. As mentioned, the need of every research situation is different and the LLF can be adapted to meet these needs. The representation typology used to capture knowledge obtained is primarily in the form of declarative and procedural concept maps (Moon, Hoffman, Novak, & Canus, 2011; Zaff et al., 1993) often obtained from both individuals and team members for contrast and comparison.
Scaled World Simulation
Just as bottom-up knowledge from the environment and subject matter experts are informative for contextualizing theories and research experiments and creating useful scenarios, scaled world development is critical for realistically representing processes that exist in an environment. The key is to be able to place them into a simulation testbed that is both replicable and controllable. One hopes to maintain the fidelity and richness of the environment to the extent possible (i.e., to make it a scaled world). A scaled world should be a practical experimental venue that has much of the operative knowledge, contextual variation, and situated complexity that is extant in the situation the scaled world purports to model. It should also be flexible, adaptable, and extensible in that the interface, systems architecture, technological milieu, and the sense surround jointly act to portray realistic work settings.
NeoCITIES is a scaled world simulation that has evolved and been adapted for use in many experiments (McNeese & Forster, 2017). It is essentially the same team task as CITIES described in the previous chapter but housed in a client-server architecture for maximal adaptation and control. As communicated in the previous section it has evolved somewhat differently over the years as a function of the scenarios developed with the help of knowledge elicitation and expert judgments. As related to LLF, scenarios capture the problem elements and issues that need resolved. Scenarios can be thought of as complex stories which have various degrees of ill-definition present. Oftentimes the stories connect to the theoretical phenomena to be studied in that they require (1) forms of team situation awareness, (2) shared understanding of teamwork (team schemas, team mental models), (3) ways of addressing information overload and temporal requirements, and (4) coordinating resources across members at the right point in time to yield the greatest outcome (distributed information spaces). The stories that underlie actual data presented contain the elements of situations and events that unfold across time.
The primary team task, team resource allocation, is based on each team member allocating resources (under their control) to alleviate the problems/issues inherent in the situation. For a given situation, the demand may be for one, two, or three members to respond to an event appropriately with the right type and number of resources to resolve the situation. If a situation escalates then the team begins to lose points (performance decrements). NeoCITIES typically consists of a three-person team wherein each member performs a different role: police, fire, or hazmat operations.
Each role has a specialized but limited number of resources (e.g., five police squad cars, five fire emergency response vehicles to begin the simulation) which may be allocated within a set timeframe. The simulation is emergent in that differing situations and events pop up within the time constraints of the entire simulation. Team members must exchange timely information as needed (using distributed chatrooms) in order to enhance coordination, collaboration, and communication. Research has included the derived performance score as a pertinent dependent variable while communications across the team are captured for further analyses (see, for example, Pfaff, 2012). Additionally, many independent and dependent variables have been incorporated based on the phenomena under study (Hamilton et al., 2017).
An important part of a scaled world simulation is the ability to provide interfaces to technological products that are embedded for testing within the simulation. NeoCITIES has been very flexible for this purpose and various technological innovations have been the focus of experimental investigation (e.g., fuzzy cognitive maps, Jones, 2006: intelligent group interfaces, Connors, 2006; and team-computer interfaces, Hellar & McNeese, 2010).
One point of emphasis in this handbook is the integration of new technologies within distributed cognition contexts. This particular element provides the basis of developing with designs as prototypes which can be informed by the bottom-up processes, with the output of a prototype being the basis for assuaging the problem identified and represented in the LLF. The idea of “reconfigurable” connotes that designs are introduced as transmutable ideas, developed through an iterative fashion into working technologies (i.e., a prototype). Making a prototype reconfigurable helps to ensure there is a good fit between team processes and the tool or technology being designed. The prototype can be tested in various ways by users themselves or as an active component in the simulation to further refine its user-centeredness, hence making it reflect both researcher and user experience. The prototype is evolved through use by targeted users (naive and expert) contributing to the ecological validity of the design as an intended solution to the problem at hand. By iterating studies over time with continuous process improvement, effective user-centered designs emerge and can be validated against the problem state. The scenarios used in the simulation can also provide specific timelines and demands to determine what attributes and information fields need to be present to support user functionality.
When applied to the NeoCITIES research case, reconfigurable prototyping has been valuable for developing the team interface (i.e., the human-computer interface for the team(s) participating in the simulation). Early designs developed for the NeoCITIES simulation employed the AKADAM techniques wherein concept mapping elements were translated into design elements using a timeline-based design storyboarding (see McNeese et al., 1995). This is evidenced by various formulations generated first as prototypes and then instantiated as an actual interface designed for individual roles and team use. Some of these interfaces emphasized more perceptual elements (e.g., geography orientation, map-based awareness, visual arrays) whereas others were based on typical human-computer interaction frames that were more verbal-language oriented, and many interfaces utilized both perspectives. Regardless, both were reliant upon chat room technology for team communication.
One open-source programming environment the senior author’s group has utilized to facilitate prototype-to-use applications within client-server architectures is described by Hamilton et al. (2010):
The newest iteration, NeoCITIES 3.0, was built using Web 2.0 technologies to be a more flexible and distributable application. It was designed using a Model-View-Controller (MVC) architecture and built using Java, Adobe BlazeDS and Adobe Flex technologies (Hellar, 2009). NeoCITIES 3.0 is a cross-platform application (e.g., Windows, Mac, Linux) that runs on an Apache Web Server within a Java Virtual machine. It has afforded much adaptability, the production of engaging interface designs, and yields fast mockups of interfaces for the simulation.
In addition to developing interfaces for the simulation per se, prototyping may be used to develop other design innovations (e.g., aids, virtual environment adaptation, information displays) that require information and communications technologies (McNeese, Brewer, Jones, & Connors, 2006). Typically, these designs become independent variables that are part of a test and evaluation protocol to determine whether they enhance distributed team cognition/communication in solving a given scenario within the simulation. Examples of these kinds of designs would be intelligent group interfaces (Connor, 2006), fuzzy cognitive map aids (Jones, 2006), and predictive attention aids (Minotra & McNeese, 2017).
One of the major advantages of the NeoCITIES series of simulations is that the base architecture can be adapted for other potential domains that contain a given form of resource allocation and situation assessment as primary task requirements. Scenarios contain stories and problems that are indigenous to the domain under consideration and are constructed in an iterative manner. New scenarios provide new content that is relevant for a scaled world representative of the new domain. Examples of extensible adaptations are intelligence operations incorporating team-to-team transactions (Connors, 2006), cybersecurity (network access) simulation (Reiffers, 2010), distributed cyber teams (Mancuso, Minotra, Giacobe, McNeese, & Ту worth, 2012), and cybersecurity event monitoring (Minotra, 2012). Although domain extensibility involves more investment to acquire and refine legitimate knowledge for scenario development, the basic NeoCITIES focus in emergency crisis management has been iterated, transformed, and reified many times for unique experimental studies involving elements of distributed team cognition (e.g., see Pfaff & McNeese, 2010 for study of moods and stress within teamwork; Endsley, 2016, for how culture impacts team performance; Tesler et al., 2018, for the effects of storytelling and guided team reflexivity on teamwork).
Over the years the ideas behind team cognition and distributed cognition have been reified and tested through the development of team simulations, as utilized by various researchers testing many aspects of team process and performance, leading to valuable contributions and insights. In certain cases, these insights have informed design resulting in the proliferation of useful technologies that help to assist, enable, or amplify cognition at the individual and team level. In other cases, the simulations provide a nexus that contributes to emergence of team and distributed cognition being blurred (therein the term distributed team cognition). As has been pointed out, the Living Laboratory Framework has been a faithful and extremely useful mechanism in taking a holistic approach to understanding the interplay among humans, technologies, and their environments. It has provided a basic-level approach in determining how to go about implementing a team simulation that has a good chance of improving understanding of the crossroads of psychological processes, ecological niches, and technological innovation. Team simulation holds great promise for future progress in understanding and responding to the extant needs within distributed team cognition. Hopefully, this chapter will make that promise come true sooner rather than later.
1. The MINDS group originally was established at Pennsylvania State University as a laboratory and interdisciplinary research group which specifically utilized the acronym MINDS as representative of Multidisciplinary Initiatives in Naturalistic Decision-Making Systems. The group focuses on conducting interdisciplinary research studies that integrate information, people, technologies, and contexts in a meaningful nexus.
Beggs, R. J., Brett, J. M., & Weingart, L. R. (1989). Teaching note for Towers Market. Evanston, IL: Northwestern University. Dispute Resolution Research Center.
Biron. H. C„ Burkman, L. M., & Warner. N. (2008. April 15). A re-analysis of the collaborative knowledge transcripts from a non-combatant evacuation operation scenario:
The next phase in the evolution of a team collaboration model. Naval Air Systems Command Technical Report, NAWCADPAX/TR-2008/43. Patuxent River. MD: Naval Air Warfare Center.
Bransford, J. D„ Brown, A. L„ & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience and school. Washington, DC: National Academies Press.
Bransford. J.. & Stein, B. (1984). The IDEAL problem solver. New York: W. H. Freeman.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-41.
Chen, G., Thomas, B.. & Wallace, J. C. (2005). A multilevel examination of the relationships among training outcomes, mediating regulatory processes, and adaptive performance. Journal of Applied Psychology, 90(5). 827-841.
Chen. G„ Webber. S. S.. Bliese. P. D.. Mathieu, J. E.. Payne. S. C„ Born, D. H., & Zaccaro, S. J. (2002). Simultaneous examination of the antecedent and consequences of efficacy beliefs at multiple levels of analysis. Human Performance, 15(4), 381-409.
Christian. J. S„ Pearsall. M. J.. Christian. M. S., & Ellis. A. P. J. (2014). Exploring the benefits and boundaries of transactive memory systems in adapting to team member loss. Croup Dynamics: Theory, Research, and Practice, 18(1), 69-86.
Clapway (n.d.). Grandfather of Virtual Reality Embarks on New Adventure, available at https://clapway.com/2015/07/18/grandfather-of-virtual-reality-embarks-on-new-adven-ture-233/ (accessed April 22. 2020).
Connors, E. S. (2006). Intelligent group interfaces: Envisioned designs for exploring team cognition in emergency crisis management. Unpublished doctoral dissertation. The Pennsylvania State University, University Park. PA.
Cooke, N. J. (1994). Varieties of knowledge elicitation. International Journal of Human-Computer Studies, 41(6), 801-849.
Cooke, N. J.. Gorman, J. C., Duran, J. L„ & Taylor, A. R. (2007). Team cognition in experienced command-and-contro) teams. Journal of Experimental Psychology: Applied, 13(3), 146-157.
Cooke, N. J., Gorman, J. C.. Myers, C. W, & Duran, J. L. (2013). Interactive team cognition. Cognitive Science, 37. 255-285.
Cooke, N. J., Kiekel, P. A., & Helm, E. (2001). Measuring team knowledge during skill acquisition of a complex task. International Journal of Cognitive Ergonomics: Special Section on Knowledge Acquisition, 5, 297-315.
Cooke, N. J.. & Shope, S. M. (2002). The CERTT-UAV task: A synthetic task environment to facilitate team research. In Proceedings of the Advanced Simulation Technologies Conference: Military, Government, and Aerospace Simulation symposium (pp. 25-30). San Diego, CA: The Society for Modeling and Simulation International.
Cooke, N. J., & Shope, S. M. (2004). Designing a synthetic task environment. In S. G. Schiflett, L. R. Elliott, E. Salas, & M. D. Coovert (Eds.), Scaled worlds: Development, validation, and application (pp. 263-278). Surrey, UK: Ashgate.
Demir, M., McNeese, N. J., & Cooke, N. J. (2017). Team situation awareness within the context of human-autonomy teaming. Cognitive Systems Research, 46, 3-12.
Dochy. F., Segers, M„ Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: A meta-analysis. Learning and Instruction, 13(5), 533-568.
Dubrovsky, V. Kiesler, S„ & Sethna, B. (1991). The equalization phenomenon: Status effects in computer-mediated and face-to-face decision making groups. Human Computer Interaction, 6, 119-146.
Ellis, A. P. J. (2006). System breakdown: The role of mental models and transactive memory in the relationship between acute stress and team performance. Academy of Management Journal, 49(3), 576-589.
Ellis, A. P. J., & Pearsall, M. J. (2011). Reducing the negative effects of stress in teams through cross-training: A job demands-resources model. Croup Dynamics: Theory, Research, and Practice, /5(1), 16-31.
Endsley, M. R. (1995). Measurement of situation awareness in dynamic systems. Human Factors, 57(1), 65-84.
Endsley, T. C. (2016). An examination of cultural influences on team cognition and information sharing in emergency crisis management domains: A mixed methodology approach. Unpublished doctoral dissertation. The Pennsylvania State University, University Park, PA.
Fan. X.. McNeese. M.. Sun, B.. Hanratty. T. Allender. L., & Yen. J. (2009). Human-agent collaboration for time stressed multi-context decision making. IEEE Transactions on Systems, Man, and Cybernetics (A), 40(2). 306-320.
Gao. E. Cummings, M. L., & Solovey. E. (2014). Modeling teamwork in supervisory control of multiple robots. IEEE Transactions on Systems, Man, and Cybernetics (B), 44(4), 441-453.
Gibson, J. J. (1979). The ecological approach to visual perception. Boston, MA: Houghton Mifflin.
Glantz, E. (2017). Police cognition and participatory design. In M. D. McNeese & P. K. Forster (Eds.), Cognitive systems engineering: An integrative living laboratory framework (pp. 299-312). Boca Raton. FL: CRC Taylor & Francis Publishing, Inc.
Gorman. J. C., & Cooke, N. J. (2011). Changes in team cognition after a retention interval: The benefits of mixing it up. Journal of Experimental Psychology: Applied, 17, 303-319.
Gorman. J. C., Cooke, N. J., & Amazeen, P. G. (2010). Training adaptive teams. Human Factors, 52. 295-307.
Gorman. J. C., Cooke, N. J., Pederson, H. K.. & DeJoode, J. A. (2005, September). Coordinated Awareness of Situation by Teams (CAST): Measuring team situation awareness of a communication glitch. In Proceedings of the Human Factors and Ergonomics Society annual meeting (Vol. 49. No. 3, pp. 274-277). Los Angeles. CA: SAGE Publications.
Gorman, J. C., Cooke, N. J., & Winner, J. L. (2006). Measuring team situation awareness in decentralized command and control environments. Ergonomics, 49(12-13), 1312-1325.
Greeno, J. G. (1998). The situativity of knowing, learning, and research. American Psychologist, 55(1), 5-26.
Gross, T, Stary, C., & Totter, A. (2005). User-centered awareness in computer-supported cooperative work-systems: Structured embedding of findings from social sciences. International Journal of Human-Computer Interaction, 18(3), 323-360.
Hamilton, K., Mancuso, V., Minotra, D., Hoult, R., Mohammed, S., Parr, A.,. . . McNeese, M. (2010). Using the NeoCITIES 3.1 simulation to study and measure team cognition. In Proceedings of the 54th annual meeting of the Human Factors and Ergonomics Society (pp. 433-437). San Francisco, CA: Human Factors and Ergonomics Society.
Hamilton. K., Mancuso, V.. Mohammed. S.. Tesler. R.. & McNeese, M. (2017). Skilled and unaware: The interactive effects of team cognition, team metacognition, and collective efficacy on team performance. Journal of Cognitive Engineering and Decision Making, 11(4). 382-395.
Hellar, D. B. (2009). An investigation of data overload in team-based distributed cognition systems. Unpublished doctoral dissertation. The Pennsylvania State University, University Park, PA.
Hellar. D. B.. & McNeese. M. D. (2010). NeoCITIES: A simulated command and control task environment for experimental research. In Proceedings of the 54th meeting of the Human Factors and Ergonomics Society (pp. 1027-1031). Los Angeles, CA: SAGE Publications.
Hmelo-Silver, С. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235-266.
Hollan, J., Hutchins, E., & Kirsh, D. (2000). Distributed cognition: Towards a new foundation of HCI. ACM Transactions on Computer Human Interaction, 7(2), 174-196.
Hudlicka, E. (2003). To feel or not to feel: The role of affect in human-computer interaction. International Journal of Human-Computer Studies, 59(1-2), 1-32.
Hutchins, E. (1995). How a cockpit remembers its speeds. Cognitive Science, 19(3), 265-288. Johnson. M. D.. Hollenbeck. J. R.. Humphrey. S. E„ Ilgen, D. R., Jundt, D. K., & Meyer, C.
J. (2006). Cutthroat cooperation: Asymmetrical adaptation to changes in team reward structures. Academy of Management Journal, 49, 103-119.
Jones, A. C., & McNeese, M. D. (2006). Cognitive fieldwork in emergency medical services: Developing the I-T-P abstraction hierarchy. In Proceedings of the 50th annual meeting of the Human Factors and Ergonomics Society (pp. 501-505). Los Angeles, CA: SAGE Publications.
Jones, R. E. T. (2006). Studying the impact of fuzzy cognitive map decision aid on team performance and team cognition in a crisis-management, team-based, resource allocation simulation. Unpublished doctoral dissertation, The Pennsylvania State University, University Park, PA.
Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking: Alternative perspectives. Intelligent Systems, 21(4), 70-73.
Klein, G. A.. Orasanu, J., Calderwood, R.. & Zsambok. С. E. (Eds.). (1993). Decision making in action: Models and methods. Westport, CT: Ablex Publishing.
Kruijff. G. J. M.. Colas. E, Svoboda. T.. Van Diggelen, J.. Balmer. P, Pirri. E, & Worst. R. (2012, March). Designing intelligent robots for human-robot teaming in urban search and rescue. In AAAI Spring symposium: Designing Intelligent Robots. Cambridge, MA: AAAI Press.
Mancuso, V. E, Hamilton. K., Tesler. R.. Mohammed. S., & McNeese. M. (2013). An experimental evaluation of the effectiveness of endogenous and exogenous fantasy in computer-based simulation training. International Journal of Games and Computer-Mediated Simulations, 5, 50-65.
Mancuso, V. E, Minotra, D.. Giacobe, N., McNeese, M., & Tyworth, M. (2012, March). idsNETS: An experimental platform to study situation awareness for intrusion detection analysts. In 2012 IEEE international multi-disciplinary conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA) (pp. 73-79). Piscataway, NJ: IEEE.
Marks. M. A.. Sabella. M. J.. Burke, C. S., & Zaccaro. S. J. (2002). The impact of crosstraining on team effectiveness. Journal of Applied Psychology, 87(1), 3-13.
Marks, M. A., Zaccaro, S. J., & Mathieu, J. E. (2000). Performance implications of leader briefings and team-interaction Training for team adaptation to novel environments. Journal of Applied Psychology, 85(6). 971-986.
Mathieu, J. E.. Heffner, T. S.. Goodwin, G. E. Cannon-Bowers, J. A., & Salas. E. (2005). Scaling the quality of teammates’ mental models: Equifinality and normative comparison. Journal of Organizational Behavior, 26, 37-56.
Mathieu, J. E.. Heffner, T. S.. Goodwin, G. E. Salas. E„ & Cannon-Bowers. J. A. (2000). The influence of shared mental models on team process and performance. Journal of Applied Psychology, 85. 273-283.
McComb, S., Kennedy, D., Perryman, R., Warner, N.. & Letsky, M. (2010). Temporal patterns of mental model converge: Implications for distributed teams interacting in collaboration space. Human Factors, 52(2), 264-281.
McGrath, J. E. (1984). Groups: Interaction and performance. Englewood Cliffs, NJ: Prentice Hall.
McNeese, M. D. (1992). Analogical transfer in situated cooperative learning. Unpublished doctoral dissertation, Vanderbilt University, Nashville, TN.
McNeese, M. D. (1996). An ecological perspective applied to multi-operator systems. In O. Brown & H. L. Hendrick (Eds.), Human factors in organizational design and management (Vol. VI, pp. 365-370). The Netherlands: Elsevier.
McNeese, M. D. (2004). How video informs cognitive systems engineering: Making experience count. Cognition, Technology, and Work, 6(3), 186-196.
McNeese, M. D., & Ayoub, P. J. (2011). Concept mapping in the design and analysis of cognitive systems: A historical review. In В. M. Moon. R. R. Hoffman, J. D. Novak, & A. J. Canas (Eds.), Applied concept mapping: Capturing, analyzing and organizing knowledge (pp. 47-66). Boca Raton. FL: CRC Taylor & Francis Publishing.
McNeese. M. D.. Bains. P.. Brewer. I.. Brown. C. E.. Connors, E. S.. Jefferson, T.....Terrell.
I. S. (2005). The NeoCITIES simulation: Understanding the design and methodology used in a team emergency management simulation. In Proceedings of the 49th annual meeting of the Human Factors and Ergonomics Society (pp. 591-594). Los Angeles, CA: SAGE Publications.
McNeese, M. D„ Brewer, I., Jones, R. E. T, & Connors, E. S. (2006). Supporting knowledge management in emergency crisis management: Envisioned designs for collaborative work. In J. Yen. & R. L. Popp (Eds.), Emergent information technologies and enabling policies for counter-terrorism (pp. 255-280). Hoboken, NJ: Wiley-IEEE Press.
McNeese, M. D., & Brown, C. E. (1986). Large group displays and team performance: An evaluation and projection of guidelines, research, and technologies. AAMRL-TR-86-035. Wright-Patterson Air Force Base, OH: Armstrong Aerospace Medical Research Laboratory.
McNeese, M. D., & Forster, P. K. (Eds.). (2017). Cognitive systems engineering: An integrative living laboratory framework.
McNeese, M. D., Perusich, K.. & Rentsch, J. R. (2000, July). Advancing socio-technical systems design via the living laboratory. In Proceedings of the Human Factors and Ergonomics Society annual meeting (Vol. 44, No. 12, pp. 2-610). Los Angeles, CA: SAGE Publications.
McNeese, M. D., Salas, E., & Endsley, M. (Eds.). (2001). New trends in cooperative activities: System dynamics in complex environments. Santa Monica, CA: Human Factors and Ergonomics Society.
McNeese. M. D„ Theodorou. E.. Ferzandi, L.. Jefferson. T. Jr., & Ge. X. (2002). Distributed cognition in shared information spaces. In Proceedings of the 46th annual meeting of the Human Factors and Ergonomics Society (pp. 556-560). Santa Monica, CA: Human Factors and Ergonomics Society.
McNeese, M. D.. Zaff, B. S., & Brown, C. E. (1992). Computer-supported collaborative work: A new agenda for human factors engineering. In Proceedings of the IEEE National Aerospace and Electronics Conference (NAECON), 2 (pp. 681-686). Dayton, OH: IEEE, Aerospace and Electronic Systems Society.
McNeese. M. D., Zaff. B. S.. Citera, M.. Brown. C. E.. & Whitaker, R. (1995). AKADAM: Eliciting user knowledge to support participatory ergonomics. International Journal of Industrial Ergonomics, 15(5), 345-363.
McNeese, N.. Pfaff, M., Santoro, G., & McNeese, M. (2008). Team performance in real and virtual worlds: The perceived value of Second Life. In Proceedings of the 52nd Annual Meeting of the Human Factors and Ergonomic Society (pp. 1435-1439). New York
City, NY, Human Factors and Ergonomics Society. Sage CA: Los Angeles, CA: SAGE Publications.
McNeese, N. J., Demir, M., Cooke, N. J., & Myers, C. (2018). Teaming with a synthetic teammate: Insights into human-autonomy teaming. Human Factors, 60(2), 262-273.
Miller. D. L., Young, P., Kleinman, D.. & Serfaty, D. (1998). Distributed dynamic decisionmaking simulation phase I: Release notes and user’s manual. Woburn. MA: Aptima.
Minotra, D. (2012). The effect of a workload-preview on task-prioritization and task-performance. Unpublished doctoral dissertation, The Pennsylvania State University, University Park, PA.
Minotra. D., & McNeese, M. D. (2017). Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring. Cognition, Technology and Work. /9(1), 161-177.
Mohammed, S., Hamilton, K., Tesler, R.. Mancuso, V., & McNeese, M. (2015). Time for temporal team mental models: Expanding beyond “what” and “how” to incorporate “when”. European Journal of Work and Organizational Psychology, 24,693-709.
Mohammed, S., & Ringseis, E. (2001). Cognitive diversity and consensus in group decision making: The role of inputs, processes, and outcomes. Organizational Behavior and Human Decision Processes, 85, 310-335.
Moon, B., Hoffman, R. R., Novak, J., & Canus, A. (Eds.). (2011). Applied concept mapping: Capturing, analyzing, and organizing knowledge.
Moon. H.. Hollenbeck, J. R.. Humphrey, S. E., Ilgen, D. R.. West. B., Ellis. A. P. J., & Porter.
C. O. L. H. (2004). Asymmetric adaptability: Dynamic team structures as one-way streets. Academy of Management Journal, 47(5), 681-695.
Nisser, T, & Westin, C. (2006). Human factors challenges in unmanned aerial vehicles (UAVs): A literature review. Lund University School of Aviation, Tech Rep. TFHS, Vol. 5.
Norman, D. A. (2013). The design of everyday things. New York: Basic Books.
Norman, G. R., & Schmidt, H. G. (1992). The psychological basis of problem-based learning: A review of the evidence. Academic Medicine, 67(9), 557-565.
Patrick, J., James, N.. Ahmed, A., & Halliday, P. (2006). Observational assessment of situation awareness, team differences and training implications, Ergonomics, 49(4), 393-417.
Pattipati, K. R.. Kleinman. D. L.. & Ephrath, A. R. (1983). A dynamic decision model of human task selection performance. IEEE Transactions on Systems, Man, & Cybernetics, SMC-I3(2). 145-166.
Pearsall. M. J., Ellis, A. P. J.. & Bell. B. S. (2010). Building the infrastructure: The effects of role identification behaviors on team cognition development and performance. Journal of Applied Psychology, 95, 192-200.
Pfaff, M. S. (2012). Negative affect reduced team awareness: The effects of mood and stress on computer-mediated team communication. Human Factors, 54(4), 560-571.
Pfaff, M. S., & McNeese, M. D. (2010). Effects of mood and stress on team cognition in a simulated task environment. Theoretical Issues in Ergonomic Science, 11(4), 321-339.
Porter, С. O. L. H., Gogus, C. L, & Yu, R. C. (2010). When does teamwork translate into improved team performance? A resource allocation perspective. Small Croup Research, 41(2), 221-248.
Randall, K. R., Resick, C. J., & DeChurch, L. A. (2011). Building team adaptive capacity: The roles of sensegiving and team composition. Journal of Applied Psychology, 96, 525-540.
Reichenbach, M.. Frederick, T, Cubrich, L., Bircher, W., Bills, N.. Morien, M.,... Oleynikov,
D. (2017). Telesurgery with miniature robots to leverage surgical expertise in distributed expeditionary environments. Military Medicine, 182, 316-320.
Reiffers, A. (2010). Network access control list situation awareness. Unpublished doctoral dissertation. The Pennsylvania State University, University Park, PA.
Rentsch, J. R., Delise, L. A.. Mello, A. M., & Staniewicz, M. J. (2014). Knowledge building in distributed problem-solving teams: The integrative team knowledge building training strategy. Small Group Research, 45(5), 568-591.
Rentsch, J. R., Delise, L. A., Salas, E., & Letsky, M. (2010). Facilitating knowledge building in teams: Can a new team training strategy help? Small Group Research, 41(5), 505-523.
Resick. C. J.. Dickson, M. W.. Mitchelson, J. K.. Allison, L. K„ & Clark, M. A. (2010). Team composition, cognition, and effectiveness: Examining mental model similarity and accuracy. Group Dynamics: Theory, Research, and Practice, 14, 174-191.
Resick. C. J.. Murase, T. Bedwell. W„ Sanz. E„ Jimenez. M.. & DeChurch. L. A. (2010). Mental model metrics and team adaptability: A multi-facet multi-method examination. Group Dynamics: Theory, Research, & Practice, 14, 332-349.
Resick. C. J.. Murase, T, Randall. K. R., & DeChurch, L. A. (2014). Information elaboration and team performance: Examining the psychological origins and environmental contingencies. Organizational Behavior and Human Decision Processes, 124, 165-176.
Salomon, G. (Ed.). (1993). Distributed cognitions: Psychological and educational considerations. Cambridge, UK: Cambridge University Press.
Santoro, G. M. (1995). What is computer-mediated communication? In Z. L. Berge & M. P. Collins (Eds.), Computer mediated communication and the online classroom. Vol. 1: Overview and perspectives (pp. 11-27). Cresskill, NJ: Hampton.
Schmidt, H. G„ & Moust, J. H. (2000). Factors affecting small-group tutorial learning: A review of research. In D. Evensen & С. E. Hmelo (Eds.), Problem-based learning: A research perspective on learning interactions (pp. 19-52). Hillsdale, NJ: Erlbaum.
Schmidt, K., & Bannon, L. (2013). Constructing computer supported cooperative work: The first quarter century. Computer Supported Cooperative Work, 22(4-6), 345-372.
Stachowski, A. A.. Kaplan, S. A.. & Waller. M. J. (2009). The benefits of flexible team interaction during crisis. Journal of Applied Psychology, 94, 1534-1544.
Stasser, G., & Stewart, D. (1992). Discovery of hidden profiles by decision-making groups: Solving a problem versus making a judgment. Journal of Personality and Social Psychology, 63(3), 426-434.
Stasser, G., Vaughan, S. I., & Stewart. D. D. (2000). Pooling unshared information: The benefits of knowing how access to information is distributed among group members. Organizational Behavior and Human Decision Processes, 82(1), 102-116.
Stout, R. J., Cannon-Bowers, J. A., Salas, E., & Milanovich, D. M. (1999). Planning, shared mental models, and coordinated performance: An empirical link is established. Human Factors, 41(1), 61-71.
Stout, R. J., Salas, E., & Carson, R. (1994). Individual task proficiency and team process behavior: What’s important for team functioning? Military Psychology, 6(3), 177-192.
Tenenberg, J., Roth, W.-F., & Socha. D. (2016). From I-awareness to we-awareness in CSCW. Computer Supported Cooperative Work. 25(4-5), 235-278.
Terrell, I. S„ McNeese, M. D., & Jefferson Jr., T. (2004). Exploring cognitive work within a 911 dispatch center: Using complementary knowledge elicitation techniques. In Proceedings of the 48th annual meeting of the Human Factors and Ergonomics Society (pp. 605-609). Santa Monica, CA: Human Factors and Ergonomics Society.
Tesler, R., Mohammed, S., Hamilton, K.. Mancuso, V., & McNeese, M. (2018). Mirror, mirror: The effects of storytelling and guided team reflexivity on team mental models and performance in distributed teams. Small Group Research, 49(3), 267-305.
Tiainen, T., & Koivunen, E.-R. (2006). Exploring forms of triangulation to facilitate collaborative research practice: Reflections from a multidisciplinary research group. Journal of Research Practice, 2(2), 1-16.
Tvaryanas, A. P. (2006). Human factors Considerations in migration of unmanned aircraft system (UAS) operator control. HSW-PE-BR-TR-2006-0002. Brooke City-Base, TX: 311th Performance Enhancement Directorate.
Volpe, C. E., Cannon-Bowers, J. A., & Salas, E. (1996). The impact of cross-training on team functioning: An empirical investigation. Human Factors, 38(1), 87-100.
Waller, M. J., Gupta, N.. & Giambatista, R. C. (2004). Effects of adaptive behaviors and shared mental models on control crew performance. Management Science, 50(4), 1536-1543.
Wellens, A. R. (1986). Use of a psychological distancing model to assess differences in telecommunication media. In L. Parker & C. Olgren (Eds.), Teleconferencing and electronic media (Vol. V, pp. 347-361). Madison, WI: Center for Interactive Programs.
Wellens, A. R. (1993). Group situation awareness and distributed decision making: From military to civilian applications. In J. Castellan (Ed.), Individual and group decision making: Current issues (pp. 267-291). Hillsdale, NJ: Lawrence Erlbaum Associates.
Whitaker, R. D., Selvaraj. J. A.. Brown, C. E.. & McNeese, M. D. (1995). Collaborative design technology: Tools and techniques for improving collaborative design. AL/ CF-TR-1995-0086. Wright-Patterson Air Force Base, OH: Armstrong Aerospace Medical Research Laboratory.
Wohl, J. G. (1981). Force management decision requirements for air force tactical command and control. IEEE Transactions on Systems, Man, and Cybernetics, /7(9), 618-639.
Young, M. F., & McNeese, M. D. (1995). A situated cognition approach to problem solving. In P. Hancock, J. Flach, J. Caird, & K. Vincente (Eds.), Local applications of the ecological approach to human-machine systems (pp. 359-391). Hillsdale, NJ: Erlbaum.
Zaff, B. S., McNeese, M. D., & Snyder, D. E. (1993). Capturing multiple perspectives: A user-centered approach to knowledge acquisition. Knowledge Acquisition, 5(1), 79-116.