Situation Awareness in Teams: Models and Measures

Mica R. Endsley

CONTENTS

Introduction................................................................................................................2

Situation Awareness in Teams....................................................................................3

Team SA (TSA).....................................................................................................3

Shared SA (SSA)...................................................................................................4

Relevance of TSA and SSA..................................................................................5

Model of Team SA.....................................................................................................6

Team SA Requirements.........................................................................................6

Team SA Devices..................................................................................................7

Tradeoffs across TSA Devices.........................................................................9

Team SA Mechanisms.........................................................................................10

Team SA Processes.............................................................................................12

Challenges for Team S A..........................................................................................13

Poor Support for Distributed Teams....................................................................14

A Lack of Shared Displays and Information Overload.......................................14

Lack of Temporal Overlap..................................................................................15

Problems with Social and Cultural Differences in Teams of Teams...................15

Teams of Teams and Organizational Structures..................................................15

Team Composition..............................................................................................16

Leadership...........................................................................................................16

Ad-hoc Teams.....................................................................................................16

Measurement of Team and Shared SA.....................................................................17

Team SA Process Measures................................................................................17

Team SA Objective State Measures....................................................................18

Combined TSA...............................................................................................18

Collaborative TSA..........................................................................................19

SSA.................................................................................................................19

Team Meta-SA...............................................................................................20

SA Correlation................................................................................................20

Conclusions..............................................................................................................20

References................................................................................................................21

INTRODUCTION

Situation awareness (SA) has been studied extensively in individuals and teams over the past three decades. SA, an understanding of what is happening in the current situation, has been shown to be critical for performance in a wide variety of domains, including aviation, air traffic control, military operations, emergency management, healthcare, and power grid operations (Endsley, 2015b; Parasuraman et al., 2008; Wickens, 2008). In each of these contexts people operating in various types of team settings must quickly understand the state of a complex and often rapidly changing environment in order to make good decisions, formulate effective plans, and carry out assigned duties.

At the level of the individual, SA has been defined as “the perception of the elements in the environment, within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley, 1988). Thus, it includes:

  • (1) Level 1 SAThe perception of key information relevant to the decision maker’s needs. This may include the direct perception of information when the individual is embedded directly in the world (e.g. an infantry solider observing enemy movement or a pilot observing relevant terrain), but also often involves the receipt of information from other team members via verbal or nonverbal communications, written reports, and electronic information displays (Endsley, 1995a, 1995b). Thus, it includes information from natural, engineered, and human sources. For example, an air traffic controller who receives information from a controller in an adjacent sector or from an aircraft pilot is obtaining relevant information from other team members that is then compared to and combined with information from other sources.
  • (2) Level 2 SAThe comprehension or understanding the significance of that information with regard to the decision makers’ goals. SA involves knowing more than just data; it also includes being able to put together disparate pieces of data to inform relevant decisions. For the air traffic controller, knowing that an aircraft is at a particular altitude, location and heading is level 1 SA; understanding that it is below its assigned altitude and therefore has a deviation is Level 2 SA. The formation of Level 2 SA is highly dependent on the goals and decision requirements of the individual, which may vary significantly, based on the person’s role, within and across teams.
  • (3) Level 3 SAProjection of the current situation to inform likely or possible future situations. Projection forms the third and highest level of SA, and is the hallmark of expertise in SA (Endsley, 1995b, 2018). Situation dynamics forms an important part of SA. By constantly projecting ahead, decision makers are able to act proactively instead of just reactively. For example, the air traffic controller is able to project that two aircraft will collide in the future, based on their current assigned trajectories. Similar to Level 2 SA, there can be considerable variance in Level 3 SA projections based on the differing goals and decision requirements of different team members.

It should be pointed out that these three levels of SA represent ascending levels of SA quality (i.e. a person who is able to make accurate projections about the situation has better SA than one who only knows lower-level information), but they are not necessarily linear in terms of process (Endsley, 1995b, 2004, 2015a). While Level 1 SA many generally lead to later integration and comprehension, in many cases Level 2 and 3 SA are also used to drive the search for low-level information or to compensate for data that is unknown (i.e., provide default values for missing data).

The cognitive processes and mechanisms involved in deriving SA have received considerable attention (Endsley, 1995b, 2015a). These include (1) the important role of goal-directed processing that drives the search for information, alternating witli data-driven processing that helps drive the prioritization of goals, (2) limited attention and working memory that can act to constrain SA in complex environments, particularly for novices or those in novel situations, and (3) the formation of mental models and schema that provide mechanisms for rapidly comprehending and projecting information into the future, overcoming these limits to a large degree. This foundation sets the stage for understanding the factors that drive differences in SA across team members and the mechanisms available for supporting coordinated SA with and across teams.

SITUATION AWARENESS IN TEAMS Team SA (TSA)

Teams are defined as “a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/objective/mission, who have each been assigned specific roles or functions to perform, and who have a limited life span of membership” (Salas, Dickinson, Converse, & Tannenbaum, 1992). Critical features that define a team therefore include (1) a common goal, (2) interdependence, and (3) specific roles. The specific roles of the individual team member determine their individual goals, decisions, and SA needs.

In that the performance of team members is mutually interdependent for achieving the common goal, team SA (TSA) is defined as “the degree to which every team member possess the SA required for his or her responsibilities” (Endsley, 1995b). That is, every team member must have good SA for the information associated with his or her role, in order to support overall team performance. It is not sufficient for some members of the team to have a piece of information if the person on the team who needs it does not know it. For example, in the crash of Midland Flight 92 in Kegworth, UK, the flight attendants in the back knew that the pilots had shut off the wrong engine in response to an engine fire, but the pilots in front did not, contributing to the accident (United Kingdom Air Accidents Investigation Branch, 1990). In this way, the SA of individual team members are all relevant to the effective functioning of the team, as shown in Figure 1.1.

Team SA arises from the unique goals and SA requirements of all team members needed to achieve overall team performance

FIGURE 1.1 Team SA arises from the unique goals and SA requirements of all team members needed to achieve overall team performance.

Source: From Endsley & Jones, 1997, 2001. Reprinted with permission from a model of inter- and intrateam situation awareness: Implications for design, training and measurement, in New trends in cooperative activities: Understanding system dynamics in complex environments, 2001. Copyright 2001 by the Human Factors and Ergonomics Society. All rights reserved.

Shared SA (SSA)

Although the specific aspects of the situation that are relevant to each team member’s SA may be different (in that they are determined by the unique goals of each role), because teams inherently involve some interdependence of their members, there also will exist a subset of information requirements that are common amongst the team members, as shown in Figure 1.2. It is this overlap in SA requirements that defines the need for shared SA (SSA). SSA is defined as “the degree to which team members possess the same SA on shared SA requirements” (Endsley & Jones, 1997, 2001). A consistent mental representation of the status of these overlapping requirements is essential for effective team coordination and performance, and drives much of the need for information sharing across teams. On these common SA requirements, two members may possess SA that is (1) shared and correct, (2) shared, but incorrect, (3) not shared, with one member correct and the other incorrect, or (4) not shared with both incorrect (Endsley & Jones, 1997, 2001).

As shown in Figure 1.2, not all the SA of the team members needs to be shared— just the SA associated with common SA requirements. So for example, the air traffic controller and the pilot do not need to share everything about their current situations, which will just create overload (Endsley & Jones, 1997, 2001). They do however, need to be on the same page with respect to the aircraft’s current location, speed and altitude, clearance, the location of other aircraft that are traffic for it, its proximity to nearby terrain or restricted airspace, and the presence of turbulence or weather on its projected flight path that may create a problem for it (Endsley, Hansman, & Farley, 1998). In some cases the air traffic controller may have the more accurate information to share (such as in the case of other aircraft) and in some cases the pilot may have the more accurate information (such as is the case with weather information) (Farley, Hansman, Amonlirdviman, & Endsley, 2000).

The need for shared SA is a function of the overlap in individual goals

FIGURE 1.2 The need for shared SA is a function of the overlap in individual goals.

Source-. From Endsley & Jones, 1997, 2001. Reprinted with permission from A model of inter- and intrateam situation awareness: Implications for design, training and measurement, in New trends in cooperative activities: Understanding system dynamics in complex environments, 2001. Copyright 2001 by the Human Factors and Ergonomics Society. All rights reserved.

Relevance of TSA and SSA

SSA has been shown to be predictive of team performance in a number of studies (Bonney, Davis-Sramek, & Cadotte, 2016; Cooke, Kiekel, & Helm, 2001; Coolen, Draaisma, & Loeffen, 2019; Rosenman et al., 2018). In Bonney et ah’s (2016) study of business markets, team performance was predicted by both SSA on the team (all three levels of SA contributing over 34% of the variance) and by having a shared team strategy. Similarly, in the medical domain, Rosenman et al. (2018) demonstrated that SSA was predictive of performance, and Coolen et al. (2019) showed that SSA on both the problem and the diagnosis were highly predictive of good team performance. Cooke et al. (2001) found that both TSA and SSA of level 1 and level 3 queries was highly predictive of team performance in a study involving the operation of unmanned air vehicles (UAVs).

Overall TSA (based on combined or average SA across the team) has also been found to be predictive of overall team performance (Cooke et ah, 2001; Crozier et ah, 2015; Gardner, Kosemund. & Martinez, 2017; Parush et ah, 2017; Prince, Ellis, Brannick, & Salas, 2007), however, some studies have not found this to be the case (Brooks, Switzer, & Gugerty, 2003; Morgan et ah, 2015; Sorensen, Stanton, & Banks, 2010). For example, Prince et ah (2007) demonstrated that combined TSA scores collected on pilots in low-fidelity simulations were predictive of performance in high- fidelity simulations. Gardner et ah (2017) showed that combined TSA scores were correlated with teamwork ratings in a medical trauma simulation. Similarly, Crozier et ah (2015) showed that a combined TSA score correlated with the experience level of medical trauma teams and was predictive of checklist performance measures in their simulation. Price and LaFiandra (2017) found that stress doubled the level of overconfidence in teams and negatively affected TSA scores, but that efforts to increase team engagement helped to moderate these stress effects. Sorensen et ah (2010), however, were not successful in comparing median TSA across the team to performance, and Morgan et al. (2015) found only a weak correlation between a combined TSA score and performance.

MODEL OF TEAM SA

Endsley and Jones (1997, 2001) developed a model of TSA that includes four major contributors to achieving TSA and SSA. This includes (1) TSA requirements as needed to support the SA and SSA of the team; (2) TSA devices that provide methods for sharing information across the team; (3) TSA mechanisms that support the ability of the team to achieve accurate SSA; and (4) TSA processes that have been found to be important for effective communication and coordination in teams for supporting high levels of TSA and SSA. Each of these will be discussed in more detail.

Team SA Requirements

At the basic level, the SSA requirements of teams can be defined in terms of the overlap in SA requirements between any two roles, as shown in Figure 1.2. As an example of this, Bolstad, Riley, Jones, and Endsley (2002) analyzed the SA requirements associated with army brigade command and control officers. Using a goal- directed task analysis (GDTA) process, we showed how each role had different goals creating the need for very unique sets of information requirements associated with terrain, and how they needed to make very different assessments and projections (Level 2 and 3 SA) based on that information (TSA). A comparison across different roles, however, provided a clear indication of which terrain information needed to be shared across officers to support a common understanding of the battlefield (SSA).

In another example we compared the goals and SA requirements of pilots and air traffic controllers based on GDTAs of each role (Endsley, Hansman et al., 1998; Farley et al., 2000; Farley, Hansman, Endsley, Amonlirdviman, & Vigeant-Langlois, 1998). While we discovered much commonality between these roles in terms of their high-level goals (such as assuring flight safety, avoiding conflicts, and handling perturbations such as weather and emergencies), at the lower levels there are a number of sources of divergence that create conflict for Level 2 and 3 SA. With respect to re-routing decisions, for example, pilots assess potential changes in terms of time or fuel efficiency, while controllers primarily consider their effect on separation and traffic flows. In general, pilots’ aircraft-centered goals were often in conflict with controllers’ system-centered goals, creating the potential for less collaborative negotiations. The analysis was able to identify not only where the SA needs of these two members of the aviation team overlapped, but also the ways in which their higher- level SA assessments were significantly different from each other.

In addition to each team member needing the SA required for his or her individual job, a number of other aspects related to the team form a part of their shared SA needs. For instance, information regarding the actions other team members have taken and their capabilities (e.g., as affected by training, injuries, workload, or fatigue for example) may be important to another team member’s SA (Endsley, Farley, Jones, Midkiff, & Hansman, 1998). Team members need to know the status of all team member activities in terms of how they impact on their own goals and requirements. For instance, one maintenance technician may need to alert other team members that he is opening a valve that would affect the operations or safety of other team members (Endsley & Robertson, 1996; Endsley & Rodgers, 1994). A shared understanding of the impact of the other team members’ task status on one’s own functions, and thus the team overall goal, is important. Similarly, team members need to know how their owm task status and actions impact on other team members so that they can coordinate appropriately (Endsley & Robertson, 1996). Taken together, this set of shared SA requirements forms the basis for a significant amount of information sharing that must occur within teams, based on one or more TSA devices.

Team SA Devices

Several classes of devices are available for supporting information sharing associated with SSA requirements across team members. This includes:

  • (1) Communications—Direct communication, both verbal and nonverbal, is a central method for sharing information across teams and for facilitating a common understanding and projection of events. Team communication has been emphasized as having an important role in supporting the development of SSA across the team in many domains (Endsley, 1995b; Salas, Prince, Baker, & Shrestha, 1995). Communications can occur face-to-face or via phone, radio, or other electronic means. Nonverbal communication has also been shown to be important for SSA in both aviation (Segal, 1994) and medical teams (Xiao, Mackenzie, & Patey, 1998). See Tiferes and Bisantz (2018) for a review' on situational and team factors affecting team communications.
  • (2) Shared environment—Being a part of a shared environment that allows team members to directly observe the same information can also provide SSA. For example, the pilot and co-pilot are in a shared in environment in a cockpit, so they can both directly observe the aircraft takeoff and may not need to verbally communicate that event to each other. They also have the advantage of sharing a view of the same displays. Additional information could be communicated verbally, or through nonverbal cues, such as a look of confusion or fatigue.
  • (3) Shared displays—The development of effective SSA displays has been a theme of interest in many domains. Not all devices are equivalent in terms of their ability to support SSA, however. Bolstad and Endsley (2005) compared a number of tools for support collaboration in teams including face-to-face, video conferencing, audio conferencing, telephones/ radios, chat/instant messaging, white boards, file transfers, program sharing, email, groupware, bulletin boards, geographic information systems, and domain-specific tools. These devices were significantly different in terms of their ability to support synchronous vs. asynchronous collaboration, scheduled vs. unscheduled interactions, collocated vs. distributed, and low intensity vs. high intensity collaboration. They also differed in their level of traceability, identifiability of participants, and ability to support unstructured as well as structured communication. Overall, highly flexible tools such as phones or video/audio conferencing were found to be highly beneficial to developing SSA, particularly for Level 2 and 3 SA in that they are good for handling unstructured, distributed interactions. While computer-based tools such as chat, email, and file transfers have become very popular because they support distributed, unscheduled, and asynchronous communications, they often create substantial data overload to process large volumes of unstructured data. Domain-specific tools consist of customized displays that structure the needed information to support SA and SSA and which are updated in real time. This type of shared display device is the most efficient in that needed information can be integrated and displayed in ways that are easier to comprehend, substantially reducing the workload compared to other TSA devices. These displays require more up-front work to design and build, however, and often may need to be augmented with additional tools for unstructured communications if all the information needed for SSA is not included.

Strater, Cuevas, Connors, Ungvarsky, and Endsley (2008) examined the use of collaborative tools in an army command and control exercise, and found that participants rated face-to-face communication the highest for both routine and non-routine conditions. They rated chat and shared maps/domain tools the next most effective, although not for critical communications; however, they felt these tools contributed the most to SSA, even over face-to-face communications.

In the command and control environment, there has been significant focus on the development of common operating pictures (COP) to support the shared SA of a widely distributed team (i.e., domain-specific shared displays). It is widely recognized that information on the COP must be tailored to each team member’s SA needs, even when working in teams. Bolstad and Endsley (1999, 2000) showed that when the COP was carefully tailored to each team member’s SA needs performance was enhanced, however, when everyone’s SA needs were provided to team members, SA quickly became degraded due to information overload. Javed, Norris, and Johnston (2012) demonstrated improvements in both TSA and SSA with a new display designed to improve SA in an emergency management scenario.

Endsley and Jones (2012) provide a number of guidelines for the design of displays for supporting TSA and SSA.

  • (1) Build a common picture to support team SA—Shared displays should be fed by a common database of information, or if different team members’ displays come from different sensors and sources, information should be provided so that team members are aware of what each other knows.
  • (2) Avoid overload in shared displays—Common operating pictures should be carefully tailored to the SA needs of each team member, avoiding the overload associated with providing all of the team’s information.
  • (3) Provide flexibility to support shared SA across functions—The perspectives and information provided to each team member need to be flexible to support not only their current SA needs, but also their need to understand each other’s tasks and perspectives, including shifts in physical vantage points, goal orientations, and semantics.
  • (4) Support transmission of different comprehensions and projections across teams—Most shared displays provide only low-level data. However, in that different team members will create very different Level 2 and 3 SA, even from the same basic data, shared displays should assist individuals in making cross-team assessments of information to support their higher SSA requirements (e.g., understanding task status and usage of common resources, as well as projections of task timing across the group).
  • (5) Limit non-standardization of display coding techniques—While it is very common to want to allow individuals to customize their displays, in team settings this can act to lower SA in that different team members who share displays may interpret displays very differently (e.g., if red means bad to one team member, but good to another, miscommunications are more likely). Thus, standardization of iconology, color coding, and other user interface features can help support SSA.
  • (6) Support transmission ofSA within positions by making status of elements and states overt—In many systems considerable local knowledge may be needed, for example, who has what expertise, or who has been contacted about a problem. When this knowledge only exists in a person’s head, the ability for other team members to assist in problem solving or take over when needed is significantly lowered. SSA can be improved when hidden, implicit information is made explicit to support team communication and coordination.

Supporting the SA required for team operations first requires that systems are designed to support the SA of each individual team member (see Endsley & Jones, 2012). In addition, display features should be provided to ensure that data, comprehension, and projections to support SSA requirements are quickly and accurately communicated across team members without creating overload. Endsley, Bolstad, Jones, and Riley (2003) show an example of applying these design principles to create effective shared displays for army command and control. Based on a GDTA of each role on the brigade staff, tools for supporting a common understanding of unit status, mission schedule, mission planning, and the geo-spatial location of friendly, enemy, and civilian units were developed. These tools were highly flexible, providing each officer with the ability to customize views and information content to address their various SA needs, as well as to support SSA during mission planning and execution.

Tradeoffs across TSA Devices

Individuals may draw from any one of these TSA devices for forming SA, resulting in dynamic tradeoffs in the degree to which different people rely on different TSA devices at different times. Further, as new TSA devices become available, significant shifts in their reliance on other devices can occur. For example, Bolstad and Endsley (1999, 2000) show that when shared SA displays were provided to a team, their verbal communications decrease significantly. Artman (1999) discovered information overload problems that hampered SSA when information flowed independently to team members, possibly discouraging team communication and coordination. Parush and Ma (2012) demonstrated an advantage for team displays, particularly in overcoming communication breakdowns.

In that new, shared displays are being provided in many domains, it is important to consider that some subtle information embedded in verbal communications may be inadvertently lost in the process. For example, controllers routinely assess the experience level of pilots based on how they sound (Midkiff & Hansman, 1992), and military officers may gage the level of fatigue or stress of their troops based on voice communications (Endsley et al., 2000). These important cues may be lost when shared displays become the norm and voice communications subsequently decline, requiring new forms of support.

Team SA Mechanisms

In addition to team SA devices, many teams can be aided in developing SSA based on the presence of shared mental models (SMM) (Salas et al., 1995; Stout, Cannon- Bowers, & Salas, 2017). SMM are generally considered a consistent understanding and representation of how systems work (components, content, structural relationships, and cause and effect). Since people’s comprehension and projection are largely dependent on mental models, the degree to which two team members have an SSM will help drive similar interpretations of information that is perceived, as well as agreement in the drive for relevant information from the environment.

When people have an SMM they are more likely to arrive at a common understanding of the current situation without needing as much verbal communication. For example, Mosier and Chidester (1991) showed that better-performing aircrews communicated less than poorer-performing ones, most likely due to SMM. In contrast, teams without SMM will most likely require a great deal of real-time coordination and communication to ensure that their activities are carried out properly and will be far more susceptible to lapses in this process. The National Transportation Safety Board (1994) found that 73% of commercial aircraft accidents happen on the first day of a new crew pairing, most likely because strong SMM has not yet been developed to support SSA and team coordination. Bolstad and Endsley (1999) demonstrated that teams with SMM (developed through cross training) performed significantly better on a team task than those without. Bolstad, Cuevas, Gonzalez, and Schneider (2005) and Saner, Bolstad, Gonzalez, and Cuevas (2009) calculated SSA for members of a military team performing exercises and found that both shared knowledge (SMM) and organizational hub distance were predictive of SSA.

SMM are thought to be developed through common training (joint training or cross training), shared experiences in working together as a team, and direct communications between team members as needed to develop SMM in advance of operations (Endsley & Jones, 2001). They provide for standardized communication patterns and vernacular for interactions (Foushee, Lauber, Baetge, & Acomb, 1986; Kanki, Lozito, & Foushee, 1989), create an understanding of who has what information, and put everyone on the same page with respect to the problem being addressed (Orasanu & Salas, 1993).

Bolstad and Endsley (1999) and Gorman, Cooke, and Amazeen (2010) both found that cross training resulted in higher levels of SSA and team performance. Cooke et al. (2003), however, found that while cross training improved shared knowledge, it failed to improve team performance in their study. Espevik, Johnsen, and Eid (2011) showed that naval teams who had trained together over a long period of time developed much better SSM, which provided better team performance than teams who had not trained together, even though they had experienced cross-training sessions. The teams with SMM were better at communicating and providing back-up behaviors. The researchers believe that the superior knowledge of other team member’s characteristics, abilities, and tendencies, developed by training together were important to this finding.

In that mental models are important mechanisms for directing people’s attention to critical data and for forming the higher levels of SA, often almost automatically or subconsciously, they form both a powerful tool for SSA and simultaneously an opportunity for the divergence of SA across teams of individuals working together whose mental models may be different. While it can be argued that teams may benefit from divergent mental models (DMM), in order to bring in new and different ideas or to support the different types of tasks that people need to do on teams, this depends significantly on the types of tasks being performed. Just as SA does not need to be identical among teammates, with SSA needed on only the aspects of the situation that are common to both teammates due to their interdependencies, this is also true of SMM. They only need to be consistent with regard to the degree to which they will drive consistent comprehension and projections of the situation to support shared tasks, as long as they are accurate. As SMM are often not present in many teams (due to the divergent roles, responsibilities, and training of heterogeneous team members, for example), these teams will need considerable communication and coordination to resolve inconsistencies in interpretations and understanding so as to achieve common team goals.

This discussion has primarily focused on the issue of SMM focused around the systems and environmental information associated with a domain. In addition, Mohammed, Ferzandi, and Hamilton (2010) discuss team mental models (TMM) as a “shared, organized understanding of the key elements in a team’s environment.” They also include in TMM common ideas of “taskwork (what the team must do in order to complete goals), teamwork (who team members interact with and how they work together collectively), and..., timework (when members interact with each other)” (Marhefka et al., this book). These aspects of TMM are additionally important for team coordination and performance, in addition to the formation of SMM to support SSA. SMM of teamwork has been found to predict team performance (Cooke et al., 2001).

These concepts are depicted in Figure 1.3, which shows the different aspects of TMM and how they contribute to both effective team processes and to TSA and SSA, each of which contribute to effective team performance. This model is largely consistent with Cooke, Salas, Kiekel, and Bell (2004), but stresses the independent role that team processes can have on team performance in addition to its role on TSA and SSA, as well as the additional factors that affect TSA and SSA per Endsley and Jones (1997, 2001).

Team mental models

FIGURE 1.3 Team mental models, including the degree to which team members share a common understanding of the sociotechnical systems, environment, and team operations, provide TSA mechanisms for supporting TSA, SSA, and team processes.

Team SA Processes

Considerable research has been conducted on the types of processes that teams use and their impact on team coordination and TSA. Orasanu and Salas (1993) summarize a number of studies involving aircrew and military teams to show that effective teams (1) engage in contingency planning that helps setup shared mental models for emergencies, (2) have leaders who establish a democratic environment that supports better sharing of information, and who explicitly state more plans, strategies, and intentions, consider more options, provide more explanations, and give more warnings or predictions (Chidester, Kanki, Foushee, Dickinson, & Bowles, 1990; Orasanu, 1990), and (3) develop a shared understanding of the problem prior to looking for solutions, thus avoiding getting bogged down (Hirokawa, 1983), which is particularly important the more diverse the team members’ backgrounds (Citera et ah, 1995). Klein, Zsambok, and Thordsen (1993) additionally point to the importance of

(1) clear delineation and understanding of tasks, goals, and roles, (2) avoidance of micro-managing but willingness to help other teammates, (3) avoidance of fixation and willingness to examine various factors of the situation, (4) encouragement of different opinions and a process to come to convergence, and (5) the ability to manage time and make changes as needed.

Taylor, Endsley, and Henderson (1996) found that effective teams established a group norm of information sharing and self-checking to make sure everyone was on the same page at each step. They coordinated as a group, delegated tasks, and gathered information from each other. They imagined possible events in the future and came up witli contingency plans for addressing them. They also actively prioritized goals, so that overall performance was not sacrificed due to distractions or unexpected problems. In contrast, poorly performing teams had a group norm in which pertinent information was not shared, so that they went along with the group without contributing important, but conflicting information. They were easily distracted by unexpected problems and were unable to prioritize tasks effectively. They tended to rely more on expectations, which may have been incorrect, and had no team processes in place for detecting this. In some cases, a strong personality acted to lead the others astray based on a strong, but erroneous, picture of the situation, creating what one aviator described as an “SA black hole.”

Other researchers have studied the ways in which TSA varies between high- and low-performing teams. Hallbert (1997) found that high-performing teams had better SSA that he attributed to the importance of team cohesion and coordination. Working with trauma resuscitation teams, Crozier et al. (2015) showed that SSA, complementary SA (knowledge needed by individuals that is not shared), and overall TSA all increased with team experience. Sulistyawati, Chui, and Wickens (2008) also found that good teamwork behaviors (such as coordinating when necessary and providing feedback and support) were significantly correlated with good SA of both a pilot’s own tasks and the teammate’s tasks.

In studying the processes used by control room teams to solve challenging scenarios, Patrick, James, Ahmed, and Halliday (2006) found differences in planning, problem-solving techniques, team coordination, attention, communication, and knowledge that they felt contributed to differences in TSA. Gross and Kluge (2014) demonstrated that effective team processes for knowledge sharing directly improved SMM in teams in the steel industry. Stout, Cannon-Bowers, Salas, and Milanovich (1999) also showed that team planning could improve SMM among team members, creating more efficient communications patterns and more coordinated team performance. Berggren, Prytz, Johansson, and Nahlinder (2011) showed a strong relationship between teamwork (coordination, support, and communication) and subjective SA in a simulated firefighting experiment, with workload negatively affecting both teamwork and SA. Altogether, a strong body of research supports the importance of team processes for achieving effective SSA and TSA in teams and for achieving overall team performance.

CHALLENGES FOR TEAM SA

A number of challenges have been found that inhibit good TSA and SSA. These are often particularly problematic as the teams become distributed (separated by space, time, or obstacles), and when teams of teams are involved. For example, Endsley and Robertson (1996; Robertson & Endsley, 1997a) investigated TSA in aircraft maintenance teams conducting both an SA requirements analysis of the multiple teams involved and a contextual inquiry to examine information flow across the organization and found:

(1) Poor shared mental models—The teams from different parts of the organization had poor mental models of the activities and information needs of the other teams leading to not fully understanding the implications of transmitted information, which contributed to errors and inefficiencies.

  • (2) Poor verbalization of decisions—Teams were poor at passing on to other teams why they chose a particular course of action, resulting in misunderstandings and sub-optimizing performance as the knowledge of other teams could not effectively come into play.
  • (3) Inefficient shift meetings and teamwork—Team leaders failed to adequately convey common goals for the team, provide a clear understanding of task assignments, point out the inter-relationships between different people’s tasks, and provide clear expectations regarding teamw'ork.
  • (4) Poor feedback on the effect of actions across the distributed organization—As the aircraft would move on to other geographic locations, people rarely got good feedback on the result of repair actions they had taken, which significantly inhibited the learning needed for SA and diagnostics.
  • (5) Problems with poor individual SA—Many common SA challenges were found, such as forgetting steps due to interruptions, missing critical information due to other task distractions, misinterpreting information due to inaccurate expectations, and failing to pass on important information or communicating poorly. These SA failures at the individual level w'ould then propagate across the team to result in TSA failures.

A number of other challenges for TSA and SSA have been found.

Poor Support for Distributed Teams

Distributed teams need to achieve the same SSA as co-located teams, but are separated by time, space, or obstacles (Endsley & Jones, 2001). They frequently do not have many TSA devices available to them. For example, a pilot and an air traffic controller generally do not share the same environment, cannot communicate via nonverbal cues, and historically have a limited amount of information about each other on shared displays. Thus, communications channels can be highly overloaded as the main pathway for building SSA.

As technology has improved, new sensors have been added that allow controllers to see basic information about each aircraft and allow' pilots to gather some information about each other (Jones, 1997), however, much of their SSA must still be achieved through verbal communications via the radio. Farley and colleagues showed that the addition of datalink significantly improved the SSA of pilots and controllers with respect to weather and air traffic (Farley et al., 2000; Farley et al., 1998). This was accomplished w'ith less radio communication, even though the rate of route negotiations increased, demonstrating the tradeoffs between TSA devices that can occur.

A Lack of Shared Displays and Information Overload

In many environments, such as command and control or emergency management, for example, information is not shared well across team members via shared displays. Instead, different team members each see a subset of information, often generated via different sensors or information sources. This situation places a very high demand on the use of voice communications or chat rooms to try to compensate for the inadequacies of the information systems, resulting in a heavy manual load and often poor SA as information falls through the cracks or is interpreted differently across the distributed team. When distributed teams also need to deal with poor voice communications lines, such as may be common in many of these same settings, the problems are compounded.

Lack of Temporal Overlap

In many environments work occurs across a 24/7 time schedule or moves across different geographic units, with different teammates assuming responsibility over time. For example, different air traffic controllers must pass aircraft across sectors of responsibility, and must pass responsibility for their sector to other controllers across shifts. The same situation occurs in other settings, including healthcare in hospital settings, aircraft maintenance, and command and control of military, space, and power systems.

The need for people to coordinate across multiple shifts and locations adds to the difficulty of fully communicating an understanding of what is happening as work shifts across teammates. In particular, the challenge of communicating pertinent status information, watch items, concerns, and task statuses can be significant, particularly as shift turnovers are often hurried and poorly supported by information displays. The use of well-structured shift turnover practices and tools have been extended from air traffic control to many other areas, including aviation maintenance (Parke & Kanki, 2008) and healthcare (Jeffs et al., 2013).

Problems with Social and Cultural Differences in Teams of Teams

Robertson and Endsley (1997b) found a lack of trust across distributed teams, as well as the development of different cultures across distributed teams that contributed to challenges with teamwork and developing SSA. Given that many distributed teams are often not functionally or organizationally integrated, resolving these differences can be quite difficult.

Teams of Teams and Organizational Structures

Wellens (1993) investigated distributed teams involved in responding to emergencies, including police and firefighter units. He found that as the workload increased, the degree to which people communicated across team boundaries decreased. This challenge is often exacerbated if shared SA devices are poor, such as was the case in the response on 9/11.

In a study of SSA in military command and control teams, Saner, Bolstad, Gonzalez, and Cuevas (2010) found that the most important predictor of SSA in a team was the organizational proximity of the team members. While communication frequency was related to better Level 1 SA, for more complex information higher SSA was associated with a higher cognitive workload that was required to achieve it. They also showed that greater similarity in experience was related to better SSA.

Buchler et al. (2016) discovered that a few individuals dominated information sharing across the staff in a command and control setting, but that those who engaged in high levels of email output had lower SA. Higher SA on the other hand was correlated with more information inputs and fewer information outputs on the email system. While this may be due to the workload demands of email generation, these findings may also reflect the hierarchical nature of military command and control where less experienced personnel are assigned more mundane tasks (i.e. a division of labor decision strategy). They also found that team members with high SA were more likely to communicate with others with high SA and low SA people were more likely to communicate more with other low SA people.

Team Composition

A number of researchers have demonstrated that team composition significantly affects TSA. Some teams are much better at sharing and promoting SA across the team, and being in such a team can improve the SA of its members. Stetrevik (2012) showed that team membership significantly influenced individual SA in a study of emergency handling in teams in the energy industry. Several researchers have also demonstrated that SA is sensitive to which teams people belong to in military command control exercises (Bolstad & Endsley, 2003; Leggatt, 2004; Seet, Teh, Soo, & Teo, 2004). So, an individual’s SA is impacted by not only his or her own cognitive skills, but also the skills and knowledge of the team they are a part of.

Other research, however, found that individuals working in teams had lower SA than when working alone on a simulated firefighting task, presumably due to the overhead costs of communicating to share information within the team (Parush, Hazan, & Shtekelmacher, 2017). Thus, some tasks may be better performed by individuals. But for those that inherently require the contributions of multiple team members, TSA and SSA are critical to effective team performance.

Leadership

Team leaders also have a marked effect on TSA. Cuevas and Bolstad (2010) showed over three separate studies that team leader SA was positively correlated with the SA of the team members, accounting for between 12% and 49% of the variance in SA between teams.

Ad-hoc Teams

While many teams remain relatively fixed, in many cases people may flow onto and off of teams in a more dynamic fashion, for example, design teams or teams formed to solve special problems in an organization (often called tiger teams or kaizen). Strater et al. (2010, 2008) studied ad-hoc teams set up by the military. We found that ad-hoc teams were challenged by being largely distributed and lacking common training which made it difficult to formulate trust and team cohesion. They did not possess the common knowledge and common background that was needed to develop effective SMM. Their goals tended to be more abstract and ill defined, and they often worked in unfamiliar environments and with poorly defined roles and responsibilities. Further, as they worked across different temporal timelines, they experienced the challenges of regaining SA across shift changes on a frequent basis. Often the members of these ad-hoc teams were required to balance other competing duties associated with their regular jobs in addition to the ad-hoc team, creating SA problems due to multi-tasking. As knowledgeable team members leave the team, TSA often suffered as well, since there may be poor documentation or information to cover that person’s knowledge.

MEASUREMENT OF TEAM AND SHARED SA

TSA and SSA have been assessed in a number of ways including inferring it from team processes, communications, group transactions, and information sharing, or by directly measuring the SA state of team members and comparing it within and/ or across teams.

Team SA Process Measures

Some researchers have focused on individual and TSA by examining the processes used to achieve it. See Cooke and Gorman (2009) for a review. This research records team communications as a window into what teams are thinking and how they are interacting. These communications are then analyzed using techniques including network models (Buchler et al., 2016; Gorman, Weil, Cooke, & Duran, 2007), time series analysis (Kitchin & Baber, 2017), and latent semantic analysis (Bolstad et ah, 2007; Cooke et ah, 2004). Others have focused on team behaviors and interactions to investigate the ways in which teams coordinate to achieve SA. Cooke, Stout, and Salas (2001) review a number of possible techniques, including verbal protocols, structured interviews, and process tracing to gain insights into TMM and TSA.

As examples of this work, Gorman, Cooke, and Winner (2006) examined team behaviors in responding to unexpected situational changes, with an emphasis on team coordination. Similarly, Gorman, Cooke, Pederson, Connor, and DeJoode (2005) classified the effectiveness of team communications following a problem with a scenario, confirming that it is important that relevant team members have SA of a problem, but not necessarily all team members.

Patrick et ah (2006) subjectively observed team behaviors in a process control room and evaluated how effective the team processes were in handling programmed disturbances. Hauland (2008) used eye-tracking to assess the attentional strategies of air traffic controllers as a means of inferring TSA. Salmon et ah (2008) focused on the transactions between team members and team artifacts to examine information flow using network-based analyses. This approach, however, combines a consideration of SA sources (which can vary significantly between individuals and between situations) and team members, making its interpretation limited.

Overall, while there are valuable insights to be gained from studying team communications and the processes that teams use to develop SA, it is ultimately difficult to say how successful such processes are in contributing to TSA or SSA without an independent objective measure of their effectiveness. While factors such as communication and teamwork are undoubtedly important for good SA, accurate SSA can also be achieved outside of these processes, such as by team members who gather accurate shared understanding of the world by virtue of both getting the same information directly from displays or the environment that they share, or via SMM. Therefore, team process measures may only provide a partial understanding of SSA. Further, even teams attempting to communicate and coordinate may end up doing so ineffectively. And much of SA (particularly comprehension and projection) may be largely cognitive and not revealed in typical team communications or interactions. An objective measure of TSA and SSA provides the relevant outcome metric for comparing to team process measures in order to fully appreciate their effectiveness for achieving TSA.

Team SA Objective State Measures

Another approach has been to directly assess the SA of team members and to use that to draw comparisons across team members, or across different teams. The Situation Awareness Global Assessment Technique (SAGAT) (Endsley, 1995a) provides for a simultaneous assessment of each team member’s SA during periodic freezes in team operations. It is then possible to identify problems due to inadequate tools or team processes by comparing the SA of different team members or sub-teams at the same point in time.

Endsley (2019) conducted a review of 24 studies that used SAGAT to evaluate team SA. Researchers were found to use a number of approaches in examining team SA, including: (1) forming an overall team SA score based on the total or average SA across the team (11 studies), (2) allowing for a collaborative team response to SA queries (two studies), (3) assessing SSA (or SA similarity) based on the degree of concurrence between teammates on information elements relevant to both roles (14 studies), (4) team meta-SA, examining the degree to which team members are aware of the SA of each other (three studies), and (5) determining the degree of correlation between the SA of different team members, or sub-teams (six studies). Each of these scoring approaches will be considered separately.

Combined TSA

Bolstad and Endsley (2003) created a combined TSA score for different teams involved in a large command and control exercise, showing significant differences between teams associated with poor team collaboration tools. Gardner et al. (2017) calculated composite TSA scores by calculating the accuracy of each team member’s SA on each SAGAT query and summing the scores across all team members. They showed that TSA for medical teams involved in trauma care was significantly related to team work scores (r2 = .50 and .55), as well as team performance scores (r2 = .30 and .38).

Cooke, Stout, Rivera, and Salas (1998) created a combined TSA score in a helicopter simulation and showed it was significant correlated with team knowledge accuracy. Shared knowledge decreased over time in their study, however. Cooke et al. (2001) also showed that TSA was predictive of team performance, and Crozier et al. (2015) similarly demonstrated that TSA was correlated with checklist performance.

Collaborative TSA

In a different approach, some researchers have allowed team members to collaborate when answering SAGAT queries, rather than providing independent scores that are later compared (Hallbert, 1997; Price & LaFiandra, 2017). While this may make sense in some operational settings, it does not allow for a comparison of SA within teams, only across teams.

SSA

Endsley, Bolte, and Jones (2003) recommend directly comparing the SA of different team members on SA queries that they share in common in order to get a measure of their shared SA. Using this approach in an air operations center, they show how the SSA of two team members falls into one of the four categories: (1) both are correct, (2) one team member correct and the other incorrect, (3) both incorrect in the same way, and (4) both incorrect in different ways. This provides for a scoring of SSA for each relevant SAGAT query between any set of teammates.

Using this approach, Bonney et al.’s (2016) study of business markets showed that team performance was predicted by both SSA of the team (all three levels of SA contributing over 34% of the variance) and by having a shared team strategy. Similarly, in the medical domain, Rosenman et al. (2018) demonstrated that SSA was predictive of performance, and Coolen et al. (2019) showed that SSA on both the problem and the diagnosis were highly predictive of good team performance. Artman (1999) demonstrated differences in SSA between teams in a simulated firefighting task that was significantly related to the types of communications that were provided (serial or parallel). Javed et al. (2012) demonstrated improvements in SSA with new displays for emergency management.

Other researchers have created a combined SSA score. Bolstad et al. (2005) investigated the SSA of team members as measured by SAGAT scores in military teams by assigning a 1 for all queries where team members answered the same and a 0 for all queries where they were different to create a SSA similarity score across SAGAT queries. They then compared the teams’ SSA scores to their physical distance, social network distance as calculated from frequency of communications, rank similarity, and branch of service similarity. Only physical distance was a significant predictor of SSA between team members, accounting for about half of the variance, as well as the vast majority of the difference in social network distance.

Saetrevik and Eid (2014) proposed examining SSA by comparing each team member’s answer to the average of all team member answers. Thus, the measure did not reflect SA correctness, but only how cohesive the teams were in their perceptions. They also compared how similar each team member’s SA was to the team leader. These assessments did not show any sensitivity to team membership, numbers of team meetings, or the time since the last meeting, leaving the value of this approach questionable.

In contrast, Saner et al. (2009) argue that there is no shared SA unless it is also accurate. Therefore, their SSA similarity index was calculated on the basis of SSA accuracy for any two team members. They showed that both organizational hub distance and the levels of shared knowledge of the team members predicted SSA. It should be noted that these approaches create a combined SSA score across different SA queries. In a meta-analysis of SA metrics, Endsley (2019) found that combined SAGAT scores are generally less sensitive that those that make comparisons by query or by SA level.

Team Meta-SA

Some researchers have examined the degree to which team members are aware of the accuracy of their own SA and the level of SA their teammates. Sulistyawati et al. (2008) distinguish between the need to (1) have SA related to one’s own goals, (2) have SA required to back up team member SA, (3) have an accurate understanding of how good one’s own SA is (meta-SA), and (4) awareness of a teammate’s workload and SA levels. They found that pilot teams with good SA were significantly less likely to have over-confidence bias (r = .85, p < .01), demonstrating that those with good SA also had good meta-SA of their own knowledge. However, they did not find any relationship between accurate SA of one’s own SA requirements and that of the teammate’s SA or workload. Pilots in their study were generally poor at estimating the SA and workload of their teammate.

Sulistyawati et al. (2008; Sulistyawati, Wickens, & Chui,2009) showed that individual’s SAGAT scores were highly predictive of survivability (r = .69), but awareness of the teammates’ SA was not predictive. Yuan, She, Li, Zhang, and Wu (2016) found that awareness of teammate SA was negatively correlated with own SA in their study, likely due to competing task demands. Thus, the value of team meta-SA appears not to be supported.

SA Correlation

A number of researchers examined the correlations between the SA of different team members, or sub-teams. For example, Stetrevik (2012) and Cuevas and Bolstad (2010) examined correlations between team members’ and team leaders’ SA. In analyzing the correlation between SA of one’s own requirements and of the teammates’ situation as measured by SAGAT, Sulistyawati et al. (2008) found a marginal correlation (r = .60, p = .06).

These studies demonstrate that TSA and SSA can be derived from SAGAT data which is both objective and highly validated as a measure of individual SA (Endsley, 2000, 2019). These measures provide an independent assessment of the quality of TSA and SSA to support research on the various factors that effect it, and to provide a clear outcome measure for comparison to TSA process measures.

CONCLUSIONS

TSA has become widely recognized as critical for effective team performance in a wide variety of domains. Considerable research has been conducted over the past 30 years demonstrating the importance of team SA processes, team SA devices, and team SA mechanisms for supporting team SA requirements. This body of research provides a solid foundation for the development of training programs and display design guidelines for improving SSA and TSA. Further, existing TSA measurement approaches have demonstrated utility for supporting research on TSA and SSA, and can be used to evaluate the effectiveness of design and training interventions. Overall, the study of TSA has achieved a considerable degree of maturity, however, extensive work is still needed to extend these findings into many real-world training and design applications.

REFERENCES

Artman, H. (1999). Situation awareness and co-operation within and between hierarchical units in dynamic decision making. Ergonomics, 42(11), 1404-1417.

Berggren, P., Prytz, E., Johansson, B., & Nahlinder, S. (2011). The relationship between workload, teamwork, situation awareness, and performance in teams: A microworld study. In Proceedings of the proceedings of the human factors and ergonomics society annual meeting (pp. 851-855). Los Angeles, CA: Sage Publications.

Bolstad, C. A., Cuevas, H., Gonzalez, C., & Schneider, M. (2005). Modeling shared situation awareness. Paper presented at the 14th conference on Behavior Representation in Modeling and Simulation (BRIMS). Los Angles, CA.

Bolstad, C. A., & Endsley, M. R. (1999). Shared mental models and shared displays: An empirical evaluation of team performance. In Proceedings of the 43rd annual meeting of the human factors and ergonomics society (pp. 213-217). Santa Monica, CA: Human Factors and Ergonomics Society.

Bolstad, C. A., & Endsley, M. R. (2000). The effect of task load and shared displays on team situation awareness. In Proceedings of the 14th triennial congress of the international ergonomics association and the 44th annual meeting of the human factors and ergonomics society (pp. 189-192). Santa Monica, CA: Human Factors and Ergonomics Society. Bolstad, C. A., & Endsley, M. R. (2003). Measuring shared and team situation awareness in the army’s future objective force. In Proceedings of the human factors and ergonomics society 47th annual meeting (pp. 364-373). Santa Monica, CA: Human Factors and Ergonomics Society.

Bolstad, C. A., & Endsley, M. R. (2005, Fall). Choosing team collaboration tools: Lessons learned fron disaster recovery efforts. Ergonomics in Design, 7-13.

Bolstad, C. A., Foltz, P., Franzke, M., Cuevas, H. M., Rosenstein, M., & Costello, A. M. (2007). Predicting situation awareness from team communications. In Proceedings of the human factors and ergonomics society 51st annual meeting (pp. 789-793). Santa Monica, CA: Human Factors and Ergonomics Society.

Bolstad. C. A.. Riley, J. M.. Jones. D. G., & Endsley, M. R. (2002). Using goal directed task analysis with Army brigade officer teams. In Proceedings of the 46th annual meeting of the human factors and ergonomics society (pp. 472-476). Santa Monica, CA: Human Factors and Ergonomics Society.

Bonney, L., Davis-Sramek, B., & Cadotte, E. R. (2016). Thinking about business markets: A cognitive assessment of market awareness. Journal of Business Research, 69(8), 2641-2648.

Brooks, J. O., Switzer, F. S„ & Gugerty, L. (2003). Effects of situation awareness training on novice process control plant operators. In Proceedings of the human factors and ergonomics society annual meeting (pp. 606-609). Los Angeles, CA: Sage.

Buchler, N„ Fitzhugh, S. M., Marusich, L. R., Ungvarsky, D. M., Lebiere, C., & Gonzalez, C. (2016). Mission command in the age of network-enabled operations: Social network analysis of information sharing and situation awareness. Frontiers in Psychology, 7,937.

Chidester, T. R., Kanki, B. G., Foushee, H. C., Dickinson, C. L., & Bowles, S. V. (1990). Personality factors in flight operations: Volume /. Leadership characteristics and crew performance in a full-mission air transport simulation (NASA Tech Memorandum No. 102259). Moffett Field. CA: NASA Ames Research Center.

Citera, M., McNeese, M. D., Brown, С. E„ Selvaraj, J. A.. Zaff, B. S.. & Whitaker, R. D. (1995). Fitting information systems to collaborating design teams. Journal of the American Society for Information Science, 46(1), 551-559.

Cooke, N. J„ & Gorman, J. C. (2009). Interaction-based measures of cognitive systems. Journal of Cognitive Engineering and Decision Making, 5(1), 27-46.

Cooke, N. J., Kiekel, P. A., & Helm, E. E. (2001). Measuring team knowledge during skill acquisition of a complex task. International Journal of Cognitive Ergonomics, 5(3), 297-315.

Cooke, N. J.. Kiekel, P. A., Salas. E.. Stout, R., Bowers, C„ & Cannon-Bowers. J. A. (2003). Measuring team knowledge: A window to the cognitive underpinnings of team performance. Croup Dynamics, Theory, Research and Practice, 7(3), 179-199.

Cooke, N. J., Salas, E., Kiekel, P. A., & Bell, B. (2004). Advances in measuring team cognition. In Team cognition: Understanding the factors that drive process and performance (pp. 83-106). Washington, DC: American Psychological Association.

Cooke, N. J., Stout, R. E., Rivera, K., & Salas, E. (1998). Exploring measures of team knowledge. In Proceedings of the human factors and ergonomics society annual meeting (pp. 215-219). Los Angeles, CA: Sage.

Cooke, N. J., Stout, R. J., & Salas, E. (2001). A knowledge elicitation approach to the measurement of team situation awareness. In M. McNeese, E. Salas, & M. R. Endsley (Eds.), New trends in cooperative activities: Understanding system dynamics in complex environments (pp. 114-139). Santa Monica, CA: Human Factors and Ergonomics Society.

Coolen, E., Draaisma, J., & Loeffen, J. (2019). Measuring situation awareness and team effectiveness in pediatric acute care by using the situation global assessment technique. European Journal of Pediatrics, 1-14.

Crozier, M. S., Ting, H. Y., Boone. D. C.. O'Regan. N. B.. Bandrauk. N.. Furey, A.....Hogan,

M. P. (2015). Use of human patient simulation and validation of the Team Situation Awareness Global Assessment Technique (TSAGAT): A multidisciplinary team assessment tool in trauma education. Journal of Surgical Education, 72(1), 156-163.

Cuevas, H. M., & Bolstad, C. A. (2010). Influence of team leaders’ situation awareness on their team’s situation awareness and performance. In Proceedings of the human factors and ergonomics society annual meeting (pp. 309-313). Los Angeles, CA: Sage.

Endsley, M. R. (1988). Design and evaluation for situation awareness enhancement. In Proceedings of the human factors society 32nd annual meeting (pp. 97-101). Santa Monica, CA: Human Factors Society.

Endsley, M. R. (1995a). Theoretical underpinnings of situation awareness: A critical review. In D. J. Garland & M. R. Endsley (Eds.), Experimental analysis and measurement of situation awareness (pp. 17-23). Daytona Beach, FL: Embry-Riddle University.

Endsley, M. R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32-64.

Endsley, M. R. (2000). Direct measurement of situation awareness: Validity and use of SAGAT. In M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and measurement (pp. 147-174). Mahwah, NJ: LEA.

Endsley, M. R. (2004). Situation awareness: Progress and directions. In S. Banbury & S. Tremblay (Eds.), A cognitive approach to situation awareness: Theory, measurement and application (pp. 317-341). Aldershot: Ashgate Publishing.

Endsley, M. R. (2015a). Situation awareness misconceptions and misunderstandings. Journal of Cognitive Engineering and Decision Making, 9(1), 4-32.

Endsley, M. R. (2015b). Situation awareness: Operationally necessary and scientifically grounded. Cognition, Technology and Work, /7(2), 163-167.

Endsley, M. R. (2018). Expertise and situation awareness. In K. A. Ericsson, R. R. Hoffman, A. Kozbelt & A. M. Williams (Eds.), Cambridge handbook of expertise and expert performance (2nd ed., pp. 714-744). Cambridge: Cambridge University Press.

Endsley, M. R. (2019). A systematic review and meta-analysis of direct, objective measures of SA: A comparison of SAGAT and SPAM. Human Factors, https://doi.org/10.1177 %2F0018720819875376

Endsley. M. R., Bolstad. C. A., Jones, D. G., & Riley, J. M. (2003). Situation awareness oriented design: From user’s cognitive requirements to creating effective supporting technologies. In Proceedings of the 47th annual meeting of the human factors and ergonomics society (pp. 268-272). Santa Monica, CA: Human Factors and Ergonomics Society.

Endsley, M. R., Bolte, B., & Jones, D. G. (2003). Designing for situation awareness: An approach to human-centered design.

Endsley. M. R.. Farley. T. C.. Jones, W. M„ Midkiff, A. H., & Hansman, R. J. (1998). Situation awareness information requirements for commercial airline pilots (ICAT-98-1). Cambridge, MA: Massachusetts Institute of Technology International Center for Air Transportation.

Endsley, M. R., Hansman, R. J., & Farley, T. C. (1998). Shared situation awareness in the flight deck—АТС system. In Proceedings of the A1AA/IEEE/SAE I7th digital avionics systems conference. Bellevue, WA: Isogen International Corporation.

Endsley. M. R.. Holder, L. D„ Leibrecht. В. C„ Garland, D. C„ Wampler. R. L., & Matthews. M. D. (2000). Modeling and measuring situation awareness in the infantry operational environment (1753). Alexandria, VA: Army Research Institute.

Endsley, M. R., & Jones, D. G. (2012). Designing for situation awareness: An approach to human-centered design

Endsley, M. R., & Jones, W. M. (1997). Situation awareness, information dominance, and information warfare (AL/CF-TR-1997-0156). Wright-Patterson AFB, OH: United States Air Force Armstrong Laboratory.

Endsley, M. R., & Jones, W. M. (2001). A model of inter- and intrateam situation awareness: Implications for design, training and measurement. In M. McNeese, E. Salas, & M. Endsley (Eds.), New trends in cooperative activities: Understanding system dynamics in complex environments (pp. 46-67). Santa Monica, CA: Human Factors and Ergonomics Society.

Endsley, M. R„ & Robertson, M. M. (1996). Team situation awareness in aviation maintenance. In Proceedings of the 40 th annual meeting of the human factors and ergonomics society (pp. 1077-1081). Santa Monica, CA: Human Factors and Ergonomics Society.

Endsley, M. R., & Rodgers, M. D. (1994). Situation awareness information requirements for en route air traffic control. In Proceedings of the human factors and ergonomics society 38th annual meeting (pp. 71-75). Santa Monica, CA: Human Factors and Ergonomics Society.

Espevik, R., Johnsen, В. H., & Eid, J. (2011). Outcomes of shared mental models of team members in cross training and high-intensity simulations. Journal of Cognitive Engineering and Decision Making, 5(4), 352-377.

Farley, T. C., Hansman, R. J., Amonlirdviman, K., & Endsley, M. R. (2000). Shared information between pilots and controllers in tactical air traffic control. Journal of Guidance, Control and Dynamics, 23(5), 826-836.

Farley, T. C., Hansman, R. J., Endsley, M. R., Amonlirdviman, K., & Vigeant-Langlois, L. (1998). The effect of shared information on pilot/controller situation awareness and re-route negotiation. In Proceedings of the 2nd FAA/Eurocontrol ATM R&D seminar. Washington. DC: FAA.

Foushee. H. C.. Lauber. J. K.. Baetge. M. M, & Acomb. D. B. (1986). Crew factors in flight operations: III. The operational significance of exposure to short-haul air transport operations (NASA Tech Memorandum 88322). Moffett Field. CA: NASA Ames Research Center.

Gardner, A. K., Kosemund, M., & Martinez, J. (2017). Examining the feasibility and predictive validity of the SAGAT tool to assess situation awareness among medical trainees. Simulation in Healthcare. /2(1), 17-21.

Gorman, J. C., Cooke, N. J., & Amazeen, P. G. (2010). Training adaptive teams. Human Factors, 52(2), 295-307.

Gorman, J. C„ Cooke, N. J.. Pederson. H. K.. Connor, O. O., & DeJoode, J. A. (2005). Coordinated Awareness of Situation by Teams (CAST): Measuring team situation awareness of a communications glitch. In Proceedings of the human factors and ergonomics society 49th annual meeting (pp. 274-277). Santa Monica, CA: Human Factors and Ergonomics Society.

Gorman, J. C., Cooke, N. J., & Winner, J. L. (2006). Measuring team situation awareness in decentralized command and control environments. Ergonomics, 49(12-13), 1312-1325.

Gorman, J. C., Weil, S. A., Cooke, N., & Duran, J. (2007). Automatic assessment of situation awareness from electronic mail communication: Analysis of the Enron dataset. In Proceedings of the proceedings of the human factors and ergonomics society annual meeting (pp. 405-409). Los Angeles, CA: Sage Publications.

Gross, N., & Kluge, A. (2014). Predictors of knowledge-sharing behavior for teams in extreme environments: An example from the steel industry. Journal of Cognitive Engineering and Decision Making, S(4), 352-373.

Hallbert, В. P. (1997). Situation awareness and operator performance: Results from simulator-based studies. In Proceedings of the IEEE sixth conference on human factors and power plants (pp. 18/11-18/16). New York: IEEE.

Hauland, G. (2008). Measuring individual and team situation awareness during planning tasks in training of en route air traffic control. The International Journal of Aviation Psychology, 18(3), 290-304.

Hirokawa, R. Y. (1983). Group communication and problem solving effectiveness: An investigation of group phases. Human Communication Research, 9, 291-305.

Javed, Y., Norris, T., & Johnston, D. (2012). Evaluating SAVER: Measuring shared and team situation awareness of emergency decision makers. Paper presented at the 9tli international ISCRAM conference. Vancouver, Canada.

Jeffs, L.. Acott, A., Simpson. E., Campbell, H.. Irwin, T.. Lo. J.....Cardoso. R. (2013).

The value of bedside shift reporting enhancing nurse surveillance, accountability, and patient safety. Journal of Nursing Care Quality, 28(3), 226-232.

Jones, W. M. (1997). Enhancing team situation awareness: Aiding pilots in forming initial mental models of team members. In Proceedings of the ninth international symposium on aviation psychology (pp. 1436-1441). Columbus, OH: The Ohio State University.

Kanki, B. G., Lozito, S., & Foushee, H. C. (1989). Communication indices of crew coordination. Aviation, Space and Environmental Medicine, 60, 56-60.

Kitchin, J., & Baber, C. (2017). The dynamics of distributed situation awareness. In Proceedings of the human factors and ergonomics society annual meeting (pp. 277- 281). Los Angeles, CA: Sage Publications.

Klein, G. A., Zsambok, С. E., & Thordsen, M. L. (1993, April). Team decision training: Five myths and a model. Military Review, pp. 36-42.

Leggatt, A. (2004). Objectively measuring the promulgation of commander’s intent in a coalition effects based planning experiment (MNE3). In Proceedings of the 9th international command and control research and technology symposium (pp. 143-177). Copenhagen: Defense Command Denmark.

Midkiff, A. H., & Hansman, R. J. (1992). Identification of important party-line information elements and implications for situational awareness in the datalink environment. In Proceedings of the SAE Aerotech conference and exposition. Anaheim, CA: Sage.

Mohammed, S., Ferzandi, L., & Hamilton, K. (2010). Metaphor no more: A 15-year review of the team mental model construct. Journal of Management, 36(4), 876-910.

Morgan, P., Tregunno, D., Brydges, R., Pittini, R., Tarshis, J., Kurrek, M.,. . . Ryzynski, A. (2015). Using a situational awareness global assessment technique for interprofessional obstetrical team training with high fidelity simulation. Journal of Interprofessional Care, 29(1), 13-19.

Mosier, K. L., & Chidester, T. R. (1991). Situation assessment and situation awareness in a team setting. In Y. Queinnec & F. Daniellou (Eds.), Designing for everyone

National Transportation Safety Board. (1994). A review of flight crews involved in major accidents of U.S. air carriers 1978-1990. Washington, DC: Author.

Orasanu, J. (1990, July). Shared mental models and crew decision making. In Proceedings of the 12th annual conference of the cognitive science society. Cambridge, M A: Cognitive Science Society.

Orasanu, J., & Salas, E. (1993). Team decision making in complex environments. In G. A. Klein, J. Orasanu, R. Calderwood, & С. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 327-345). Norwood, NJ: Ablex.

Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2008). Situation awareness, mental workload and trust in automation: viable empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making, 2(2), 140-160.

Parke, B., & Kanki, B. G. (2008). Best practices in shift turnovers: Implications for reducing aviation maintenance turnover errors as revealed in ASRS reports. The International Journal of Aviation Psychology, 18(1), 72-85.

Parush, A., Hazan, M., & Shtekelmacher, D. (2017). Individuals perform better in teams but are not more aware-performance and situational awareness in teams and individuals. In Proceedings of the human factors and ergonomics society annual meeting (pp. 1173— 1177). Los Angeles, CA: Sage.

Parush, A., & Ma, C. (2012). Team displays work, particularly with communication breakdown: Performance and situation awareness in a simulated forest fire. In Proceedings of the proceedings of the human factors and ergonomics society annual meeting (pp. 383-387). Los Angeles, CA: Sage Publications.

Parush, A., Mastoras, G., Bhandari, A., Momtahan, K., Day, K., Weitzman, В.....Calder, L.

(2017). Can teamwork and situational awareness (SA) in ED resuscitations be improved with a technological cognitive aid? Design and a pilot study of a team situation display. Journal of Biomedical Informatics, 76, 154-161.

Patrick, J., James, N., Ahmed, A., & Halliday, P. (2006). Observational assessment of situation awareness, team differences and training implications. Ergonomics, 49(4), 393-417.

Price, T. F., & LaFiandra, M. (2017). The perception of team engagement reduces stress induced situation awareness overconfidence and risk-taking. Cognitive Systems Research, 46, 52-60.

Prince, C., Ellis, E., Brannick, M. T., & Salas, E. (2007). Measurement of team situation awareness in low experience level aviators. The International Journal of Aviation Psychology, /7(1), 41-57.

Robertson, M. M., & Endsley, M. R. (1997a). Development of a situation awareness training program for aviation maintenance. In Proceedings of the human factors and ergonomics society 41st annual meeting (pp. 1163-1167). Santa Monica, CA: Human Factors and Ergonomics Society.

Robertson, M. M., & Endsley, M. R. (1997b). Impact of distributed teams on organizational effectiveness. In Proceedings of the 13th triennial congress of the international ergonomics association (pp. 519-521). Helsinki: Finnish Institute of Occupational Health.

Rosenman, E. D., Dixon, A. J., Webb, J. M., Brolliar, S., Golden, S. J., Jones, K. A., . . . Chao, G. T. (2018). A simulation-based approach to measuring team situational awareness in emergency medicine: A multicenter, observational study. Academic Emergency Medicine, 25(2), 196-204.

Ssetrevik, B. (2012). A controlled field study of situation awareness measures and heart rate variability in emergency handling teams. In Proceedings of the human factors and ergonomics society annual meeting (pp. 2006-2010). Los Angeles, CA: Sage.

Ssetrevik, B., & Eid, J. (2014). The “similarity index” as an indicator of shared mental models and situation awareness in field studies. Journal of Cognitive Engineering and Decision Making, 8(2), 119-136.

Salas, E., Dickinson, T. L., Converse, S., & Tannenbaum, S. I. (1992). Toward an understanding of team performance and training. In R. W. Swezey & E. Salas (Eds.), Teams: Their training and performance (pp. 3-29). Norwood, NJ: Ablex.

Salas, E., Prince, C., Baker, D. P., & Shrestha, L. (1995). Situation awareness in team performance: Implications for measurement and training. Human Factors, 37(1), 123-136.

Salmon, P., Stanton, N., Walker, G., Baber, C., Jenkins, D., McMaster, R., & Young, M. (2008). What is really going on? Review of situation awareness models for individuals and teams. Theoretical Issues in Ergonomics Science, 9(4), 297-323.

Saner, L. D., Bolstad, C. A., Gonzalez, C., & Cuevas, H. M. (2009). Measuring and predicting shared situation awareness in teams. Journal of Cognitive Engineering and Decision Making, 5(3), 280-308.

Saner, L. D., Bolstad, C. A., Gonzalez, C., & Cuevas, H. M. (2010). Predicting shared situation awareness in teams: A case of differential SA requirements. In Proceedings of the human factors and ergonomics society annual meeting (pp. 314-318). Los Angeles, CA: Sage Publications.

Seet, A. W„ Teh, C. A., Soo, J. K., & Teo, L. (2004). Constructible assessment for situation awareness in a distributed C2 environment. Singapore: DSO National Laboratories.

Segal, L. D. (1994). Actions speak louder than words. In Proceedings of the human factors and ergonomics society 38th annual meeting (pp. 21-25). Santa Monica, CA: Human Factors and Ergonomics Society.

Sorensen, L. J., Stanton, N., & Banks, A. P. (2010). Back to SA school: Contrasting three approaches to situation awareness in the cockpit. Theoretical Issues in Ergonomics Science, /2(6), 451-471.

Stout, R. J., Cannon-Bowers, J. A., & Salas, E. (2017). The role of shared mental models in developing team situational awareness: Implications for training situational awareness (pp. 287-318). Abingdon: Routledge.

Stout, R. J., Cannon-Bowers, J. A., Salas, E., & Milanovich, D. M. (1999). Planning, shared mental models, and coordinated performance: An empirical link is established. Human Factors, 4/(1), 61-71.

Strater, L. D., Cuevas, H. M., Connors, E. S., Gonzalez, C., Ungvarsky, D. M., & Endsley, M. R. (2010). An investigation of technology-mediated ad hoc team operations: Consideration of components of team situation awareness. In K. Mosier & U. Fischer (Eds.), Informed by knowledge: Expert performance in complex situations (pp. 153-168). Boca Raton, FL: Taylor and Francis.

Strater, L. D.. Cuevas, H. M.. Connors, E. S„ Ungvarsky, D. M„ & Endsley, M. R. (2008). Situation awareness and collaborative tool usage in ad hoc command and control teams. In Proceedings of the proceedings of the human factors and ergonomics society annual meeting (pp. 468-472). Los Angeles, CA: Sage Publications.

Sulistyawati, K., Chui, Y. P., & Wickens, C. D. (2008). Multi-method approach to team situation awareness. In Proceedings of the human factors and ergonomics society annual meeting (pp. 463-467). Los Angeles, CA: Sage.

Sulistyawati, K., Wickens, C. D., & Chui, Y. P. (2009). Exploring the concept of team situation awareness in a simulated air combat environment. Journal of Cognitive Engineering and Decision Making, 3(4), 309-330.

Taylor, R. M., Endsley, M. R., & Henderson, S. (1996). Situational awareness workshop report. In B. J. Hayward & A. R. Lowe (Eds.), Applied aviation psychology: Achievement, change and challenge (pp. 447-454). Aldershot: Ashgate Publishing Ltd.

Tiferes, J., & Bisantz, A. M. (2018). The impact of team characteristics and context on team communication: An integrative literature review. Applied Ergonomics, 68, 146-159.

United Kingdom Air Accidents Investigation Branch. (1990). Report on the accident to Boeing 737-400 G-OBME near Kegworth, Leicestershire on 8 January 1989 (Report No. 4/90 (EW/C1095)). Farnborough, UK: Department for Transport.

Wellens, A. R. (1993). Group situation awareness and distributed decision making: From military to civilian applications. In N. J. Castellan Jr. (Ed.), Individual and group decision making (pp. 267-291). Hillsdale, NJ: Lawrence Erlbaum.

Wickens, C. D. (2008). Situation awareness: Review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Human Factors, 50(3), 397-403.

Xiao, Y., Mackenzie, C. F., & Patey, R. (1998). Team coordination and breakdowns in a real-life stressful environment. In Proceedings of the human factors and ergonomics society 42nd annual meeting (pp. 186-190). Santa Monica, CA: Human Factors and Ergonomics Society.

Yuan, X., She, M., Li, Z., Zhang, Y., & Wu, X. (2016). Mutual awareness: Enhanced by interface design and improving team performance in incident diagnosis under computerized working environment. International Journal of Industrial Ergonomics, 54, 65-72.

 
Source
< Prev   CONTENTS   Source   Next >