One fundamental issue for implementation of VR in experimental contexts is the role of perception-action coupling, namely, can sensory stimulation be action-independent? We can seek insight from the action selection literature to understand the links between sensory experience and decision-making (Barron et al., 2015), motor leaming/control, and agency (Sidaras, Vuorre, & Haggard, 2017). We can also use computational models to understand these relationships better. Using an artificial life model, Seth (2007) defines action selection as a set of coupled sensorimotor processes. Overall, action selection allows us to prioritize our actions given a continuous flow of stimuli. Viewing the flow of perception from an ecological standpoint (Gibson, 2015) is essential to understanding the nature and dynamic aspects of virtual experience.
Another way to understand the link between cognition and virtual reality is through more formal cognitive modeling (Figure 16.1). Whereas cognitive modeling has been used to predict and explain multiple interacting components of human performance in a wide range of task environments, the role of higher-order phenomena such as awareness and engagement has largely been avoided. One way to view a VR-specific cognitive model is to focus on the components of the hypothetical virtuality network.
Toward a “Virtuality Network”
It is unclear whether cognition in virtual environments is mediated by a combination of well- characterized cognitive functions or whether it requires a novel set of cognitive functions heretofore unknown. Much as there are attentional, emotional, and default activity networks in the brain, there might also be a “virtuality” network that mediates environmental features unique to VR (Figure 16.2).
The functional components of a virtuality network might involve interactions between premotor and motor cortex, spatial cognition centers, multimodal integration centers, and brain regions involved in object and face processing. Yet other types of functionality, such as spatial navigation in high-dimensional artwork, might also involve a mechanism related to abstract reasoning or consciousness. Alternatively, we might also leam something about the evolutionary cognitive substrate of virtuality by surveying the literature on the use of virtual reality in animal models (Alicea, 2015).
Figure 16.1 An Example of a Cognitive Mode Based on the Pandemonium Model of Selfridge (1959)
Note: Layers include decision-making (top), cognitive demons (second from top), computation demons (second from
bottom), and data demons (bottom).
Figure 16.2 Model of a Virtuality Network as a Putative Cognitive Model
WHEN VIRTUAL REALITY BECOMES USEFUL
To discuss how virtual reality becomes useful, several topical areas will be highlighted. These include but are not limited to the necessary creation of new methods and phenomenology, the inclusion of sustained behavioral measurement and unique modes of interaction, and the enabling of unique behavioral modes.
New Methods and Phenomenology
Aside from considering how VR can be advantageous (or deleterious) in measuring neurobehavior, it is quite useful for enabling both new methods and phenomenology. These include, but are not limited to, sustained behavioral measurement, unique modes of interaction, and unique modes of behavior. As we will see, a combination of ingenious methods and the unique features of virtual environments can provide an exciting new perspective on cognition and brain function.
Sustained Behavioral Measurement
From a quantitative standpoint, permitting participants to engage in behaviors such as navigation, aggression, and attentional focusing over long periods of time (more than a single trial or set of trials) has a number of advantages. Whereas specific behaviors can be segmented and classified for hypothesis testing, continuous media stimuli can reveal longer-term cumulative and emergent cognitive effects. Continuous measurement allows us to acquire time-series data, which requires tools such as frequency domain and dynamic causality analysis (Ozaki, 2012). Continuous data may also provide a window into a whole class of dynamical phenomena such as self-organized criticality, phase transitions, and metastability to be observed (Beggs, 2008; Kozma & Freeman, 2017). More specifically, it allows for time-series data to be collected, which in turn allows for new types of analyses and contains information content not measurable as a set of discrete, randomized observations.
Unique Modes of Interaction
Aside from effects relevant to individual cognition, YR also enables people to interact in ways that are impossible in the physical world. When this is scaled up so that many people have access to the same content, it provides a imique window into social neuroscience by allowing us to understand how groups of people behave in a dynamic fashion. In particular, a method called hyperscanning (Balconi & Vanutelli, 2017) can provide a link between human social interactions and the simultaneous imaging of multiple brains. One example comes fr om the practice of neuroeconomics, wherein hyperscanning is used to enable the dynamic monitoring of continuous and virtually-mediated economic transactions (Rangel, Camerer, & Montague, 2008). Overall, hyperscanning and other social measurement techniques hold great promise for communication neuroscience.
The use of affordances in YR object design also provides a means to introduce new semantic information into the medium. In particular, affordances provide signifiers within the YR environment (Norman, 2008) in a way that provides meaning and constructivist potential (Olson, 2017). Rather than providing opportunities for perceptual illusion, signifiers provide the symbolic glue for perception and action by clarifying the specific role of affordances in the environment. Affordances also provide physical cues to the individual from the virtual environment. According to Regia-Corte et al. (2013), affordances (in this case, a slanted surface) allow an individual to infer physical properties that support judgments that influence perception and action.
Unique Modes of Behavior
Ultimately, exposure to VR may lead to new behaviors. At a basic level, perceptual adaptations, such as the prism effect (Chapman et al, 2010) are well-known after-effects of exposure to VR environments. According to Gonzalez-Franco and Lanier (2017), there are three types of illusion associated with YR: place, embodiment, and plausibility. The place illusion occurs when the virtual environment transports the user to a new spatial location. It could be a familiar location or an exotic one. The embodiment illusion involves experiencing the virtual environment either in a disembodied form or as an avatar. As the intersection of both the place and embodiment illusion, the plausibility illusion involves self-examination as to whether or not an experience is actually happening. In this way, the plausibility illusion overlaps with some of the theoretical issues faced by the presence community.
There are other perceptual illusions that are enabled as a consequence of the embodiment illusion. One of these is called body transfer and is exemplified by the rubber (sometimes called marble) hand illusion (Senna, Maravita, Bolognini, & Parise, 2014). In this example, body transfer occurs when a participant begins to treat a virtual hand as part of his or her own body. While this illusion becomes behaviorally manifest by the artificial hand becoming equivalent to an additional appendage, the phenomenon is represented in the brain as a complex set of multimodal interactions (Ehrsson, Holmes, & Passingham, 2005). Returning to the virtuality network shown in Figure 16.1, there are a number of other possible illusions involving the interactions between the peripheral nervous system and the central nervous system. This is particularly true in cases where the body is not in the same contextual frame as the visual and auditory systems.