AN INTRODUCTION TO INTERMODAL INTERFACES
This chapter discusses intermodal interfaces, which are a class of HCI systems designed to facilitate the transformation of real-world information into computer- usable data with the intention of transforming that data in some way, so that it can be presented back to users to aid in decision making. Intermodal interfaces address a gap between technology’s ability to capture data, and the need for some data to be interpreted and refined anthropologically, prior to being collected, due to subtleties of information meaning that computers cannot handle.
Definition and Overview
Note the distinction between the terms information and data in the aforementioned description. In this work, data are treated as a physical phenomenon, whether that is the movement of electrons, the rendering of magnetic fields, or the polarization of beams of light. An easy way to think about this is that technology is confined to working with data only. This treatment of the term data pushes the term information into the realm of cognitive systems. That is to say, information can only exist within a mind. (And we will leave it to philosophers and artificial intelligence researchers to define the term mind.)
With this model in place, many examples can be identified where the simple collection of data is not synonymous with the collection of information. An easy example of this is having a computer work with spoken language. Even if a system could be built, which would provide perfect transcription of speech, there would remain a question of correct interpretation. Reliable interpretation of speech is beyond the current capabilities of technology-only systems, even when reliable transcription is available. People learn from an early age to appreciate the nuances of verbal communication. Even without accompanying visual cues such as body language or spatial referencing (i.e., pointing at something), humans normally develop the ability to use tone of voice, speaking speed, pauses between words, and similar auditory artifacts to modify their interpretation of the actual spoken words. Moreover, people are able to very rapidly incorporate other auditory cues, inferring demographic information such as gender, age, ethnicity, and so on, to further contextualize our interpretations.
Intermodal interfaces form a partial bridge between information available to a human in the environment—what some refer to as the infosphere (Floridi 2006), and the data-handling capabilities of technology. The other major component of this bridge is one or more people tasked with the interpretive work needed to capture useful information into a digital format.
Intermodal interfaces might be cast along with all other HCI designs, but for these factors:
- 1. The role of intermodal interfaces is to collect information-dense data, which is difficult for technology to interpret, for purposes of decision support. This stands in contrast to the body of work dedicated to the presentation and visualization of data for decision support.
- 2. Consideration must be given for the anticipated decision(s) to be made in order to build-in the facilities needed to support the development of situational awareness as data is collected.
The contrasting role of intermodal interfaces remains when comparisons are made to command and control types of designs, as many of those systems are focused on collecting affective directives from a user rather than collecting situational descriptors. The extent of those directives can be defined programmatically. Although intermodal interfaces may similarly programmatically limit the scope of a user’s inputs, and thus their interpretive range, the decision-support purpose is paramount.
For instance, consider information that is presented in an audio format and must be collected quickly in order to be analyzed and/or shared efficiently as part of a decision-making process. The use-case for this example, and the basis for the experiment described later, is the communications within an emergency operations center (EOC). (In the United States, EOC’s are often referred to as 911-centers in reference to the telephone number used to access them.) Many of the verbal messages coming into the EOC must be interpreted beyond the meaning of the spoken words in order to assess the situation and assign appropriate resources.