Prospective Principles of Memory, Perception, and Learning
Let's review a few prospective principles, starting with perception.
At first glance, the perceptual process is a matter of receiving information from the environment. It has, however, been a long time since anyone regarded this process as mere passive reception. Although our attention is sometimes seized by an unexpected sight or sound, it can also be consciously directed and focused, thereby shaping the perceptual stream, such as when our ice fisherman closely studies the slowly swimming fish. And perceptual processing must impose some organization on the flow of sensory impressions if the mind is to make any sense of what it sees and hears. In both these cases, however, the sights and sounds still flow from the outside in. After all, if the sights and sounds were to come from within instead, then that would be hallucination rather than perception, wouldn't it? And yet it increasingly appears that much of our perceptual experience has exactly that character.
Like time and energy, mental processing and nerve channel capacities are limited. So why waste them receiving, interpreting, and storing visual inputs that aren't changing or that could easily be predicted? Information value is the inverse of predictability, so the more a fact could be predicted on the basis of what you already know, the less learning that fact will contribute to what you know. At the limit, wholly redundant information is no information gain at all.
The science of communication theory was born with the recognition of this inverse relationship between information value and predictability. It is this relationship that enables technologists to identify effective and efficient ways of encoding information. Morse code, for example, can pack as many messages as possible into a single time interval by choosing shorter and simpler codes for the more frequent letters in English. A single "dot," for example, stands for "e," while "q" is coded with three "dashes" and one "dot." Your perceptual system, too, is designed for efficient coding. For example, if the visual scene is unchanging, or changing in some perfectly predictable way, then there would be no need for the whole of this scene to be recorded directly and passed along the perceptual pathway. Instead, the brain could, like an efficient graphic processor or file compression program in a computer, use prediction to generate what is static or predictable and reserve valuable channel and processing capacity for what is not predictable. This enables the brain to focus its resources where the greatest potential for information gain lies. And a growing body of evidence suggests that perception does indeed follow this principle (Clark, 2013; King, Zylberberg, & DeWeese, 2013), and the following is true:
Perception is as much about self-generating the information you already know as receiving the information you don't.
The human eye has a "blind spot," corresponding to where the optic nerve attaches to the retina. Yet we never see a blank area in our visual field. The perceptual system uses information about adjacent areas of the visual field to predict what would be present in the blind spot and actively retouches the visual field to fill this spot in. This point can be applied more widely: Efficient perceivers, able to keep up with the potentially huge flow of information from the eye, will also be effective anticipators of what they are about to see and hear, and so be able to triage the incoming signal for what is of greatest interest. Indeed, only a small fraction of our visual field is actually in focus at a given time—the area of the tiny fovea centralis in the retina—while the rest is given its full, focused appearance by supplementation from internal sources, much like the blind spot. And, at the limit, a perfectly predictable signal, such as the ticking of the clock in your kitchen, need not be perceived at all once you've been in the room for a few moments. Until your attention turns to the clock—then your perceptual system helpfully puts the ticking back in.
This last point ties the role of anticipation in perception directly to anticipation's role in learning. A rethinking of learning in intelligent animals has taken place within the last generation. The previous generation thought of animal learning as the ingraining of habits by repeated reinforcement of a link between a particular stimulus and a particular response, for example, a rat's turning left and finding food in a simple T-maze. But this theory proved incapable of accounting for the behaviors actually observed in animal learning, because intelligent animals, human infants included, seemed more interested in predictive relations among kinds of events rather than repetitious links among particulars (Aslin, Saffran, & Newport, 1998; Rescorla, 1988). Thus consider the following inference:
Expectations serve as "hypotheses" about the world around us, and the perceptual system is designed to project these hypotheses and detect errors, generating negative feedback and updating when actual perceptual input does not match the expectation—all experience is
experimentation, and error serves as a "teaching signal."
This implication is foundational in modern formal learning theory. As we will see when we discuss the philosophical case for prospection, "unbiased" learning is a nonstarter. A system that began with no expectations about how the world will be, and simply allowed individual experiences to accumulate, would end up with a pile of disconnected facts and no idea of what might come next. After all, its accumulated information is entirely about particular events that have already taken place and so in itself says nothing about what will happen next. The system would simply draw a blank if asked to guide future action. More effective learners will start off with certain ideas of what to expect, and then use feedback from how well actual experience fits these ideas to reshape their ideas going forward. Expectations thus support both the guidance of "trial" and the possibility of "error" in trial-and-error learning.
Building expectations into the perceptual system, therefore, can serve a dual role: Expectations are the key to efficient coding of incoming information, and they are essential for effective learning from what perception, with help of coding, tells us. It is small wonder that a prospection-based approach to understanding the mind promises to be so powerful.
Let us now turn to a third mental system—memory. Thanks to decades of research and the clinical record of individuals who suffered selective damage in brain regions linked to memory, neuroscientists have come to see memory as a multipart system with a functional division of labor. The mind is not, as some early modern philosophers supposed, a wax tablet upon which experience is simply impressed. Memory and recall are activities of the brain, expending energy and tying up resources. So here, too, the demands of efficiency and effectiveness play a shaping role. And once again, we will see how a prospection-based account of memory enables us to make sense of this processing.
In the broadest division, there is short-term and long-term memory. Short-term memory is seen as an on-line mental workspace that holds very recent information, say, the phone number you have just looked up, the last few moments of your experience, or the idea of what you came upstairs to look for. Short-term memory is highly flexible and readily accessible, but has limited capacity—most of us reach a limit at between 4 and 7 separate items. As new items are being pulled into short-term memory, older items are crowded out.
Long-term memory is in many ways the opposite. It is often seen as an off-line archive where information is stored for hours, days, or years, without requiring active attention. It is much less flexible, and for practical purposes, it is almost limitless, so that adding new information need not drive out old. But the resulting massive collection of information is not always easy to access, although the categories used in storing information make following a thread of association or retrieving a given memory easier.
What is kept in long-term memory? Traditionally, it is divided into three kinds. There is "episodic memory," which stores representations of particular events (the time you knocked over a glass of red wine at your first lunch with your boss). "Procedural memory" is a stored repertoire of ways of doing things (such as looking up an unknown word by finding some appropriate source and following alphabetical order). "Motor memory" stores patterns of implementation (such as pedaling a bicycle or writing the letter "q"). (In humans and some other intelligent species there is also "semantic memory," stored concepts, categories, or words, e.g., the distinction between "animate" and "inanimate" objects, or the fact that a "joist" is a beam holding up floors, while a "rafter" is a beam holding up a roof.)
All of this makes good sense from the standpoint of design—a well-built robot would need to perform all three (indeed, if intelligent, all four) of these functions in some coordinated way, too. And yet, memory as we actually find it appears to have a number of peculiar features that make the preceding description a bit misleading.
First consider the "archive" of long-term memory. Papers deposited in an archive might yellow and become brittle, but they are not supposed to be removed or altered in content. This constancy across time enables us to use archives to check our recollections, search land titles, settle lawsuits, and construct and test historical hypotheses. It is a singularly important feature of an archive that the information contained therein is not revised in light of more recent events. Researchers are required to leave documents in just the condition in which they find them and can take into archival collections only pencils and cameras to record what they learn. And for good reason—the whole point of an archive would seem to be defeated if users could change what they find to fit their own version of reality. Why, then, is the archive of human memory so different? It is now widely appreciated that human memory is dynamic, not static. It is suggestible, prone to conflation, guilty of imposing narrative unity upon what was experienced as a disordered sequence. Recalling an event in a new context, thus pulling the information from episodic memory into working memory, can lead to new information being inserted seamlessly into the memory when it is restored. Coaching of witnesses by police, prosecutors, or defense attorneys need not force the witness to lie about what they remember, instead, it subtly causes them to store revised versions of what they originally recalled, so that when they are called on to testify in court, they sincerely remember the revised version.
Other forms of dynamism in memory are more spontaneous. Newly learned information can silently enrich or delete existing memories. The emotions people felt or the goals they are pursuing will affect what they store from current experience, and how it is interpreted, elaborated, or connected to other memories. In an essay that marked a turning point in oral history, Alessandro Portelli (2010) described how individuals in Terni, Umbria, who had been present at a protest years earlier during which a worker, Luigi Trastulli, was killed, "recalled" different versions of events. These versions did not vary randomly, but displaced the date and cause of the demonstration in ways that invested Trastulli's death with greater symbolic significance for the labor movement as a whole, and reflected the recollecting individuals' differing subsequent relationship to this movement. Memory, it seemed, silently complied with the demands of narrative meaning. Oral history was never the same again.
If the point of memory were to archive history, these would all be defects. And no doubt sometimes they are. But suppose that the main point of memory is to make a positive contribution to one's ability to face the present and future. "Memory is for doing," we might say. And what we need from memory if we are to improve the odds of contending with the present and future is not a warehouse of disconnected documents, but an ability to extract and recombine bits of information in light of subsequent evidence and along lines of relevance to current or anticipated situations, however novel they might be (Genovesio, Wise, & Passingham, 2014; Schacter, Addis, & Buckner, 2007). This process should draw on the full information available to the individual, synthesizing imaginative representations of what is likely or possible by bringing together the old with the new. And this in turn dictates a continual process of enrichment and reorganization in memory, pulling information out of one context to use it in another, correcting past thoughts or perceptions, developing projectable forward trajectories by giving plausible narrative coherence to sequences of events in ways we did not appreciate at the time, and selectively storing and recalling information of greater pertinence upon what we care about or are called upon to do.
A memory made for doing will not be a static archive but a dynamic relational database, which permits updating records and projecting and evaluating new possibilities in new settings. An archival memory might be spectacularly accurate in some ways, but would lack all benefit of hindsight and be spectacularly time and energy consuming to use. Just ask any archivally based historian, who labors for months or years to reconstruct sequences of events only to arrive at an outcome that we already know to have occurred. Archival historians typically will, as a matter of fidelity to their sources and their craft, reject demands that they predict the future, or say what would have happened if things had gone differently. "The archive cannot tell me that," they'll tell you, "I'd be speculating as a rank amateur rather than speaking as a professional." Yet as individuals, we have no choice but to do our best to predict, identify, and learn from ways things might have gone better had we made different choices. If we really had no expertise in projecting from the past into the future, or from the actual to the possible, then accurate memory would do us little good in any event. So it follows that
Memory must be active and constructive, not passive and fixed—it
must metabolize information into forms that are efficient and effective for the forward guidance of thought and action.
Intriguingly, it appears that our mental circuitry for decisionmaking through imagining the future and counterfactual possibilities is integral with the circuitry for memory (Doll, Shohamy, & Daw, 2015), and this circuitry is alive in all three of these modes in the "default" mode of brain activity (Buckner, Andrews-Hanna, & Schacter, 2008). The brain treats delving into the past and projecting into the future as a unified, ongoing task—because they are. We will discuss this further in the following section, and, especially, in Chapter 4.