Response-Reinforcer Contiguity
Behavior
The interval between the occurrence of the response and the delivery of the reinforcer is a powerful determinant of instrumental learning. This delay of reinforcement gradient has been
Figure 6.15 The effects of marking stimuli on instrumental learning under long delays. Upon entering a Black or a White compartment, rats received one of four treatments designed to mark the choice with a Light, a White Noise, or Handling; the fourth group was not stimulated. Rats spent a 2-min delay interval in the delay box and again received the corresponding marking stimulus before entering the goal box, where they were either reinforced or not, depending on whether the choice had been correct or incorrect. Marking was administered for both correct and incorrect choices.
Source: Lieberman, McIntosh, and Thomas (1979); reproduced with permission from the American Psychological Association.
studied in a variety of procedures. In one experiment (Grice, 1948), rats trained in a simultaneous brightness discrimination in a T-shaped maze were detained in a holding compartment immediately after making a choice. Choice performance deteriorated significantly with delays longer than 0.5 s. Similar effects were described in free-operant lever pressing with rats (Dickinson, Watt, & Griffiths, 1992).
Several procedures can be used to bridge temporal delays so that reinforcement can still strengthen the instrumental response, such as limiting the occurrence of competing responses during the delay period. Rats detained in a small box learned faster with long delays than rats detained in a large box that allowed for a substantial amount of exploratory activity, thus interfering more effectively with the memory of the choice made previously (Spence, 1956).
Introducing a marking stimulus after both correct and incorrect choices, and again before the outcome for that trial, ameliorates the disrupting effects of temporal gaps (Lieberman, McIntosh, & Thomas, 1979). Marking stimuli such as lights, white noise, and even handling by the experimenter facilitated learning of a visual discrimination, even when 2-min delays of reinforcement were imposed throughout training (Figure 6.15).
Brain Mechanisms of Delay Discounting
Suppose an animal has a choice between options paired with a small and a large reward delivered immediately upon responding. In most cases, the animal would choose the larger reward. Now suppose the small reward is delivered immediately, but the large reward is delivered after some temporal delay. Such choices highlight a behavioral dimension that goes from impulsivity (choice of the small/immediate reward) to self-control (choice of the large/ delayed reward). In such situations, choice depends on such factors as the size of the large reward, the length of the temporal delay, prior experience, and individual variations that promote either impulsivity or self-control. Impulsive decisions suggest that the value of the large reward is reduced by the temporal delay, a phenomenon called delay discounting (Van-derveldt, Oliveira, & Green, 2016). Devaluing a reward that will be delivered in the future makes some sense because delays introduce uncertainty, particularly in species with short life span or high metabolic rate, with food resources that decompose rapidly, or when the organism is starving. Delay discounting has been described in mammals (rats, primates) and birds (pigeons), but the degree to which delays are tolerated vary across species. One factor is body size: larger animals (e.g., apes) tolerate longer delays better than smaller animals (e.g., marmosets, tamarins, rats, or pigeons) (Stevens & Mühlhoff, 2012). Of course, body size is correlated also with longevity and metabolic rate, among other factors.
In mammals, the circuit activated by delay-discounting tasks is distributed, involving the amygdala, hippocampus, and several cortical areas. Lesions in any of these areas are likely to lead to impulsive choice in rats. These areas have extensive communication with a portion of the ventral striatum, the nucleus accumbens. Rats with damage to the nucleus accumbens can discriminate between small and large rewards, and choose the large one when there are no delays involved. However, when the large reward is delayed, they consistently choose the small-reward option, showing impulsivity (Cardinal, Pennicott, Lakmali, Robbins, & Everitt, 2001). Cortical damage, especially in prefrontal areas, also affects delay discounting. The prefrontal cortex is important in planning, expectancy, and in behaviors that depend on future outcomes. One of its areas, the orbital prefrontal cortex, plays a complex role in delay discounting. In rats that are prone to impulsivity, inactivation of orbital neurons enhances impulsive choice if the delays are clearly marked by a stimulus, but self-control if the delays are not signaled (Zeeb, Floresco, & Winstanley, 2010).
