Schedules of Reinforcement

Behavior

The relationship between behavior and reinforcement is described by schedules of reinforcement, studied in detail by B. F. Skinner (1904-1990) (Ferster & Skinner, 1957; Skinner, 1938). Any situation involving reinforcer presentation or omission contingent on some behavior can be characterized in terms of some schedule. In the Skinner box situation, an animal (typically a rat or pigeon) is given access to a manipulandum (a lever or a localized key light) that, if operated behaviorally (lever pressing or key pecking), leads to the delivery of the reinforcer (pellets or grain). In free-operant training, the animal can emit the instrumental response at any moment during the session. In discrete-trial training, response contingencies are separated by intertrial intervals. Many of the experiments reviewed previously involved discrete-trial procedures.

Reinforcement can be delivered according to a variety of schedules. The two most basic are continuous reinforcement (every response is reinforced) and extinction (every response is nonreinforced). Between these two extremes lie a variety of schedules in which not every response results in reinforcement—partial reinforcement. There are two basic ways to schedule partial reinforcement, either as a function of responses emitted or as a function of time. Moreover, simple schedules can be combined to yield complex schedules of reinforcement. Box 6.4 describes common schedules and their behavioral effects.

Box 6.4 Some Examples of Schedules of Reinforcement

Schedules can be arranged to mimic natural situations. Time-based schedules are analogous to food resources that either take time to replenish after consumption (a flower’s nectar) or appear according to a certain cycle (prey active at night). Response-based schedules resemble a foraging situation in which an animal is collecting food in a patch. The higher the rate of responding (or searching in a patch), the higher the rate of reinforcement; but, of course, the rate of reinforcement also determines the rate of responding. The causal relationship between responses and reinforcers is reciprocal. Not surprisingly, the allocation of responses and the obtained reinforcements often match, so it is deviations from such matching that are interesting (Poling, Edwards, Weeden, & Foster, 2011).

Discrete-trial procedures break down the reciprocal causality between responding and reinforcement, making it possible to manipulate probability of reinforcement as an independent variable. In one experiment (Wilson, Oscar, & Bitterman, 1964), rhesus monkeys (Macaco mulatto) were trained in a Red/Blue discrimination in which one alternative yielded food on 70% of the trials, while the other did so on the remaining 30% (70:30 problem). Both types of trials were intermixed randomly, and errors led to a correction procedure that kept the probability of reinforcement for each alternative as scheduled. Under such conditions, animals exhibited maximization; that is, they chose the alternative paired with the largest reinforcement probability most of the time. Monkeys did show a tendency toward matching on 50:50 problems, when the alternatives were equally reinforced. Interestingly, this matching originated from a tendency for a choice on one trial to be influenced by the choice made on the preceding trial. Specifically, monkeys tended to avoid on a given trial the side rewarded on the preceding trial. Sequential effects suggest that matching can be a byproduct of local processes operating on a trial-by-trial basis (Shimp, 1982). Maximization and matching are illustrated in Figure 6.14.

Some reinforcement schedules have specific properties. The fixed-interval scallop involves a patterned distribution of responses that occurs after extensive training on FI schedules (Box 6.4). Animals typically show response suppression immediately after reinforcement— postreinforcement pause—followed by a transition to a higher response rate. The length of the postreinforcement pause tends to be directly proportional to the size of the temporal interval. Interval timing, the ability to estimate temporal intervals in the order of seconds to

Figure 6.14 Probability learning involves a choice between two options (e.g., Red vs. Blue stimuli), each with its own percentage of reward. For example, 70:30 means that the reward occurs in 70% of Red trials, but on 30% of Blue trials. In 70:30 or 30:70 problems, animals tend to choose most of the time the option most frequently rewarded—maximization. In 50:50 problems animals tend to choose each option in about half of the trials—matching. The horizontal dashed lines mark the probability associated with one of the stimuli.

hours, plays an important role in the development of the FI scallop. In a FI schedule, the highest degree of temporal control is represented by responding immediately before the interval is over, whereas the poorest is represented by responding at a constant rate. Comparative research shows a higher degree of temporal control in rats, woodmice, cats, and monkeys (mammals) than in pigeons, doves, turtles, and tilapia fish (nonmammalian species) (Lejeune & Wearden, 1991).

Brain Mechanisms of Interval Timing

Interval timing, as studied in FI schedules, is a fundamental process underlying behavior in diverse functional areas, including learning, foraging, and decision making. Time estimation occurs in the brain. There is no dedicated sensory system devoted to time, so brain activity has to create the conditions for assessing durations on the basis of a variety of sensory inputs combined with internal states. As a result, a series of distributed circuits contribute to this vital function. Extensive training in a FI task develops a timing estimate of the moment in which the next reward should become available. For example, training in an FI 60-s schedule provides an estimate of when the next reward will become available relative to the previous reward. Responding increases as the interval approaches the target time, which implies that this is not a perfect estimate. The error (e.g., the standard deviation of response frequencies) is proportional to the size of the interval, such that estimating a 60-s interval involves less error than estimating a 120-s interval. Interestingly, the proportion of error size to interval length (error/interval duration) is constant for a wide range of values. Such constancy is referred to as the scalar property of interval timing. So, if the standard deviation for estimating a 60-s interval were 15 s, then the fraction would be 15/60 = 0.25. One would expect that estimating a 120-s interval would involve a standard deviation of about 30 because 30/120 also equals 0.25 (Gibbon, 1977). The scalar property applies to behavior as well as to neural activity.

Using the fMRI technique mentioned previously, human participants were asked to estimate the duration of two intervals: Ils and 17 s. When the hemodynamic changes detected by the fMRI technique as an index of brain activity were represented in terms of percentage change relative to the peak activation, the scalar property was observed in the putamen, a part of the dorsal striatum (see Chapter 5). Thus, putamen activation had a constant error/interval duration for Ils and 17 s (Hinton, 2003). Studies show that the striatum is critical for interval timing, being engaged early in timing tasks. However, other brain regions are recruited as time estimation develops with practice, including the motor and parietal cortices, and later on, when movement is required, the cerebellum (Merchant, Harrington, & Meek, 2013). A similar brain circuit is activated during interval timing tasks in other mammals. The putamen and the caudate nucleus, both components of the dorsal striatum, receive dopaminergic axons from the ventral tegmental area, an input that is critical for interval timing in rats (Meek, 2006). This pathway is also important for movement, as shown by dopaminergic deficits in the striatum of patients suffering from Parkinson’s or Huntington’s disease. Many such patients also exhibit impairments in interval timing tasks (Smith, Harper, Gittings, & Abernethy, 2007).