The hummingbird flits to its favorite garden, drawing nectar from flowers, knowing it can return for more. Every now and then, however, a trusted source dries up. What to do? In uncertainty, the bird might keep up the habit, hoping the flowers will again provide. Or it might search for a garden farther afield.
"The real world is often changeable, sometimes cruel," says Allen Neuringer, PhD, a Reed College psychologist who studies animal behavior. "The question is, what happens when what you've been reinforced for doing no longer produces the reinforcement?"
Across decades of research on reinforcement and extinction--the period after rewards are withdrawn--two apparently inconsistent answers to that question have emerged, Neuringer says.
Research in the operant-conditioning tradition has revealed that withholding reinforcement slows the rate of learned responses and also causes animals' behavior to become more unpredictable and variable. Animals test new behaviors and vary the sequence, speed and forcefulness with which they perform the old behaviors.
Other research, in the Pavlovian and operant-conditioning traditions, however, has shown that the structure of responses remains stable, with the behaviors that are most frequent during reinforcement remaining dominant during the extinction period. Even long after reinforcement ends, studies show, animals often show a sudden resurgence of learned behaviors, demonstrating that such responses have not died out entirely.
"The very basic question is, what's going on here?" says Neuringer. "How could both of these occur?"
In the January issue of APA's Journal of Experimental Psychology: Animal Behavior Processes (Vol. 27, No. 1), Neuringer and two students report on experiments with rats that, they believe, reconcile the two findings for the first time.
In their work, the researchers found that when an animal is no longer reinforced for a behavior, it will remain faithful to what rewarded it before but will also experiment with novel strategies. Neuringer's new findings show that variability can arise both when it is reinforced and when that reinforcement is curtailed, says Peter Balsam, PhD, a psychologist at Barnard College of Columbia University, who also studies variability.
"These results show that if you want to teach individuals new, creative solutions," Balsam says, "you can directly reward them, or you can teach them to behave in a specific way, then arrange things so that solution no longer works, increasing creativity. Both of these principles could be harnessed in educational and therapy settings."
Reinforcement and extinction
In their first experiment, Neuringer and his colleagues reinforced rats for pressing combinations of levers and keys in a variable pattern. Then, they observed how removing that reinforcement affected the animals' subsequent behavior.
After 21.5 hours without food, rats were placed in a small chamber with two response levers and a food tray on one side and three response keys on the other. To get food, the rats had to press the two levers and the middle response key in sequences of three--for example, left lever, key, right lever. The animals could perform any combination of three responses, but received food only for sequences that had occurred very infrequently. Each time the rats met this criterion during a 30-minute session, they received a 45-milligram food pellet.
After 15 of these reinforcement sessions, the researchers next extinguished the learned response. For four sessions, the rats received no food, regardless of their actions. The researchers then compared the animals' behavior in the last four reinforcement sessions with behavior in the four extinction sessions.
They found that although the rats responded more slowly during extinction than during reinforcement, the structure of responses that was learned during reinforcement was maintained during extinction. That is, the rank ordering of most frequent and least frequent response sequences remained steady, with the response sequences that had been most dominant during reinforcement continuing to be so during extinction.
Neuringer and his colleagues also uncovered another pattern, which they believe helps explain the earlier findings of both stability and variability during extinction.
During extinction, the investigators found, those response sequences that had been least dominant during reinforcement became increasingly likely. Although the increase in these least probable response sequences was not large enough to change the rank order of response frequencies, they did significantly boost the overall variability of the rats' behavior.
"These results show that when things go badly, as is the case during extinction, animals are pretty sophisticated in their approach to figuring out how to get reinforced again," concludes co-author Nate Kornell, now a graduate student at Columbia University. "They continue to do what works, but they also try new things."
In a second and third experiment, Neuringer and his colleagues extended their results, finding the same pattern of both stability and variability during extinction for rats that had been reinforced randomly or for repetitive, invariant responses.
In addition, in the final experiment, the rats demonstrated further evidence of novel responding. Not only did they use the familiar response levers and center response key, but they also tried to win food by pressing the two other keys--keys that had never produced rewards during the reinforcement periods.
Jennifer McComas, PhD, an applied behavior analyst at the University of Minnesota, remarks, "One myth about behaviorism is that it doesn't promote individuality and creativity. This is a solid demonstration to the contrary."
The authors speculate that the particular mix of stable and variable behavior that animals exhibit may vary, depending on species, personality, motivation and situational demands. For example, individuals who are shy may be less likely to demonstrate variability than would bold individuals. Likewise, contexts in which the rules of reinforcement often change may produce more variable behavior than would contexts in which such rules are more constant.
University of North Texas behavior analyst Sigrid Glenn, PhD, commends the research on theoretical grounds. Most interesting to her, Glenn says, is Neuringer's examination of how behavior is shaped at the transitions between environmental states such as reinforcement and extinction.
"Much of behavior analytic work focuses on steady states, and the transitions from one state to another haven't historically been studied as much," Glenn explains. "Neuringer's work and other work on variability addresses the idea that variability is good--that it's not something to be gotten rid of. This approach allows us to see a parallel between operant, behavioral selection processes and natural selection processes at the level of species, because without variability, there can be no change."
Neuringer believes the new findings are likely to extend to other animals, including humans.
"Our previous research on learned variability has proven to play out similarly in a variety of species, from rats, pigeons and Siamese fighting fish to rhesus macaques and humans," he says. "We've not yet studied extinction across these species, but I believe there is likely to be wide generality of this effect as well."
If so, the effect may help address human problems, says Minnesota's McComas. For example, she explains, it's common for behavior therapists to address children's destructive behaviors through extinction, withholding reinforcement for such behaviors.
Neuringer's results point out a way that therapists may be able to capitalize on the variability that emerges during the extinction process, reinforcing appropriate behaviors as they occur.
This article is part of the Monitor's "Science Watch" series, which reports news from APA's journals.