4.2.1. Brain History and Historiography

The field of brain evolution seeks to discern the history of the brain, but that field itself has a history, and not always an attractive one. Readers might wonder why the outline of brain evolution provided here is not widely known and why statements about brain evolution have an aura of controversy. In part, this feeling results from the bad reputation that brain evolution, as a field, has developed in certain circles. Nevertheless, progress has been made in understanding the history of brains, and we attempt to summarize some of that progress as it relates to reward processing.

Unfortunately, neuroscientists have often written about brain evolution in erroneous ways and in what follows we hope to avoid the mistakes of the past. In the mid-1980s, for example, one eminent neuroscientist (Jones 1985, 763) claimed that “no writer in the field today would seriously propose that the brains of existing nonmammalian vertebrates formed a scala naturae,” while another did exactly that by speculating about how a certain behavioral capacity applied “across the entire phyletic scale” (Mishkin, Malamut, and Bachevalier 1984, 73). Evolutionary biologists long ago rejected the concept of a phyletic scale, also called the scale of nature or the scala naturae, in favor of the science of cladistics, which uses the evolutionary innovations of particular lineages to discern “evolutionary trees” through the principle of parsimony. Instead of ranking species as a linear progression from “lower” to “higher” forms—e.g., rats-cats-monkeys-chimpanzees-humans in the mammalian clade—most neuroscientists now understand that phylogenetic relationships have a branching architecture (Figure 4.1), with any two existing lineages sharing one most recent common ancestor. No existing species represents that common ancestor perfectly, and most, like laboratory rats and rhesus monkeys, do so rather poorly. Rats, for example, are highly specialized “gnawers,” with a host of evolutionary innovations that the rodent-primate common ancestor lacked. Although we draw on data from rats and rhesus monkeys to explore what we have inherited from ancestral vertebrates, mammals, and primates, these groups do not make up a “phyletic scale” or anything of the sort. We discuss these groups in that order simply because we evolved from all of them and because they first appeared in that order.

We also discuss structures that evolved before or after others in evolutionary history, a topic that has many pitfalls. For example, one of the neuroscientists quoted above claimed that “the basal ganglia antedates both the cerebral cortex and the limbic system in phylogenesis” (Mishkin, Malamut, and Bachevalier 1984, 74). To the contrary, comparative neuroanatomy shows that homologues of the basal ganglia and many limbic structures, including parts of the cerebral cortex such as the hippocampus, evolved at about the same time, in relatively early vertebrates (Northcutt 1996; Striedter 2005).

Other discredited doctrines have also muddied the waters considerably. Some, such as MacLean’s triune brain theory, have spawned nonsense such as “reptilian brains” inside human heads (MacLean 1985). In this chapter, we assume that structures such as the basal ganglia perform some conserved function, not that humans or other mammals have the same basal ganglia that some ancestral species or group had. (Also, technically, we did not evolve from “reptiles” but instead descended from ancestral amniotes.) Another discredited doctrine invokes the notion of “evolutionary trends” and schemes of connectional and functional hierarchies based on these imaginary trends. Sanides (1970) imagined that a group of medial cortical areas “arose” from the hippocampus, whereas a group of lateral cortical areas arose from the piriform cortex. He claimed that he could spot “trends” in the cytoarchitecture of these two groups of areas from phylogenetically older to newer parts of the cortex. Unfortunately, comparative evidence shows that many of his claims are wrong, while others are plausible but wholly lacking in evidentiary support. For example, whereas Sanides imagined that primary sensory areas appeared in later stages of mammalian evolution, comparative evidence shows that they were among the earliest areas to appear in mammals (Kaas 1995, 2008). Although Sanides’s views were purely speculative, some neuroanatomists and others have adopted his opinions and interpreted their data according to those views (Barbas and Pandya 1989; Mesulam 1990; Cummings 1993). Nothing here should be construed as agreeing with any of that. When we talk about medial and lateral parts of the frontal cortex, we do not do so in the context of imaginary “evolutionary trends” of the sort popularized by Sanides and his followers.

To sketch the history of the brain, three concepts (homology, analogy, and homoplasy) and just five taxonomic terms (protostome, deuterostome, gnathostome, tetrapod, and amniote) clarify matters considerably. We explain the five taxonomic terms below, but take up the three evolutionary concepts here. Homology is a statement about ancestry, not function. A structure or behavior is homologous to another if two or more descendant species have inherited it from their most recent common ancestor. Analogy is a statement about function, not ancestry. A structure or behavior is analogous to another if it subserves the same function. The classic example involves wings. Insects, birds, and bats have wings, and they all perform the same function: producing lift for flight. We can be certain that they are only analogous, and not homologous, because the last common ancestor of each pair (birds and bats, birds and insects, bats and insects) lacked wings of any kind. Furthermore, insect wings evolved about 390 million years ago, hundreds of millions of years before the origin of either birds or bats (Grimaldi and Engel 2005), in a lineage that produced neither bats nor birds as descendants (Figure 4.1). A recent common ancestor of birds and insects is marked with an A in Figure 4.1, which also applies to bats and insects. This extinct species was a worm-like animal with no capacity for flight whatsoever. It had no wings or, indeed, appendages of any kind. Similarly, the last common ancestor of birds and bats is marked with a B. These ancient amniotes, which lived roughly 320 million years ago, also lacked wings. So like bird and insect wings, bird and bat wings are analogous, not homologous. This fact leads to the concept of homoplasy: something similar that has evolved in different lineages through parallel or convergent evolution. For example, the green lizard Lacerta adopts the habit of resting on green leaves. The color of lizard and leaf resemble each other, but this trait is neither analogous nor homologous. The trait (of being green) is not analogous because the color of the lizard performs a camouflage function, but the leaf has no interest in camouflage. The trait is not homologous because the lizard and plant did not inherit these characters from their nearest common ancestor, which was a single-cell eukaryote that lacked color in this sense. Instead, lizard and leaf exhibit homoplasy in being green. Some similarities stem from common developmental constraints and others result from the necessity of dealing with the physical and physiological realities of the world. For example, wings must produce lift from muscle power by interacting with air, so insect, bat, and bird wings (not to mention those of pterodactyls) inevitably shared certain features once they independently evolved.

Because of the prevalence of homoplasy, when we make statements such as “the amygdala and the dopamine system evolved in the early vertebrates,” these conclusions are based on the identification of homologues of those structures in a diverse set of vertebrates, according to the principles of cladistics. Such statements do not imply that vertebrates are the only animals with neurons analogous to those in the mammalian amygdala or that they are the only animals with dopaminergic neurons.

4.2.2. Vertebrate Brains

A key development in the history of animals occurred when vertebrates evolved from invertebrate ancestors (Figure 4.1). Note that the term “invertebrates” refers to an arbitrary grouping of animals rather than a clade (a progenitor species and all of its descendants, living and extinct). The animals that most people think of as “invertebrates” are the protostomes, a group that includes insects, mollusks (such as squid and octopus), and segmented worms. Additional kinds of “invertebrates” are found among the deuterostomes. These two lineages, protostomes and deuterostomes, diverged more than 600 million years ago, and their last common ancestor was a worm-like animal with a diffuse nerve net (marked by an A in Figure 4.1). Both groups formed their nervous systems from ancestral conditions characterized by such nerve nets, which during parallel evolution condensed into concentrated neural structures independently (Valentine 2004; Grimaldi and Engel 2005). Many protostomes evolved to a highly advanced level, producing intelligent animals with extraordinary sensory sensitivity, motor capacity, and social organization. Nevertheless, the brains in these animals evolved independently from those in vertebrates and are not homologous to vertebrate brains. (This conclusion is not without controversy, but the analysis of the options proposed by Schmidt-Rhaesa (2007) in the context of the recent advances in understanding animal phylogeny (Dunn et al. 2008) supports this view.) The brains of protostomes differ in so many crucial respects from vertebrate brains that it is not particularly useful to apply the same name to them, although it is probably unavoidable. Most important for the present purposes, protostome brains lack homologues of the midbrain dopamine system, amygdala, hippocampus, and striatum; they lack anything remotely resembling the cerebral cortex of mammals; and to consider any part of their brains homologous to the prefrontal cortex of primates is preposterous. The brains that protostome invertebrates use to deal with reward have a long evolutionary history, one entirely separate from ours. Their brains are not only unlike those of vertebrates, they have virtually no resemblance to our common ancestral condition either.

Instead, as shown in Figure 4.1, vertebrates and their brains evolved in the deuterostome lineage, long after diverging from the line leading to modern protostomes. With these animals came a complex central nervous system, an ability to swim and eat, and to direct those actions with a brain and sensory organs on and in the head (Northcutt 1996; Holland and Holland 1999; Butler 2000; Nieuwenhuys 2002). (Protostomes developed similar features, too, but did so independently.) For the present purposes, the most important evolutionary developments involved the invention of the telencephalon and the vertebrate dopamine system. The early telencephalon included the olfactory bulb and a homologue of the piriform cortex of mammals (Wicht and Northcutt 1992, 1998; Northcutt 1996, 2001; Striedter 2005). Crucially, the telencephalon of early vertebrates contained homologues of the basal ganglia and, with somewhat less certainty, the amygdala and hippocampus, to use their mammalian names. (The strongest evidence traces all of these structures back to the early tetrapods (see Figure 4.1), but suggestive anatomical evidence indicates that homologues exist in various extant fish groups that make these structures likely to be innovations of relatively early vertebrates, if not the earliest ones (Neary 1990; Ulinski 1990; Northcutt 1996, 2008; Smeets, Marín, and Ganzález 2000; Wullimann and Mueller 2002; Wullimann and Rink 2002; Medina et al. 2005; Striedter 2005; Moreno and Gonzalez 2007).) Vertebrates also innovated paired eyes as part of their central nervous system, which, along with their olfactory system, supplied their brains with information about distant features of their environment, and these organs make crucial contributions to reward-seeking behavior. Thus, roughly 530 million years ago, early vertebrates developed the brain, several remote sensors on the head, and a motor apparatus, and they used these newly evolved systems to guide decisions and goal-directed movements. These evolutionary developments changed forever what brains—our brains by descent—can do with reward.

4.2.3. Mammalian Brains

Mammals deal with reward with a mixture of old and new features, including those inherited from the early vertebrates: the dopamine system, amygdala, hippocampus, basal ganglia, olfactory system (with its olfactory bulb and piriform cortex), paired eyes, and so forth. In addition to these traits, shared with other vertebrates, the early mammals developed a new structure that deals with reward: the neocortex.

The top part of Figure 4.1 shows something about how this came about. Fairly early in vertebrate history, jawed fish (gnathostomes) evolved from jawless ancestors. Their descendants include all of the vertebrates in the bottom two parts of Figure 4.1, including the first land vertebrates, the ancestral tetrapods, which evolved 370 million years ago or so (Clack 2002). With amniotes, a group that appeared about 320 million years ago and includes all reptiles, birds, and mammals, the three-layered cortex known as the allocortex appeared. Parts of the allocortex have homologues in other vertebrates, but without its characteristic laminated structure (which was lost secondarily in birds).

Mammals evolved about 220 million years ago and have retained the allocortex as two substantial parts of their cerebral cortex, the piriform cortex and the hippocampus, along with some smaller, obscure allocortical areas. A different kind of cerebral cortex evolved in the earliest mammals or their immediate ancestors (Kaas 1995, 2008; Northcutt and Kaas 1995; Krubitzer and Kaas 2005). We can be certain of this conclusion because these areas occur only in mammals, and they can be found in all mammals, including monotremes and marsupials (see Figure 4.1). These mammalian innovations are often called the neocortex (meaning new cortex, which is true) or isocortex (meaning homogeneous cortex, which is false), but these terms are used inconsistently. Sometimes the terms neocortex and isocortex exclude mammalian innovations such as the so-called transition areas between the allocortex and the remainder of the neocortex. Here we use the term neocortex to include these transition areas. As noted above, rodent species, including rats, have their own set of adaptations and specializations, but the architecture of their frontal cortex represents the ancestral mammalian condition reasonably well. Rodents, like other mammals, possess several distinct neocortical fields: these include medial frontal areas, specifically the anterior cingulate (AC), infralimbic (IL), and prelimbic (PL) areas, as well as orbital frontal areas, specifically the agranular insular (Ia) and agranular orbitofrontal cortex. We use standard names and abbreviations for most of these areas, the ones noted in the previous sentence. We also use a new one, OFa, which stands for agranular orbitofrontal cortex. Some of these areas, specifically the medial frontal and orbitofrontal areas, go by the name “prefrontal cortex” in rodents and other nonprimate mammals (Kolb 2007), but their homologues in primates are not called prefrontal, so we use a different term. Regardless of their names, all of these specifically mammalian areas likely contribute to what mammalian brains can do with reward.

4.2.4. Primate Brains

Primates first appeared about 60 million years ago. Early in this long history, primate brains evolved a new kind of frontal cortex, as well as some additional sensory areas. Figure 4.2 shows the basic, architectonic types of cerebral cortex in two species of primate and a rodent (Carmichael and Price 1994; Öngür, Ferry, and Price 2003; Palomero-Gallagher and Zilles 2004). In the primate frontal lobe, the main types of cortex are the agranular cortex, which lacks an internal granular layer (layer 4), and areas that have either a conspicuous or subtle layer 4, collectively called the granular prefrontal cortex (PFg). The PFg occurs in all primates and only in primates, and therefore represents a primate innovation (Preuss 1995, 2007). In appreciation of this historical fact, we use another new abbreviation, PFg, for these areas collectively. Figure 4.2 illustrates the parsimonious idea that rodents and primates, among other mammals, share the agranular parts of the frontal cortex, but primates alone have the granular parts. This idea has been controversial, but cortical topology (the spatial relationships of the frontal areas to each other and to the allocortex) and corticostriatal connectivity (from the agranular frontal areas to and near the ventral striatum, in particular) point to the same conclusions (Wise 2008). Crucially, none of the arguments brought forward to dispute the idea that PFg is a primate innovation—dopamine inputs, inputs from the mediodorsal nucleus of the thalamus, behavioral effects of brain lesions—stand up to scrutiny (Wise 2008). The lack of granular prefrontal areas in rodents should not be surprising given the antiquity of the most recent common ancestor of primates and rodents (marked with a C in Figure 4.1). Some of the primate innovations, such as granular parts of the orbitofrontal cortex (PFo), surely influence what primate brains can do with reward.

FIGURE 4.2

Basic types of frontal cortex. The key indicates three basic types of cerebral cortex by grayscale code. (a) human, (b) macaque monkey, (c) rat. Architectonics from Öngür, Ferry, and Price (2003) (a), Carmichael and Price (1994) (b), and (more...)

4.3.1. What Nonvertebrate Brains Can Do with Reward

The vertebrate brain evolved a dramatically new central nervous system, one not shared by nonvertebrate species. Nevertheless, nonvertebrates deal with reward effectively. They have the ability to seek specific rewards (e.g., salt and water) and demonstrate the ability to learn about rewards via both associative mechanisms (e.g., Pavlovian and instrumental conditioning) and nonassociative ones (e.g., habituation and sensitization). As in vertebrates, associative learning about rewards in protostomes concerns how the animal learns to predict and respond to different events in their environment, and it involves the formation of associations between representations of different stimuli (S), responses (R), and rewards (often referred to as outcomes, or O), as formalized by the concepts of Pavlovian and instrumental conditioning. The stored knowledge acquired via these mechanisms guides reward-seeking behavior in protostomes as it does in vertebrates, and it is likely that almost all animals have the capacity for Pavlovian and instrumental conditioning. In Pavlovian conditioning, rewards are referred to as unconditioned stimuli (USs). Through experience (such as temporal contiguity), an originally neutral stimulus, called a conditioned stimulus (CS), becomes associated with a US and therefore comes to elicit the same innate responses (unconditioned responses, or URs) that are produced in response to the US. It is generally accepted that underlying Pavlovian conditioning is the formation of CS-US (stimulus-outcome or S-O) associations. Activation of the US representation via this association triggers the associated URs, which become known as conditioned responses (CRs) when evoked by presentations of the CS. In addition, Pavlovian conditioning may lead to the formation of direct CS-CR associations (also known as S-R associations) under certain circumstances (such as extinction, as discussed in Section 4.4.2.3). Learning about the predictive nature of environmental events allows the animal to act in anticipation of the US. When the US is the reward, this process increases the likelihood of obtaining it.

In instrumental conditioning, associations develop when an animal learns about the effect of its actions on the environment. Therefore, unlike Pavlovian conditioning, in instrumental conditioning the availability of reward depends on an animal’s actions: its actions are instrumental in producing the reward. This type of learning involves the formation of associations between the instrumental response and reward (R-O and O-R associations). Guidance of instrumental behavior by R-O associations is conceptualized as reflecting goal-directed responding. If responding is followed by a reward in the presence of a particular stimulus, this stimulus comes to modulate the R-O association and is termed the discriminative stimulus. Furthermore, stimulus-response (S-R) associations also form between representations of the response and any stimuli present when the reward is delivered. Behavior predominantly guided by S-R associations is referred to as habitual. Chapter 14 explains these concepts in more detail.

Neither Pavlovian nor instrumental learning requires a vertebrate brain. Marine mollusks such as Aplysia, for example, demonstrate the ability to learn via both of these mechanisms (Carew, Walters, and kandel 1981; Carew and Sahley 1986; Brembs and Heisenberg 2000; Brembs et al. 2002). The garden slug, a terrestrial mollusk, shows evidence for more complex Pavlovian phenomena such as second-order conditioning, in which previously rewarded stimuli reinforce a new response. They also show the phenomena of blocking and extinction (Sahley, Gelperin, and Rudy 1981; Carew and Sahley 1986), which are described in more detail below. And nematodes (roundworms) can learn to discriminate between good and toxic food (Zhang, Lu, and Bargmann 2005). Given that nonvertebrates can do all that with reward, what is so special about what vertebrates can do with rewards?

4.3.2. Midbrain Dopamine Nucleus

The evolution of the midbrain dopamine system must have had a profound impact on what vertebrate brains can do with reward. Their function has been thoroughly reviewed (Schultz 2006, 2007) and so only selected aspects are mentioned here. In general, a reward acts as a reinforcer to strengthen associations between representations that are active in its presence. One consequence of this property is that initially neutral stimuli take on reward value, leading to the reinforcing value of secondary rewards, as mentioned in the previous section. Schultz showed in monkeys that as a stimulus becomes associated with reward, dopamine neurons increase their discharge rates when that stimulus appears. This finding explains why conditioned reinforcers work: dopamine cells respond to CSs much like they do to unconditioned reinforcers such as primary reward. Schultz also showed that dopamine-cell activity reflects an error signal that conveys the difference between predicted and received reward. The sources of the prediction and error remain unknown, but one possibility is that striatal GABAergic neurons provide a reward-prediction signal to the dopamine cells and that other inputs to the dopamine cells, such as those from the amygdala, hypothalamus, and peripeduncular nucleus or bed nucleus of the stria terminalis, provide the received reward signal. In this view, the dopamine cells compute the difference between these two inputs and their output serves as a teaching signal to its targets, among which the striatum figures especially prominently. Receipt of reward at an unexpected time or in larger-than-expected amounts leads to strengthening of associations between representations active at the time of the teaching signal, which makes it more likely that the action made in that context will be repeated when the context recurs. Failure to receive a reward at the expected time and in the expected amount leads to a form of extinction, and puts the system in a state that promotes exploration of alternative actions. Comparison of predicted and received reward is the fundamental mechanism by which vertebrate brains deal with reward.

4.3.3. Basal Ganglia

One of the most vexing problems in understanding the general role of the midbrain dopamine system, especially in relation to basal ganglia function, is that one line of research has focused on its role in motor control, while another has focused on its role in reward-seeking behavior, as exemplified by Pavlovian approach behavior (S-O associations), Pavlovian-to-instrumental transfer (PIT), and instrumental conditioning. The first line of research has, after a period of uncertainty, confirmed earlier opinions (Northcutt 1996) that the largely diencephalic group of dopamine neurons found in jawless fish (lamprey) is homologous to the midbrain dopamine system in tetrapods (Smeets, Marín, and Ganzález 2000). Therefore, it now appears even more likely that the dopamine system evolved in the earliest vertebrates (Figure 4.1). The relatively new evidence is that the same toxin that produces Parkinson’s disease (PD) in people causes similar symptoms in these jawless fish (Thompson et al. 2008). An equally ancient structure, the lateral habenula, appears to provide a signal to dopamine neurons that inhibits these neurons when rewards are not predicted (Matsumoto and Hikosaka 2007). It appears, then, that the evolution of this dopamine system in early vertebrates had profound implications for what their brains could do with reward, but it remains difficult to reconcile the motor functions of dopamine-basal ganglia interactions with those involving primary and secondary rewards. One approach toward this reconciliation has been to consider several different time courses for the actions of dopamine, extending several orders of magnitude from fractions of a second, as described above, to many hours (Schultz 2007). Alternatively, the dopamine system may compute the cost–benefit analyses that are fundamental to all decisions.

Recent work has pointed to a common function for both motor and nonmotor parts of the basal ganglia. According to Shadmehr and Krakauer (2008), the interaction between midbrain dopamine cells and the basal ganglia modifies the assessment of costs and benefits in a very general sense, one not limited to reward as traditionally construed. They based their view on a computational model and on recent psychophysical evidence that PD, which is caused by degeneration of the midbrain dopaminergic neurons, leads to a form of effort intolerance (Mazzoni, Hristova, and Krakauer 2007; Shadmehr and Krakauer 2008). Apart from tremor, two principal symptoms of PD are bradykinesia (slow movement) and akinesia (no movement). Shadmehr and Krakhauer have related these symptoms to a change in how the brain computes the value of a certain amount of energy expenditure. In a psychophysical experiment (Mazzoni, Hristova, and Krakauer 2007), human subjects made a series of voluntary reaching movements at varying speeds and distances. The investigators measured, among other things, the number of reaches needed to produce 20 reaches that fell within a given velocity range, such as 37–57 cm/s. Healthy people show a reluctance to make relatively fast reaching movements, measured by their need to attempt ~35 trials needed to make 20 within the fastest movement range. PD patients show even more reluctance to make fast movements than do healthy people, requiring ~45 trials to make the same number of correct ones. Yet their reaching movements maintain normal spatial accuracy. Similar conclusions come from the work of Desmurget and Turner (2008), who inactivated the internal segment of the globus pallidus, the output structure of the “motor” basal ganglia, in rhesus monkeys. This inactivation causes abnormally short movements (hypometria), which are slower than normal (bradykinetic), but have the normal trajectory, endpoint accuracy, and reaction times. Like Shadmehr and Krakhauer, Desmurget and Turner concluded that the basal ganglia “may regulate energetic expenditures during movement.”

Thus, the dopamine system—through its interaction with the basal ganglia—appears to lead to the minimization of a cost: the increased energy expenditure (effort) needed to make faster movements. In PD, the decrease in dopamine input to the striatum alters the sensitivity to such effort costs. According to Mazzoni, Hristova, and Krakauer (2007, 7105), “bradykinesia represents an implicit decision not to move fast because of a shift in the cost/benefit ratio of the energy expenditure needed to move at normal speed,” which is an exaggerated form of a similar shift that occurs in healthy people. The basal ganglia and its dopamine inputs thus play a key role in computing the cost of reaching movements, and this function likely generalizes to volitional, skeletal movements of many kinds, including those involved in seeking and consuming primary rewards.

This kind of cost–benefit analysis could link the well-known motor functions of parts of the dorsal basal ganglia to the standard psychological framework implicating the ventral basal ganglia in decisions based on predicted reward outcome (Schultz 2006). This computational view of reward generalizes its definition to include benefits other than typical primary and secondary reinforcers, including benefits that come in the form of reduced energy costs. As Mazzoni, Hristova, and Krakauer (2007, 7105) put it, a role for dopamine in minimizing effort cost is “analogous to the established role of dopamine in explicit reward-seeking behavior.” Note, however, that when Mazzoni et al. use the term “explicit” in this way, they do not do so in the sense usually meant in psychology, which implies conscious awareness and the ability to report on a decision and its basis. Instead, they mean to distinguish two levels of implicit decisions: one that leads to behaviors that have produced benefits in terms of primary and secondary rewards, and another that produces benefits in terms of reduced effort costs. It appears that the basal ganglia and its dopamine inputs play a key role in computing the costs and benefits in a very general sense, for use in neural functions that minimize costs and maximize benefits of many kinds.

We can only speculate about the common mechanisms between these two kinds of “reward” computations. One possibility is that dopamine cells do for effort costs more or less what they do for primary rewards. When vertebrates make a movement that uses more than the optimal amount of energy, i.e., they expend more effort than necessary to reach a movement target, the mismatch between predicted and actual effort would cause the dopamine cells to decrease their activity, and the behavior would be “extinguished.” Conversely, if they reached the target with less effort than expected, the dopamine cells would increase their activity and this would teach the system to make the movement requiring this lesser effort in the future, thus optimizing the amount of energy expended to reach a movement target.

4.3.4. Amygdala

As they learn the biological value of rewards, vertebrates tend to select the behavior that yields the highest reward value. Not only can vertebrates learn the value of particular rewards, they can rapidly update the valuation of a reward based on their current state. The reinforcer devaluation procedure assesses this updating ability. When an animal has recently consumed one kind of food to satiety, that food becomes less valued, and animals will reduce a behavior that produced the devalued food. This happens even when their behavioral choices no longer yield the devalued reward (i.e., under extinction), which implies that they can recall the reward (outcome) value in its updated form.

In contrast to intact animals, rats or macaque monkeys with amygdala lesions fail to adjust their responses after reinforcer devaluation (Hatfield et al. 1996; Malkova, Gaffan, and Murray 1997; Balleine, Killcross, and Dickinson 2003; Izquierdo and Murray 2007). Although the experiments in monkeys involved the whole amygdala, work in rodents indicates that the basolateral amygdala (BLA), not the central nucleus, subserves this ability (Hatfield et al. 1996; Balleine, Killcross, and Dickinson 2003). Importantly, rats and monkeys with amygdala lesions have intact satiety mechanisms and intact sensorimotor discrimination abilities. The deficit in shifting choices after amygdala lesions thus results from an impairment either in updating food value or in linking stimuli and responses with this new value to guide behavior.

Wellman, Gale, and Malkova (2005), also studying macaque monkeys, disrupted reinforcer-devaluation effects by temporarily inactivating the BLA with the GABA_A agonist muscimol. They replicated the result described above for amygdala lesions, but only when the inactivation occurred during the selective satiation procedure, not when inactivation occurred afterwards. Similarly, Wang et al. (2005) demonstrated in rats that, for updating to occur, protein synthesis is needed in the BLA immediately following devaluation. These findings show that the amygdala updates reward value, but has little role in guiding choices after the value of the food has been updated.

In addition to devaluation experiments, amygdala lesions in rodents and monkeys impair behavior on a number of tasks that require associating a stimulus with value, including second-order conditioning, conditioned cue preference, conditioned reinforcement, delay discounting, and PIT, among others (Everitt et al. 2003; Murray 2007). For example, in delay discounting, a lever press produces a large reward at an increasingly longer delay, which devalues or discounts that reward. As the large reward is progressively delayed, rats gradually shift their choices to a lever that produces a smaller, more immediate reward. Rats with BLA lesions shift responses more quickly than intact rats, thereby exhibiting a steeper discounting function than intact rats (Winstanley et al. 2004). This relative intolerance for delay could result from difficulty in maintaining the representation of reward value in its absence (Winstanley et al. 2004), but the results of the next experiment suggest a different account.

Rudebeck and Murray (2008) found that monkeys with selective amygdala lesions use reward information better than controls in certain circumstances. Their monkeys chose between two stimuli: one choice produced positive feedback (reward) and the other produced negative feedback (effort cost with no benefit). Monkeys with amygdala lesions performed better than controls on the trial following a correctly performed trial, indicating they used reward feedback more efficiently than controls. Both groups used negative feedback equally well. As noted above, Winstanley et al. (2004) suggested the BLA lesions facilitated delay discounting because the rats could not represent the value of the large reward in its absence. The findings of Rudebeck and Murray (2008) suggest instead that the amygdala lesion disrupted an affective signal associated with the lever that produced large rewards. Indeed, the results of both Winstanley et al. (2004) and Rudebeck and Murray (2008), and perhaps others as well (Izquierdo and Murray 2005), could result from the lack of an affective signal that biases choices towards a previously rewarded stimulus. This account implies that the amygdala can impede reward-based decision making and, without its affective signals, animals can sometimes behave more “rationally,” as assessed in terms of costs and benefits.

We focused here on the BLA, but other parts of the amygdala might produce different reward-related signals. The BLA appears to link stimuli with the specific sensory properties of reward, whereas the central nucleus of the amygdala links them to reward’s general affective properties (see Balleine and Killcross 2006, for detailed evidence). As Balleine and Killcross noted, these two roles could contribute to different aspects of foraging, with general affect promoting the preparatory and approach phase, as in arousal, and signals related to specific rewards promoting their consumption.

In sum, the amygdala mediates several mechanisms linking stimuli and responses with the affective properties of rewards, and these mechanisms likely evolved in early vertebrates.

4.3.5. Hippocampus

There is ample evidence for a role of the hippocampus in spatial navigation and spatial memory in many vertebrate species, including animals in groups as diverse as reptiles, ray-finned fishes, birds, rodents, and primates. The activity of hippocampal pyramidal neurons in rats reflects spatially discrete locations (place fields) and collectively produces a map-like representation of the environment (O’Keefe and Nadel 1978). A growing body of evidence indicates that septal and temporal sectors of the hippocampus, often referred to in rats as the dorsal and ventral hippocampus, respectively, have dissociable functions. The septal region is essential for accurate spatial navigation and the temporal region subserves appropriate behavior in anxiety-inducing environments, as well as the contextual retrieval of internal cues based on motivational state (Kjelstrup et al. 2002; Bannerman et al. 2003). Anatomical relations of the septal and temporal portions of the hippocampus mirror their functional specializations. The temporal hippocampus, unlike the septal hippocampus, has extensive, reciprocal connections with the amygdala and hypothalamus, among other regions (Petrovich, Canteras, and Swanson 2001).

A recent finding places the septal-temporal dissociation of hippocampal function in a new perspective. Place fields within the hippocampus vary in size, with smaller place fields (<1 m) found septally and larger place fields (~10 m) located temporally (Kjelstrup et al. 2008). The large place fields in the temporal hippocampus may aid in identifying and representing familiar spatial contexts, as a form of generalization, and in identifying novel contexts. This ability to identify novel contexts could underlie both the reduction in exploratory behavior that occurs in anxiety-inducing environments and the reduction in eating that occurs in novel environments. These effects have been shown to depend on the temporal hippocampus (Kjelstrup et al. 2002; Bannerman et al. 2003; McHugh et al. 2004; Trivedi and Coover 2004) and likely reflect its interactions with the amygdala as well (Petrovich, Canteras, and Swanson 2001). Furthermore, the identification of familiar contexts is necessary for the contextual retrieval of cues based on internal signals regarding motivational state (Stouffer and White 2007). Thus, the functions of the temporal hippocampus may be an emergent property of its large place fields and specialized anatomical connections.

Table 4.1 summarizes the main points in Section 4.3, which outlines in broad terms what vertebrate brains can do with reward. Note that although Table 4.1 summarizes structure-function relationships in vertebrates, generally, it relies mainly on findings from mammals. Section 4.4 takes up the brain augmentations underlying what mammals can do with reward that their nonmammalian ancestors could not.

TABLE 4.1

Vertebrates.

4.4.1. Mammal Brains: Basic Vertebrate Mechanisms Plus Neocortex

As mammals evolved from our amniote ancestors (Figure 4.1), we came equipped with a brain that had come to include some new structures, compared to those found in ancestral, nonmammalian species. As outlined in Section 4.2.3, the frontal, parietal, occipital, and temporal areas that make up the neocortex changed forever how our brains could deal with rewards. This fact does not imply, however, that the systems that had developed earlier in vertebrate evolution became redundant; reward processing in mammalian brains still relies heavily on the midbrain dopamine system, basal ganglia, amygdala, and hippocampus. The phylogenetically older systems not only continue to operate alongside the mammalian innovations, but interact with them in a manner that transforms their function to an extent. Here we consider the adaptive advantages that the neocortex provides for reward-seeking behavior, in the context of cost-benefit analysis and foraging.

As discussed earlier in this chapter (and throughout other chapters in “Part III” of this volume), the central nervous system provides the means for an animal to learn about the relationships between its own actions, different aspects of its environment (including discrete stimuli and contexts), and the occurrence of reward, along with costs. The formation of internal representations of these stimuli, contexts, responses, and outcomes, as well as the association of these representations with each other, underpins learning. Although nonmammalian vertebrate brains form complex representations too, the evolution of neocortex in mammals enabled the formation of richer, more versatile internal representations of sensory inputs to associate with reward, including the sensory aspects of reward, its motivational aspects, and its hedonic properties. For example, sensory areas of the neocortex improve, relative to nonmammalian ancestors, the processing of the sensory aspects of rewards and stimuli associated with rewards. As we discuss in more detail in Section 4.5.2, sensory neocortical areas such as the perirhinal cortex, a feature of all mammalian brains, process sensory information in advanced ways. Accordingly, the mammalian brain does what the brain of its nonmammalian ancestors did with reward but does it better in certain ways.

Although reward processing involves all of the neocortex directly or indirectly, the agranular frontal areas seem to play a particularly large and specific role. Accordingly, this section focuses on these frontal areas. As indicated earlier, rats and other modern rodents do not replicate the traits of early mammals. They, like the other mammalian lineages, have many specializations that emerged over tens of millions of years of independent evolution. However, the organization of the agranular frontal areas in rats appears to be reasonably representative of the ancestral mammalian condition, notwithstanding the many anatomical and behavioral specializations of rats. The long, separate evolutionary history of the agranular frontal cortex in each mammalian lineage makes it likely that differences in both structure and function occur in each order and species of mammal. Nevertheless, we know of no reason to think that the fundamental functions of the agranular frontal areas differ among the mammals.

Learning, along with innate mechanisms, drives reward-seeking behavior in all vertebrates and, if left unsupervised, the most dominant associations ultimately control behavior. The agranular frontal cortex changes this state of affairs. We propose that among the most important innovations of early mammals was an improved “executive function,” mediated by the agranular frontal cortex, including a top-down modulatory function that biases competition among different brain systems engaged in and competing for the control of behavior.

4.4.2. Agranular Frontal Cortex

The mammal innovations include the regions of the agranular frontal cortex called the anterior cingulate (AC), PL, IL, agranular orbital frontal (OFa), and agranular insular (Ia) cortex, which all mammals share (see Section 4.2.3). A growing body of evidence implicates these different parts of the frontal cortex in processes that, in the broadest sense, afford mammals a greater flexibility in their reward-seeking behavior and ultimately bestow upon them the ability to pick and choose between different behavioral options, selecting the one best suited for the exigencies of the moment. These areas have some unique inputs compared to other parts of the mammalian neocortex. The Ia cortex receives relatively direct inputs from parts of the gustatory cortex, located immediately caudal to it, as well as olfactory information from the piriform cortex and visceral related signals from the brainstem and thalamus (Ray and Price 1992). In addition, some “exotic” somatosensory inputs arrive there, leading to the view that it functions in interoception, including sensations conveying pain, itch, temperature, metabolic state (e.g., hypoxia or hypoglycemia), and inputs from the lungs, heart, baroreceptors, and digestive tract (Yaxley, Rolls, and Sienkiewicz 1990; Ray and Price 1992; Zhang, Dougherty, and Oppenheimer 1998; King et al. 1999; Craig 2002; Saper 2002; Drewes et al. 2006). In addition to these inputs, the agranular frontal cortex has reciprocal connections with the amygdala, hippocampus, and midbrain dopamine system, and projects to the basal ganglia. Through these and other pathways, the mammalian frontal cortex interacts with these phylogenetically older systems to promote flexibility in their function (see Sections 4.3.2–4.3.5). (See also Chapter 11 for an in-depth discussion of corticobasal ganglia circuitry and interactions.)

Aside from nonassociative processes such as sensitization and habituation, associative learning mechanisms account for much of a mammal’s ability to obtain rewards. Through Pavlovian and instrumental conditioning that occurs during reward-related episodes, an associative network builds up between internal representations of rewards, actions, and stimuli; each representation has the potential to enter into associations with other representations. This vast associative structure involves several “memory systems,” so called because different parts of the brain encode, represent, and store different kinds of information, which animals use to optimize their performance in terms of benefits and cost. Together with innate mechanisms, the associations mediated by these “memory systems” guide reward-seeking behavior, but do so in different ways. With no top-down modulation, the most dominant of these systems—perhaps based on the strength of its associations—will control behavior.

The behavior of nonmammalian vertebrates gives the impression of operating under the control of a dominant response system of the sort just described, sometimes called a prepotent response. Birds and reptiles often appear to have problems overriding their innate behavioral responses even when this prepotent response increases costs and reduces benefits. These relatively inflexible forms of behavior provide an advantage under many circumstances, especially routine ones. However, problems arise on the less frequent occasions when another behavioral choice would yield greater benefits or reduced costs. We present two examples of such inflexible behavior in nonmammals: one from a reptile, the other from a bird. Note from Figure 4.1 that both kinds of animals are crown species. Neither birds nor modern reptiles represent the ancestral vertebrate condition very well, but they do carry forward many traits from their most recent common ancestor and behavioral control by a dominant brain system may be one of them.

For instance, the green lizard Lacerta has an innate tendency to approach the color green, an instinctual response that presumably evolved to guide them towards leaves. Leaves are a good place for Lacerta to spend their time because they provide camouflage and house the insects and grubs that compose their diet. Wagner (1932) showed that it was almost impossible to train these lizards to override their innate response and reject green, even when in doing so they failed to obtain a palatable food reward. Lizards were trained on a task that required the choice of one of two panels: a red one with a palatable mealworm attached versus a green one with an unpalatable, saline-adulterated worm attached. The lizards consistently chose the green panel despite the cost of doing so: the loss of a tasty worm. Wagner (1932) reported that it took hundreds of trials for these lizards to override their innate response, if they could do so at all. Sensory and motor factors could not account for this result, as the lizards can easily distinguish the green and red panels and make the required choice. Instead, it appears that the prepotent response made it difficult or impossible for the brains of these lizards to use reward in a way that any mammal could easily do, at least any mammal that could see and discriminate red from green.

A frequently recounted study by Hershberger (1986) provides another clear demonstration of severe difficulty in using reward experience to override innate, prepotent responses, in this case from birds. Chicks (genus Gallus) were fed from a bowl with distinct visual characteristics and would subsequently approach the same food bowl if released at a distance from it. This consistent behavior shows that the bowl had acquired the properties of a Pavlovian CS, based on an association between the bowl and the reward outcome, which elicits an instinctive approach response. The chicks were then placed in a “looking-glass” scenario in which the bowl receded twice as fast as they ran towards it and approached them twice as fast as they ran away from it. They therefore had to inhibit their approach behavior in order to obtain the food. Even after a hundred minutes, the chicks continued to run toward the bowl. Apparently, they could not inhibit their conditioned approach response and alter their behavior, despite the fact that their actions cost them the reward every time. In short, the causal consequences of their behavior (i.e., the instrumental contingency) did not affect their behavior, either because they could not learn the experimentally contrived contingency or because they could not make use of it in the face of competition from the dominant response system, in this case one mediating an innate approach response.

Mammals, on the other hand, modify their reward-related behavior flexibly, adapting quickly to their changing needs and environment. This is not to say that instinct controls nonmammalian behavior but has no such control for mammalian behavior or that nonmammalian behavior lacks any flexibility. No such simplistic formulation could ever stand up to scrutiny. Rather, we suggest that mammals have brain areas—the agranular frontal areas—dedicated to modulating the balance among several behavioral-control systems. Nonmammals, lacking these areas, have more trouble doing that.

In rats, for example, the matching-to-position (MTP) task assesses the ability to override innate response tendencies. The MTP task pits a rat’s innate foraging tendency against a newly learned behavior. In order to maximize the likelihood of finding food when foraging, rats have a natural tendency to explore locations that they have not visited recently (if at all) rather than places recently explored and exploited. This foraging strategy means that when given the choice between two arms of a maze, each of which might contain food, rats usually choose the arm visited least recently. The MTP task requires rats to override this innate tendency and learn to select the most recently visited arm in order to obtain a food reward. Normal rats can learn this task easily, successfully overriding their propensity to spontaneously alternate, but rats with prior removals of the PL/IL region of the agranular frontal cortex cannot do so (Dias and Aggleton 2000). The same lesion has no effect on the acquisition of the nonmatching-to-position task, which requires behavior consistent with the rat’s innate tendencies.

The MTP task highlights the advantage of overriding innate responses, at least in a psychology laboratory. The ability to override this tendency must also have some value outside the laboratory. The tendency to vary foraging locations increases a rat’s likelihood of finding fluid or nutrients, as it would have for ancestral mammals. However, when a single location provides a large benefit at low costs, the ability to suppress the prepotent foraging strategy (“look somewhere else”) will likely reduce costs and increase benefits.

Sections 4.4.2.1–4.4.2.5 summarize some of the key research on subdivisions of the rat frontal cortex and their role in the flexible control of reward-seeking behavior, stressing two main themes: top-down biasing of behavioral control systems and flexible alterations of foraging strategies (Table 4.2).

TABLE 4.2

Mammals and Primates.

4.5.1. Primate Brains: Basic Mammalian Mechanisms Plus Granular Prefrontal Cortex

In Section 4.4, we noted that as mammals emerged from our amniote ancestors (Figure 4.1), the systems that had developed earlier in vertebrate evolution remained in operation, albeit with modifications. The midbrain dopamine system, basal ganglia, amygdala, and hippocampus continued to do more or less what they had done for hundreds of millions of years. Likewise for the emergence of primates, the ancestral systems, including the agranular frontal cortex, continued to play a central role in what primate brains can do with reward.

As explained in Section 4.2.4, primates and nonprimate mammals share the agranular frontal areas, but do not share a PFg (Figure 4.2), which evolved in primates. Among the prefrontal areas, the granular parts of the orbitofrontal cortex (PFo) play a large role in what primate brains can do with reward. These areas include area 11 as well as the light gray parts of areas 13 and 14 in Figure 4.2. In addition, primates also innovated certain sensory areas. Given the importance of vision in primates, the inferotemporal cortex (IT) is notable among these primate innovations. (Note that certain carnivores, such as domestic cats, also evolved high-order visual areas, but these likely arose independently, through parallel evolution (Preuss 2007).) Their long, separate evolutionary history means that the shared, agranular frontal areas will also have diverged in both structure and function in primates versus nonprimates.

Assuming that the emergence of the granular prefrontal areas provided primates with an advantage over the ancestral condition, what could that advantage be? Most answers to this question fall into one of two categories: (1) the new cortical areas functioned much like older parts, but performed those functions better in some way; and (2) the new areas permitted a novel function. However, there is no need to choose between the two and the emergence of a PFg probably led to both kinds of advances.

4.5.2. New Visual Areas in the Context of Feature Conjunctions

Cost-benefit analysis improves when the stimuli that predict rewards and costs do so with more fidelity, and also when those stimuli can be combined or categorized in useful ways. Some of the mechanisms in the PFg, generally, and the granular orbitofrontal cortex (PFo), in particular, seem to support such an advance.

The PFo of primates is in a pivotal position to permit novel kinds of S-O associations, if only because it receives information from visual areas, such as the IT, that other mammals lack (Figure 4.3). The PFo receives strong projections from the IT and other posterior sensory areas of cortex, including the auditory cortex. It also receives inputs from the agranular frontal areas, in particular areas specialized for gustatory, olfactory, and visceral inputs (Rolls, Yaxley, and Sienkiewicz 1990; Webster, Bachevalier, and Ungerleider 1994; Carmichael and Price 1995; Cavada et al. 2000; Kondo, Saleem, and Price 2005). By virtue of its receipt of all these kinds of information, PFo is one of the earliest sites for convergence of visual information with gustatory, olfactory, and visceral inputs. Furthermore, PFo receives inputs from the perioral and tongue representations of the primary somatosensory cortex, as well as strong projections from the perirhinal cortex, a multimodal area that has been shown to operate as an extension of the ventral visual stream or object-processing pathway ( Murray, Bussey, and Saksida 2007). Consistent with the anatomical evidence, individual neurons in the PFo of macaque monkeys respond to gustatory, olfactory, and visual stimuli presented separately or together (Rolls and Baylis 1994; Pritchard et al. 2008).

FIGURE 4.3

Convergent contributions to PFo function. PFo is depicted as receiving three inputs that influence what it can do with rewards: representations of visual feature conjunctions for objects and rewards; updated valuations of rewards; and conjunctions of (more...)

These findings suggest that primates may have a greater capacity to link visceral, olfactory, and gustatory inputs with high- and intermediate-order visual stimuli than our nonprimate ancestors. This new hypothesis originates, in part, from recent research on visual processing mechanisms in macaque monkeys. This work indicates that the perirhinal cortex represents complex conjunctions of visual features that promote object identification (Bussey and Saksida 2007; Murray, Bussey, and Saksida 2007). Other sensory modalities are also important to perirhinal cortex function, but given the predominance of vision in primates, we will focus on the visual aspects of scenes and objects. Bussey, Saksida, and Murray (2002, 2003) developed and tested this idea with a series of visual discrimination tasks in which combinations of features, rather than any individual feature, best predicted reward. In the context of discrimination learning, this objective can be attained by having individual features appear as part of both the correct (S+) and incorrect (S-) objects in a pair. In one experiment, for example, compound objects were comprised of two of four possible features: A, B, C, and D. Monkeys received rewards when they selected the compound stimuli AB and CD, but not when they selected stimuli CB and AD. This experimental design created a conflict, called feature ambiguity, because individual features were associated equally with both reward and non-reward. Accordingly, to obtain a reward, the monkeys were required to represent the combinations of features such as AB (depicted as VisAB in Figure 4.3, with the individual features depicted as VisA and VisB, respectively). Bussey, Saksida, and Murray (2002, 2003) showed that lesions of the perirhinal cortex disrupted performance in several tests based on this premise. Their results showed that the representation of feature conjunctions in the perirhinal cortex helped resolve feature ambiguity and promote object identification.

Homologues of the primate perirhinal cortex, along with lower-order visual areas such as the striate and near-striate cortex, can be found in rats and other mammals, but the evolution of additional areas such as the IT occurred in primates. What new capabilities could these new areas provide? Whereas perirhinal cortex is thought to represent the most complex (highest-order) visual features that compose objects, the IT cortex represents intermediate levels of feature conjunction. According to this view, the striate and near-striate cortex represent lower-order conjunctions, compared to the IT cortex, and the perirhinal cortex represents higher-order ones. Thus, it appears that during primate evolution, newly evolved areas emerged between the two extremes of conjunctive representations: mid-level conjunctions. Figure 4.3 depicts mid-level conjunctions as VisAB and VisCD, whereas high-level conjunctions are denoted as VisABCD. The existence of mid-level feature conjunctions changes what primate brains can do with reward, specifically by permitting categorization-based conjunctions that fall short of whole objects, but have greater complexity than their fundamental features.

4.5.3. Orbital Prefrontal Cortex

As illustrated in Figure 4.3, the PFo appears to be the earliest site at which the high- and mid-level feature conjunctions of an object (from perirhinal and IT cortex, respectively) converge with olfactory, gustatory, and visceral inputs (relayed via the agranular orbitofrontal areas). This anatomical convergence enables feature conjunctions of a particularly important kind. Comparison to mammals that lack a PFo, which is to say all nonprimate mammals, indicates that the PFo may provide the neural substrate for finer distinctions among food and other rewards based on a combination of their visual attributes with their chemosensory and visceral sensory properties (collectively depicted as “chemovisceral” feature conjunctions in Figure 4.3). In addition to representing finer distinctions among reward items, the PFo could also represent more generalized categories of foods. We propose that these visual-chemovisceral feature conjunctions promote an enhanced flexibility in responding to rewards. The sight of a particular food, a fruit at a particular stage of ripeness for example, might evoke its taste and smell, along with the pleasant sensations and positive affect that follow its consumption. Similarly, the fruit’s odor might evoke the whole, cross-modal configuration of features as well. This idea is much like that put forward in Section 4.4.2.4 for the OFa, but by making use of the richer analytical capacities of IT, for example, the PFo would provide an advantage over animals that lacked these areas.

So far, we have stressed ways in which the PFo functions much like the agranular frontal areas of rodents (and, presumably, the agranular frontal areas of primates as well). What about new functions? Some evidence points to a role for the PFo in representing value in a “common currency,” one that is independent of the specific kind of reward (benefit) or a particular kind of cost. In many neurophysiological studies in monkeys, a particular visual stimulus has been associated with a specific reward through learning, primarily via Pavlovian (S-O) associations. At the time the monkey views the stimulus, the activity of many PFo neurons reflects the value of the associated reward, even though the reward itself arrives much later. This property is by no means unique to the PFo: in this regard it resembles phylogenetically older structures, such as the midbrain dopamine neurons and many other parts of the frontal cortex. Yet, the reward-expectancy signal that the PFo generates when elicited by the sight of a particular food might differ in a subtle but important way from similar signals in these other structures. Recent neurophysiological studies in monkeys have found that the activity of PFo neurons reflects the value of rewards independent of reward type (Wallis and Miller 2003; Padoa-Schioppa and Assad 2006). In addition, neurons in the PFo (along with neurons in other parts of the cortex) encode at least three factors that enter into the calculation of reward value: the magnitude of reward, the probability of reward, and the effort required to obtain it (Kennerley et al. 2009). Thus, it appears that the PFo’s outcome representations extend beyond the particular substance (sucrose, quinine, juice) paired with a stimulus to signal a value that transcends particular types or amounts of reward (Padoa-Schioppa and Assad 2006).

To date, no neurophysiological study of the rodent frontal cortex has shown this property, although some have shown that neurons in the agranular frontal areas have activity that reflects specific expected reward outcomes and specific S-O associations (Schoenbaum, Chiba, and Gallagher 1998; Schoenbaum and Eichenbaum 1995). One possibility, then, is that the multiple levels of sensory convergence available in the PFo might enable the computation of both the “menu-independent” value signal observed by Padoa-Schioppa and Assad (2008), which indicates outcome values regardless of the reward options, and a drive-independent value signal (the “common currency”), which indicates values for rewards (and costs) of any kind (e.g., sex, food, fluid, exertion, etc.). This cost-benefit analysis can only be performed efficiently by reference to a common measure of value. It is possible that these features of PFo build on properties of the nearby agranular frontal areas, but do so in a particularly advanced way because of the extensive and varied visual inputs provided to the PFo by the perirhinal and IT cortex. Recall that the mid-order feature conjunctions provided by the IT permit a kind of generalization that would be much harder to come by in brains, such as rodent brains, that lack an IT cortex.

Autonomic signals or information about emotional state may also influence the calculation of value in the common currency described above. Experiments in monkeys have taken advantage of monkeys’ innate fear of fake or real snakes to provide findings consistent with this idea. In an experiment by Izquierdo et al. (2005), monkeys were given the opportunity to reach over either a neutral object or a fake snake to retrieve a food reward. Thus, the benefits of obtaining the food reward were directly pitted against the costs (in terms of risk) of approaching the snake. When required to reach over the fake snake, intact monkeys displayed long food-retrieval latencies or refused to take the food altogether. In contrast, monkeys with PFo lesions exhibited short food-retrieval latencies, similar to when they reached over neutral objects. Similar findings have been reported by Rushworth et al. (2007). It seems that monkeys with PFo lesions cannot integrate the sensory signals, in this case arising from visual signals about a potential predator, with food value to guide their behavior. The inability to compute value in a common currency for risks and reward could account for these findings. Figure 4.3 depicts the results of the snake test as depending on interactions between the amygdala and PFo.

Like amygdala lesions (see Section 4.3.4), PFo lesions cause a deficit on the reinforcer devaluation task (Izquierdo, Suda, and Murray 2004). Compared to controls, monkeys with bilateral removals of PFo more often choose an object that will produce the devalued reward. Accordingly, Figure 4.3 depicts this test as depending on interactions between the amygdala and PFo (Baxter and Murray 2002), with the PFo mediating the associations between the visually sensed objects, the sight of a particular food, and its updated value.

4.5.4. Other Granular Prefrontal Areas

So far, this section has concentrated on the PFo, but primate brains have many granular frontal areas, including the dorsal, dorsolateral, ventrolateral, medial, and polar prefrontal cortex (Figure 4.2). These areas function in higher-order aspects of cognitive control, including those involved in monitoring actions (Petrides 1991) and implementing rules and strategies (Murray, Bussey, and Wise 2000; Wallis, Anderson, and Miller 2001; Mansouri, Buckley, and Tanaka 2007). Rules and strategies allow animals to gain rewards when faced with novel stimuli and contexts. Rules and strategies have many similarities, but a few differences as well. Rules are input-output algorithms that can be applied to solve a behavioral problem for novel as well as familiar stimuli; strategies are either one of many solutions to a problem or a partial solution to a problem.

4.5.5. What Can Human Brains Do with Reward?

The age-old question of what separates us from other animals is beyond the scope of this article, but a couple of comments seem pertinent because we humans can do things with reward that other animals cannot. For example, we can talk about reward. We also have the ability, called mental time travel, to remember past events and to imagine future ones as if they were current, incorporating a representation of ourselves into the event (Suddendorf and Corballis 2007; Corballis 2009). This ability probably contributes to overcoming the deep devaluation of rewards that occurs at a long delay. Mental time travel allows low-frequency and temporally distant events to guide behavior and to override choices prompted by high-frequency and more proximate events. Episodic memory can provide the substrate for a single event to guide behavior decades later. The human capacity to both remember and to plan over an extraordinarily long time horizon, therefore, has to count as one of the things that human brains—and therefore primate brains—can do with rewards.

Finally, we note that humans, at least, also have the capacity to develop second-order intentions about rewards. For example, people may not want to want food rewards. Second-order intentions are another novel way in which the primate brain deals with reward. It is extraordinarily doubtful that such a capacity exists in nonprimates and it is probably absent in nonhuman primates as well.