Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans

Theories of instrumental learning are centred on understanding how success and failure are used to improve future decisions. These theories highlight a central role for reward prediction errors in updating the values associated with available actions. In animals, substantial evidence indicates that the neurotransmitter dopamine might have a key function in this type of learning, through its ability to modulate cortico-striatal synaptic efficacy. However, no direct evidence links dopamine, striatal activity and behavioural choice in humans. Here we show that, during instrumental learning, the magnitude of reward prediction error expressed in the striatum is modulated by the administration of drugs enhancing (3,4-dihydroxy-L-phenylalanine; L-DOPA) or reducing (haloperidol) dopaminergic function. Accordingly, subjects treated with L-DOPA have a greater propensity to choose the most rewarding action relative to subjects treated with haloperidol. Furthermore, incorporating the magnitude of the prediction errors into a standard action-value learning algorithm accurately reproduced subjects' behavioural choices under the different drug conditions. We conclude that dopamine-dependent modulation of striatal activity can account for how the human brain uses reward prediction errors to improve future decisions.     

Dissociation of dopamine release in the nucleus accumbens from intracranial self-stimulation

Mesolimbic dopamine-releasing neurons appear to be important in the brain reward system. One behavioural paradigm that supports this hypothesis is intracranial self-stimulation (ICS), during which animals repeatedly press a lever to stimulate their own dopamine-releasing neurons electrically. Here we study dopamine release from dopamine terminals in the nucleus accumbens core and shell in the brain by using rapid-responding voltammetric microsensors during electrical stimulation of dopamine cell bodies in the ventral tegmental area/substantia nigra brain regions. In rats in which stimulating electrode placement failed to elicit dopamine release in the nucleus accumbens, ICS behaviour was not learned. In contrast, ICS was acquired when stimulus trains evoked extracellular dopamine in either the core or the shell of the nucleus accumbens. In animals that could learn ICS, experimenter-delivered stimulation always elicited dopamine release. In contrast, extracellular dopamine was rarely observed during ICS itself. Thus, although activation of mesolimbic dopamine-releasing neurons seems to be a necessary condition for ICS, evoked dopamine release is actually diminished during ICS. Dopamine may therefore be a neural substrate for novelty or reward expectation rather than reward itself.     

Dopamine-mediated modulation of odour-evoked amygdala potentials during pavlovian conditioning

Pavlovian conditioning results when an innocuous stimulus, such as an odour, is paired with a behaviourally relevant stimulus, such as a foot-shock, so that eventually the former stimulus alone will elicit the behavioural response of the latter. The lateral nucleus of the amygdala (LAT) is necessary for the emotional memory formation in this paradigm. Enhanced neuronal firing in LAT to conditioned stimuli emerge in parallel with the behavioural changes and are dependent on local dopamine. To study the changes in neuronal excitability and synaptic drive that contribute to the pavlovian conditioning process, here we used in vivo intracellular recordings to examine LAT neurons during pavlovian conditioning in rats. We found that repeated pairings of an odour with a foot-shock resulted in enhanced post-synaptic potential (PSP) responses to the odour and increased neuronal excitability. However, a non-paired odour displayed PSP decrement. The dopamine antagonist haloperidol blocked the PSP enhancement and associated increased neuronal excitability, without reversing previous conditioning. These results demonstrate that conditioning and habituation processes produce opposite effects on LAT neurons and that dopamine is important in these events, consistent with its role in emotional memory formation.     

Neuron-type-specific signals for reward and punishment in the ventral tegmental area

Dopamine has a central role in motivation and reward. Dopaminergic neurons in the ventral tegmental area (VTA) signal the discrepancy between expected and actual rewards (that is, reward prediction error), but how they compute such signals is unknown. We recorded the activity of VTA neurons while mice associated different odour cues with appetitive and aversive outcomes. We found three types of neuron based on responses to odours and outcomes: approximately half of the neurons (type I, 52%) showed phasic excitation after reward-predicting odours and rewards in a manner consistent with reward prediction error coding; the other half of neurons showed persistent activity during the delay between odour and outcome that was modulated positively (type II, 31%) or negatively (type III, 18%) by the value of outcomes. Whereas the activity of type I neurons was sensitive to actual outcomes (that is, when the reward was delivered as expected compared to when it was unexpectedly omitted), the activity of type II and type III neurons was determined predominantly by reward-predicting odours. We 'tagged' dopaminergic and GABAergic neurons with the light-sensitive protein channelrhodopsin-2 and identified them based on their responses to optical stimulation while recording. All identified dopaminergic neurons were of type I and all GABAergic neurons were of type II. These results show that VTA GABAergic neurons signal expected reward, a key variable for dopaminergic neurons to calculate reward prediction error.     

A subset of dopamine neurons signals reward for odour memory in Drosophila

Animals approach stimuli that predict a pleasant outcome. After the paired presentation of an odour and a reward, Drosophila melanogaster can develop a conditioned approach towards that odour. Despite recent advances in understanding the neural circuits for associative memory and appetitive motivation, the cellular mechanisms for reward processing in the fly brain are unknown. Here we show that a group of dopamine neurons in the protocerebral anterior medial (PAM) cluster signals sugar reward by transient activation and inactivation of target neurons in intact behaving flies. These dopamine neurons are selectively required for the reinforcing property of, but not a reflexive response to, the sugar stimulus. In vivo calcium imaging revealed that these neurons are activated by sugar ingestion and the activation is increased on starvation. The output sites of the PAM neurons are mainly localized to the medial lobes of the mushroom bodies (MBs), where appetitive olfactory associative memory is formed. We therefore propose that the PAM cluster neurons endow a positive predictive value to the odour in the MBs. Dopamine in insects is known to mediate aversive reinforcement signals. Our results highlight the cellular specificity underlying the various roles of dopamine and the importance of spatially segregated local circuits within the MBs.     

Cortical remodelling induced by activity of ventral tegmental dopamine neurons.

Representations of sensory stimuli in the cerebral cortex can undergo progressive remodelling according to the behavioural importance of the stimuli. The cortex receives widespread projections from dopamine neurons in the ventral tegmental area (VTA), which are activated by new stimuli or unpredicted rewards, and are believed to provide a reinforcement signal for such learning-related cortical reorganization. In the primary auditory cortex (AI) dopamine release has been observed during auditory learning that remodels the sound-frequency representations. Furthermore, dopamine modulates long-term potentiation, a putative cellular mechanism underlying plasticity. Here we show that stimulating the VTA together with an auditory stimulus of a particular tone increases the cortical area and selectivity of the neural responses to that sound stimulus in AI. Conversely, the AI representations of nearby sound frequencies are selectively decreased. Strong, sharply tuned responses to the paired tones also emerge in a second cortical area, whereas the same stimuli evoke only poor or non-selective responses in this second cortical field in naive animals. In addition, we found that strong long-range coherence of neuronal discharge emerges between AI and this secondary auditory cortical area.     

A selective role for dopamine in stimulus-reward learning

Individuals make choices and prioritize goals using complex processes that assign value to rewards and associated stimuli. During Pavlovian learning, previously neutral stimuli that predict rewards can acquire motivational properties, becoming attractive and desirable incentive stimuli. However, whether a cue acts solely as a predictor of reward, or also serves as an incentive stimulus, differs between individuals. Thus, individuals vary in the degree to which cues bias choice and potentially promote maladaptive behaviour. Here we use rats that differ in the incentive motivational properties they attribute to food cues to probe the role of the neurotransmitter dopamine in stimulus-reward learning. We show that intact dopamine transmission is not required for all forms of learning in which reward cues become effective predictors. Rather, dopamine acts selectively in a form of stimulus-reward learning in which incentive salience is assigned to reward cues. In individuals with a propensity for this form of learning, reward cues come to powerfully motivate and control behaviour. This work provides insight into the neurobiology of a form of stimulus-reward learning that confers increased susceptibility to disorders of impulse control.     

正交迭代泛函网络在中短期钟差预报中的应用

 在卫星钟源无法与地面钟源进行实时比对的时段中,准确预报卫星钟差对于维持卫星的稳定运行具有重要意义。针对卫星钟差的中短期预报问题,选择多项式模型对钟差进行建模分析,设计了一种基于滑动窗模型的正交迭代泛函网络算法。利用泛函网络的非线性学习能力对钟差预报模型进行拟合分析,采用正交函数作为泛函网络的基函数簇,并引入滑动窗思想来更新输入层元素进行迭代训练,获得较小的预报误差。分析表明,预报时间小于12 h 时,预报误差为0.2~0.5 ns,预报精度与IGU P精度相当;当预报时间为24 h 时,预报误差总体在1 ns,预报精度略次于IGU P 精度;当预报时间为1 个卫星周时,最大误差达130 ns,难以满足卫星运行对钟源的要求。研究表明：该算法适合于短期卫星钟差预报,不适合中长期钟差预报。     

Lateral habenula as a source of negative reward signals in dopamine neurons

Midbrain dopamine neurons are key components of the brain's reward system, which is thought to guide reward-seeking behaviours. Although recent studies have shown how dopamine neurons respond to rewards and sensory stimuli predicting reward, it is unclear which parts of the brain provide dopamine neurons with signals necessary for these actions. Here we show that the primate lateral habenula, part of the structure called the epithalamus, is a major candidate for a source of negative reward-related signals in dopamine neurons. We recorded the activity of habenula neurons and dopamine neurons while rhesus monkeys were performing a visually guided saccade task with positionally biased reward outcomes. Many habenula neurons were excited by a no-reward-predicting target and inhibited by a reward-predicting target. In contrast, dopamine neurons were excited and inhibited by reward-predicting and no-reward-predicting targets, respectively. Each time the rewarded and unrewarded positions were reversed, both habenula and dopamine neurons reversed their responses as the bias in saccade latency reversed. In unrewarded trials, the excitation of habenula neurons started earlier than the inhibition of dopamine neurons. Furthermore, weak electrical stimulation of the lateral habenula elicited strong inhibitions in dopamine neurons. These results suggest that the inhibitory input from the lateral habenula plays an important role in determining the reward-related activity of dopamine neurons.     

A neural correlate of response bias in monkey caudate nucleus

Primates are equipped with neural circuits in the prefrontal cortex, the parietal cortex and the basal ganglia that predict the availability of reward during the performance of behavioural tasks. It is not known, however, how reward value is incorporated in the control of action. Here we identify neurons in the monkey caudate nucleus that create a spatially selective response bias depending on the expected gain. In behavioural tasks, the monkey had to make a visually guided eye movement in every trial, but was rewarded for a correct response in only half of the trials. Reward availability was predictable on the basis of the spatial position of the visual target. We found that caudate neurons change their discharge rate systematically, even before the appearance of the visual target, and usually fire more when the contralateral position is associated with reward. Strong anticipatory activity of neurons with a contralateral preference is associated with decreased latency for eye movements in the contralateral direction. We conclude that this neuronal mechanism creates an advance bias that favours a spatial response when it is associated with a high reward value.     

Neural correlates, computation and behavioural impact of decision confidence

Humans and other animals must often make decisions on the basis of imperfect evidence. Statisticians use measures such as P values to assign degrees of confidence to propositions, but little is known about how the brain computes confidence estimates about decisions. We explored this issue using behavioural analysis and neural recordings in rats in combination with computational modelling. Subjects were trained to perform an odour categorization task that allowed decision confidence to be manipulated by varying the distance of the test stimulus to the category boundary. To understand how confidence could be computed along with the choice itself, using standard models of decision-making, we defined a simple measure that quantified the quality of the evidence contributing to a particular decision. Here we show that the firing rates of many single neurons in the orbitofrontal cortex match closely to the predictions of confidence models and cannot be readily explained by alternative mechanisms, such as learning stimulus-outcome associations. Moreover, when tested using a delayed reward version of the task, we found that rats' willingness to wait for rewards increased with confidence, as predicted by the theoretical model. These results indicate that confidence estimates, previously suggested to require 'metacognition' and conscious awareness are available even in the rodent brain, can be computed with relatively simple operations, and can drive adaptive behaviour. We suggest that confidence estimation may be a fundamental and ubiquitous component of decision-making.     

Dopamine neurons modulate pheromone responses in Drosophila courtship learning

Learning through trial-and-error interactions allows animals to adapt innate behavioural ‘rules of thumb’ to the local environment, improving their prospects for survival and reproduction. Naive Drosophila melanogaster males, for example, court both virgin and mated females, but learn through experience to selectively suppress futile courtship towards females that have already mated. Here we show that courtship learning reflects an enhanced response to the male pheromone cis-vaccenyl acetate (cVA), which is deposited on females during mating and thus distinguishes mated females from virgins. Dissociation experiments suggest a simple learning rule in which unsuccessful courtship enhances sensitivity to cVA. The learning experience can be mimicked by artificial activation of dopaminergic neurons, and we identify a specific class of dopaminergic neuron that is critical for courtship learning. These neurons provide input to the mushroom body (MB) γ lobe, and the DopR1 dopamine receptor is required in MBγ neurons for both natural and artificial courtship learning. Our work thus reveals critical behavioural, cellular and molecular components of the learning rule by which Drosophila adjusts its innate mating strategy according to experience.     

热轧非稳态过程轧制力自学习模型优化

为提高热连轧非稳态过程轧制力的预测精度,提出了一种轧制力自学习模型优化方法. 将模型自学习系数分解为层别学习系数和轧制状态学习系数,表征机架间轧制力预报偏差的遗传特性及实际轧辊状态对模型预报的影响.在系数更新过程中,根据层别距离分别对学习系数进行更新,减小了轧制规格切换时轧制力的预报误差.所提方法已成功应用于某热连轧过程,与原模型相比,优化后的自学习方法的预测偏差从2.8%降低到1.4%,均方差从3.3%降低到1.7%,有效提高了非稳态过程轧制力的预测精度和鲁棒性.     

A cellular mechanism of reward-related learning.

Positive reinforcement helps to control the acquisition of learned behaviours. Here we report a cellular mechanism in the brain that may underlie the behavioural effects of positive reinforcement. We used intracranial self-stimulation (ICSS) as a model of reinforcement learning, in which each rat learns to press a lever that applies reinforcing electrical stimulation to its own substantia nigra. The outputs from neurons of the substantia nigra terminate on neurons in the striatum in close proximity to inputs from the cerebral cortex on the same striatal neurons. We measured the effect of substantia nigra stimulation on these inputs from the cortex to striatal neurons and also on how quickly the rats learned to press the lever. We found that stimulation of the substantia nigra (with the optimal parameters for lever-pressing behaviour) induced potentiation of synapses between the cortex and the striatum, which required activation of dopamine receptors. The degree of potentiation within ten minutes of the ICSS trains was correlated with the time taken by the rats to learn ICSS behaviour. We propose that stimulation of the substantia nigra when the lever is pressed induces a similar potentiation of cortical inputs to the striatum, positively reinforcing the learning of the behaviour by the rats.     

A synaptic memory trace for cortical receptive field plasticity

Receptive fields of sensory cortical neurons are plastic, changing in response to alterations of neural activity or sensory experience. In this way, cortical representations of the sensory environment can incorporate new information about the world, depending on the relevance or value of particular stimuli. Neuromodulation is required for cortical plasticity, but it is uncertain how subcortical neuromodulatory systems, such as the cholinergic nucleus basalis, interact with and refine cortical circuits. Here we determine the dynamics of synaptic receptive field plasticity in the adult primary auditory cortex (also known as AI) using in vivo whole-cell recording. Pairing sensory stimulation with nucleus basalis activation shifted the preferred stimuli of cortical neurons by inducing a rapid reduction of synaptic inhibition within seconds, which was followed by a large increase in excitation, both specific to the paired stimulus. Although nucleus basalis was stimulated only for a few minutes, reorganization of synaptic tuning curves progressed for hours thereafter: inhibition slowly increased in an activity-dependent manner to rebalance the persistent enhancement of excitation, leading to a retuned receptive field with new preference for the paired stimulus. This restricted period of disinhibition may be a fundamental mechanism for receptive field plasticity, and could serve as a memory trace for stimuli or episodes that have acquired new behavioural significance.     

Gamma-band synchronization in visual cortex predicts speed of change detection

Our capacity to process and respond behaviourally to multiple incoming stimuli is very limited. To optimize the use of this limited capacity, attentional mechanisms give priority to behaviourally relevant stimuli at the expense of irrelevant distractors. In visual areas, attended stimuli induce enhanced responses and an improved synchronization of rhythmic neuronal activity in the gamma frequency band (40-70 Hz). Both effects probably improve the neuronal signalling of attended stimuli within and among brain areas. Attention also results in improved behavioural performance and shortened reaction times. However, it is not known how reaction times are related to either response strength or gamma-band synchronization in visual areas. Here we show that behavioural response times to a stimulus change can be predicted specifically by the degree of gamma-band synchronization among those neurons in monkey visual area V4 that are activated by the behaviourally relevant stimulus. When there are two visual stimuli and monkeys have to detect a change in one stimulus while ignoring the other, their reactions are fastest when the relevant stimulus induces strong gamma-band synchronization before and after the change in stimulus. This enhanced gamma-band synchronization is also followed by shorter neuronal response latencies on the fast trials. Conversely, the monkeys' reactions are slowest when gamma-band synchronization is high in response to the irrelevant distractor. Thus, enhanced neuronal gamma-band synchronization and shortened neuronal response latencies to an attended stimulus seem to have direct effects on visually triggered behaviour, reflecting an early neuronal correlate of efficient visuo-motor integration.     

A cellular analogue of visual cortical plasticity

Neuronal activity plays an important role in the development of the visual pathway. The modulation of synaptic transmission by temporal correlation between pre- and postsynaptic activity is one mechanism which could underly visual cortical plasticity. We report here that functional changes in single neurons of area 17, analogous to those known to take place during epigenesis of visual cortex, can be induced experimentally during the time of recording. This was done by a differential pairing procedure, during which iontophoresis was used to artificially increase the visual response for a given stimulus, and to decrease (or block) the response for a second stimulus which differed in ocularity or orientation. Long-term modifications in ocular dominance and orientation selectivity were produced in 33% and 43% of recorded cells respectively. Neuronal selectivity was nearly always displaced towards the stimulus paired with the reinforced visual response. The largest changes were obtained at the peak of the critical period in normally reared and visually deprived kittens, but changes were also observed in adults. Our findings support the role of temporal correlation between pre- and postsynaptic activity in the induction of long-lasting modifications of synaptic transmission during development, and in associative learning.     

Morphine reward in dopamine-deficient mice

Dopamine has been widely implicated as a mediator of many of the behavioural responses to drugs of abuse. To test the hypothesis that dopamine is an essential mediator of various opiate-induced responses, we administered morphine to mice unable to synthesize dopamine. We found that dopamine-deficient mice are unable to mount a normal locomotor response to morphine, but a small dopamine-independent increase in locomotion remains. Dopamine-deficient mice have a rightward shift in the dose-response curve to morphine on the tail-flick test (a pain sensitivity assay), suggesting either a decreased sensitivity to the analgesic effects of morphine and/or basal hyperalgesia. In contrast, dopamine-deficient mice display a robust conditioned place preference for morphine when given either caffeine or l-dihydroxyphenylalanine (a dopamine precursor that restores dopamine throughout the brain) during the testing phases. Together, these data demonstrate that dopamine is a crucial component of morphine-induced locomotion, dopamine may contribute to morphine analgesia, but that dopamine is not required for morphine-induced reward as measured by conditioned place preference.     

Practising orientation identification improves orientation coding in V1 neurons

The adult brain shows remarkable plasticity, as demonstrated by the improvement in fine sensorial discriminations after intensive practice. The behavioural aspects of such perceptual learning are well documented, especially in the visual system. Specificity for stimulus attributes clearly implicates an early cortical site, where receptive fields retain fine selectivity for these attributes; however, the neuronal correlates of a simple visual discrimination task remained unidentified. Here we report electrophysiological correlates in the primary visual cortex (V1) of monkeys for learning orientation identification. We link the behavioural improvement in this type of learning to an improved neuronal performance of trained compared to naive neurons. Improved long-term neuronal performance resulted from changes in the characteristics of orientation tuning of individual neurons. More particularly, the slope of the orientation tuning curve that was measured at the trained orientation increased only for the subgroup of trained neurons most likely to code the orientation identified by the monkey. No modifications of the tuning curve were observed for orientations for which the monkey had not been trained. Thus training induces a specific and efficient increase in neuronal sensitivity in V1.     

BDNF controls dopamine D3 receptor expression and triggers behavioural sensitization

Brain-derived neurotrophic factor (BDNF), like other neurotrophins, is a polypeptidic factor initially regarded to be responsible for neuron proliferation, differentiation and survival, through its uptake at nerve terminals and retrograde transport to the cell body. A more diverse role for BDNF has emerged progressively from observations showing that it is also transported anterogradely, is released on neuron depolarization, and triggers rapid intracellular signals and action potentials in central neurons. Here we report that BDNF elicits long-term neuronal adaptations by controlling the responsiveness of its target neurons to the important neurotransmitter, dopamine. Using lesions and gene-targeted mice lacking BDNF, we show that BDNF from dopamine neurons is responsible for inducing normal expression of the dopamine D3 receptor in nucleus accumbens both during development and in adulthood. BDNF from corticostriatal neurons also induces behavioural sensitization, by triggering overexpression of the D3 receptor in striatum of hemiparkinsonian rats. Our results suggest that BDNF may be an important determinant of pathophysiological conditions such as drug addiction, schizophrenia or Parkinson's disease, in which D3 receptor expression is abnormal.