The frontal and parietal cortex: Eye movements and attention (1:15:18)
Date Posted:
July 5, 2017
Date Recorded:
August 19, 2016
Speaker(s):
Jacqueline Gottlieb
All Captioned Videos Brains, Minds and Machines Summer Course 2016
Description:
Jacqueline Gottlieb, The Kavli Institute for Brain Science at Columbia University
JACQUELINE GOTTLIEB: Hello, everybody. I hope you're not too tired after your tutorial. And that at least the freezing temperature in this room can keep you awake. OK, so I will speak about the physiology of the frontal and parietal cortex. And what we know about how these areas direct eye movements and attention.
And I will tell you, I will give you a broad survey of what we've learned from considerable number of years of doing physiology in these areas. But in the second part of the talk, I'm going to move on and talk about vast ignorance. A topic about which we know very little.
And this is by the way this is a little book that was published by Stuart Firestein who is a biologist at Columbia, neurobiologist at Columbia, it's called Ignorance. And it's a very cute little book. And I recommend it. It's very fast reading.
The point is that we scientists enjoy talking the most about what we know the least about. So we're going to get quite a bit of this, a question, a very widely open question that I'll touch about in the second half of my talk about how the brain actively allocates attention and assigns value to sources of information.
All right, so let's start with the frontal eye fields and the lateral intraparietal areas. Those are the best characterized areas in the monkey brain that we think have to do with controlling attention and eye movements. And there are homologous structures in the human cortex that I won't talk about today. But you should just know that they're there.
So in this impossibly complicated diagram, FEF and LIP sit about at this level. So not quite the top level of the visual hierarchy, but fairly close to the top. And here they are on the lateral view of the monkey cortex. The FEF sits in the arcuate sulcus, which is the very posterior border of the frontal cortex. This is all called the prefrontal cortex. LIP in the lateral intraparietal sulcus.
So we think of FEF and LIP neurons as providing what we call a priority signal for control-- for guiding attention. And I realize that you have not had, in this course, any real lecture about attention. So I have to summarize it in one slide, all that we know. So let me do this in one slide.
So the best way to summarize, this is in-- this is taken from a paper by John Reynolds and David Heeger, in which they propose sort of a model of attentional modulations that really synthesizes a whole host of behavioral and neural observations. And the focus of the model is on this circuitry over here, which is thought to be implemented in mid-level visual areas, in-- most of the data come from V4. There's also some data from V2.
And basically, this is a very simple proposed circuit. So basically, the model takes a stimulus input that has two sources of information. Initially, they're undifferentiated. And the output here is a biased representation in which one of the sources is enhanced, here, the Gabor on the right. And the way that Reynolds and Heeger propose that this happens is using a competitive normalization circuit, where there is a suppressive drive that multiplies this visual representation.
It's a divisive normalization. And you can get this sort of enhancement effect that can reproduce a range of effects that have been seen in V4. But the focus when we speak FEF and LIP-- so this all happens in mid-level visual areas. When we talk about FEF and LIP, we talk about this entity here, that Reynolds and Heeger call the attention field, and which is supposed to drive the attentional modulation.
So this is the signal that says, hey, visual cortex, bias your representation in such a way. And you can see that the signal is unidimensional. It, for example, is not tuned to orientation. It's only tuned to spatial location. So in a sense, the attention field just says attend to the right.
And this attention field is modeled, according to experimental data from single neuron recordings in a FEF and LIP. So we can review some of these response properties. So if you look at FEF and LIP neurons, you find that there are many cells that are simply visually responses-- visually responsive and have retinotopic receptive fields. That is every cell talks about some location in your visual field.
They have-- the neurons have sustained responses in the sense that if you see a brief visual transient, they will somehow integrate the activity and maintain it during a memory or delay period. And one of their most important properties is that they don't respond to just anything that falls in the receptive field. They are, instead, they are magically very selective for some things that are attention worthy. And this is really the main thing that differentiates them from a visual neuron, such as, say, in the retina.
So here is a typical response profile in these areas. This is from a very early study by Bruce and Goldberg, 1985, this was the study that actually defined the frontal eye field that has been studied ever since. And the typical response pattern is the visual transient. This can be pretty fast. Responses in LIP, we can have latencies of 50 milliseconds.
And then sustained activity, and then sometimes, in some neurons, so this is a task in which the monkey is shown a visual transient. Then the visual stimulus goes off, monkey has to remember the location at which the flash occurred. And then later, when he gets a signal, make an eye movement, a rapid eye movement to that location. So it's called a memory guided saccade task.
So you have visual, memory, and sometimes saccade related responses. And these are tuned. So for example, they would be tuned for the direction of the eye movement, or the visual flash, and also for the amplitude. So I told you-- so these are not just simple visual responses in the sense that they can ignore many, many stimuli that appear in the receptive field.
And I showed this when I was a postdoc with Mickey Goldberg, we showed this in area LIP. Using a task in which the monkeys were shown an array of eight objects on each trial. And one of these objects was always in the receptive field. So for example, maybe this red circle would be in the receptive field of the cell.
But the monkey received a cue on each trial, which instructed him to make an eye movement to a different object. So for example here, the cue told him to find this green diamond and make an eye movement to it. And what we find in LIP neurons is-- so these are firing spikes aligned to the beginning of the cue, so this thing here, that tells them what to pay attention to. And these are responses aligned to the time when the monkey makes the eye movements, so while he's finding that object and preparing his eye movement.
So you can-- you can see that the monkey-- that the neuron responds selectively if the cue instructs him to make an eye movement to the receptive field. This condition here is when the cue is a red circle and the monkey is preparing a saccade up. But the neuron doesn't respond to it any other times, even though the stimulus is always in its receptive field.
So this is a really important feat. In that this priority representation in LIP and also in FEF really throws out most of the information. And this is, I think, absolutely vital for our survival. Because if we didn't do this, we would be simply overwhelmed and not be able to function. And so there's a magical capability here to pull out whatever is interesting.
So we have selection here. Now, also notice that this is sort of voluntary top-down, a voluntary top-down selection response. If you follow the literature in this area, you might hear claims that LIP encodes bottom-up attention. Whereas, the frontal cortex encodes top-down attention. And I have to warn you that this is an oversimplification and not quite accurate. There are plenty of responses to top-down attention also in the parietal cortex.
At the same time, we know that these neurons also respond to bottom-up attention, just simple salience. So we can see this here. Here, in this condition, we did something very similar to what I talked about before. Except that we took this cue that flashed on every trial and flashed it in the receptive field of the cell. So there was a flash, the instruction cue, and then the monkey had to plan a saccade to any one of the eight directions.
So you can see that this evokes a visual response in every trial. And this is simply a salience-- this is most likely a salience response to that cue. And then there is the saccade planning response, which is specific to saccades that go towards the receptive field. So we have an integration of bottom-up and top-down factors on the same cells.
So this is pretty much the priority response, a selective response that combines bottom-up and top-down types of selection, and just tells you what's interesting in the world. OK, there is evidence. And so if you remember that model, I told you that this top-down-- this priority response feeds back to visual areas and somehow produces the attentional modulation.
And there's anatomical evidence for this. This is just shown for FEF. There are projections from FEF to LIP reciprocal projections, but also to many visual areas, including MT, and V4, and higher level inferior temporal cortex. And the same is true for LIP, there are feedback projections to all these areas.
And we know, so work from the laboratory of Tirin Moore provide some direct evidence that if you stimulate FEF, you can produce enhancement of visual responses in area V4. So here is how the experiment went. The monkey saw a bunch of oriented bars. And they had to pay attention to one of these bars. They were cued, for example, this bar. Sometimes there was more than one bar in the receptive field, and they were cued to pay attention to one.
And the recordings were in area V4. So the authors were monitoring the responses of a V4 neuron. But they also had the stimulating electrode in the frontal eye field. And they had to go through great pains to align the two receptive fields. So they had to find, characterize the receptive field of the recorded cell in V4, And then find the corresponding receptive field in the frontal eye field.
So here is, for example, here's an example in which the receptive fields were aligned. So the eye movement that was evoked from the frontal eye field, so that's an indication of where the receptive field was at the frontal eye field site, was sort of like this, down and to the left. And the V4 receptive field was partly overlapping with those saccade endpoints. And if the monkey was attending to a stimulus here, then microstimulation of the FEF produced a slight enhancement of the V4 receptive field.
If, on the other hand, the visual stimulus was not aligned with a saccade endpoint, there was no consistent enhancement. So this is sort of a statistical illustration of this. So this is-- the x-axis here is the separation between the saccade endpoint and the center of the V4 receptive field. And you can see that there is enhancement if this separation is small. And if the mismatch is too large, you no longer get the effect. So that's evidence-- so that suggests that the priority signal sends some spatially organized feedback to V4.
We also know that this priority response correlates with behavioral measures of attention. And this is-- so this is from a study that was done in LIP by Bisley and Goldberg a number of years ago. In which they-- in which they measured attention during the planning of an eye movement. So the task was somewhat complicated.
But all of this was necessary to accomplish their goal. So here's how it went. The monkey began by fixating. Then they were shown a target. And they were asked to remember to make an eye movement to this target. So the target was flashed here. And by the end of the trial, when they get the go signal, they have to make an eye movement to that target.
But in between, several things happened during the delay. On some trials, there was another flash. And that was always a distractor, an irrelevant distractor. And then the monkeys received a signal of whether to actually make or cancel the saccade. And this happened here with the visual display. And this was necessary in order to probe where the monkey's attention was at any point in time.
So the way the signal went is there were four circles on the screen. One of them was what's called the Landolt ring. It had a little opening. And the opening could be to the left or to the right. And the monkey learned that if the opening was to the left, this meant go ahead and make that eye movement that you planning. If the opening was to the right, that meant don't make the eye movement.
So this is an instruction. And this is a difficult perceptual discrimination. So this was the author's way of determining where the monkey was attending at. So imagine the monkey-- you flash the target and the monkey is attending at this target location all this time because he's planning to make his eye movement. Then his ability to follow this instruction would be best if the probe happens to fall at that target location. And it would be a little less if the probe is somewhere else in the visual display.
So they made this discrimination very difficult. They titrated the contrast of the probe to really be near the threshold of this discriminability. And then they could infer where the locus of attention was. So their goal here was, first of all, to see if attention stays at the target. And second, if it shifts when this distractor flashes, which is salient, but irrelevant, what happens to the attention? Does it shift to the distractor? Does it not shift, or what happens?
OK, so here's what they found. So first of all, let's look at how the neurons respond. So you see, the neurons responded both to the target and to the distractor. So in blue, you see responses to the target. If the target appears in the receptive field, there's a very fast visual response. Then there's that sustained memory activity that I spoke about. And then there's a saccade response at the end.
If a distractor appears later during the delay period, the neurons also respond very strongly to the distractor. But this is transient. So you can see that as if attention is drawn to the distractor, but by this time again, the influence of the distractor goes down. And the target again dominates. So it's if attention is drawn to the distractor and then comes back to the target.
And what they found-- and then, let's look at this in relation to the threshold measurements that tell you where attention is allocated. So their finding was that while the responses to the distractor are above those for the target, so the red is above blue here, the perceptual advantage is at the distractor location. That is there are lower contrast thresholds at the distractor location.
But when the distractor responses dip below those to the target, then the perceptual advantage goes to the target location. So you can see that the balance of activity in LIP shifts at the same time that the perceptual threshold shift between the two locations. So this is, to this date, I think, the most solid evidence that this priority response, indeed, has something to do with the perceptual modulations that are taken as an index of perceptual attention.
All right, so another theme that emerges from these studies of FEF and LIP is that the neurons in this area really encode two forms of selection. One is visual selection, so signaling which stimulus is interesting in your world. And the other is motor selection. That is signaling should you make an eye movement to that stimulus or not.
And those are-- so there has been-- you guys are too young to know about this. But back when I was starting out, there was a hot debate, the intention debate. And the question is, well, do these priority responses, do they indicate the location of a stimulus or the location of an impending eye movement. And there are many, many studies going back and forth on this. And this is what we learned from them. Is that there really are two stages. And there's a transition between visual selection and motor selection in these areas.
LIP neurons are much more strongly visual. And they do not consistently signal eye movements. If you look, whether-- can you look at LIP neurons and know whether an eye movement will occur? The answer is no. They are inconsistently related to eye movements. And this is one way in which we demonstrated this. So what we did is a prosaccade antisaccade task, which is a classic task of dissociating visual and motor selection.
So this is how it went. The monkey fixated in the center. We are recording from a neuron with a receptive field, let's say it's on the left here. And we simply flash the visual stimulus and ask the monkey to make an eye movement. Now, the color of the fixation point told the monkey which eye movement to make.
If the fixation point was black, then the monkey had to make an eye movement to the stimulus. This is called the prosaccade. So for example, if the stimulus is in the receptive field, he has to go to the receptive field. If the stimulus is away, he has to go away, just follow the light.
If the fixation point is white, the monkey has to make an eye movement away, opposite the stimulus. So if the stimulus is in the receptive field, the saccade has to go to the opposite blank screen location. If the stimulus is outside the receptive field, the saccade can go to, towards, the receptive field. This is not an easy task.
If I asked you to-- if I flashed a bright light and I told you look away, many of you would fail on some fraction of the trials. And this is a diagnostic task of frontal function. So we know that patients with frontal lesions have real great difficulties in making antisaccades, controlled saccades.
But we used it here in order to determine whether our LIP neurons encode visual selection or motor selection. So if you have a cell that encodes visual selection, it should respond whenever the stimulus flashes in the receptive fields of here and here, regardless of whether the saccade goes to the receptive field or away. If you have a stimulus that encodes motor selection, it should respond whenever the eye movement goes to the receptive field, regardless of whether there is a stimulus there or not.
And we found that the majority of cells in LIP were of this type, they're visual cells. So here are four of them. So what you see here on the left are trials in which the stimulus was in the receptive field. I'm aligning them on stimulus onset. And then here is the time that the eye movement occurred. So you can see this cell responded if the stimulus is in the receptive field. Did not respond if it was outside. And the eye movement direction makes no difference.
So here on top are eye movements to the receptive field. On the bottom are away. It made no difference for the cell. Yeah?
AUDIENCE: For the antisaccades, what's the normal trajectory? Do they normally just go straight away? Or do they inch out first towards the prosaccade and then correct?
JACQUELINE GOTTLIEB: No, we usually see-- these are highly trained monkeys. So I'm going to show you some trajectories in a minute. They're usually straight. They are straight. They're usually hypometric. They're usually a bit slower than prosaccades. But they're pretty much straight.
And we throw out trials in which the-- first of all, we train. There is, in the beginning, there is behavior of going to the stimulus and then going away. But that's usually trained away by the time we do this. So a purely visual cell, this cell here, again, visual responses, you might say maybe a little bit a stronger activity for a saccade toward than away. This cell, again, visual responses, maybe a bit of saccade response.
This cell here is a very notable exception. We had very few of them in our sample. But I'm showing it. You can see it has weaker visual responses. And it had a real clear eye movement response. So it began ramping up before the saccade if the saccade was in the preferred direction, but not if it was away.
But again, this was a minority. And if you look across the population, what we see is that the visual-- the information about cue location in the visual response is much more robust than the information about the saccade direction, even immediately before the monkey's making the eye movement. And we also found that these neurons are-- so the neurons are visually dependent.
And here's another way of demonstrating this. So what I'm showing you here are responses to prosaccades, prosaccades and antisaccades to the receptive field. So all, here, the eye movement trajectories, all these eye movements are directed to the receptive field of the cell. So you can see the trajectory for prosaccades, antisaccades, they're fairly similar, a bit more scattered and hypometric.
If there was a visual stimulus signaling the eye movement, there is this sustained visual response. And there is a saccade related burst. So you can see this burst here has motor-like timing. This is the average response. It ramps up just before the eye movement. So if you looked at it, you'd say, oh, it's a motor response.
But if the monkey makes the same eye movements, pretty much, without having had a prior visual stimulus, then even that motor burst goes away. Pretty much the whole response, including this motor-like activity goes away. So there's very strong visual dependence to the responses in LIP.
Now, this picture tells you-- raises another question. And that is the monkey is making this eye movement. So something-- but LIP neurons don't know about it. So what is it in the brain that makes the eye movement? And the answer, you might be able to guess by now, is somewhere in the frontal cortex.
So in a more recent study, we looked at this dissociation in an experiment in which we compared the activity in LIP and the dorsolateral prefrontal cortex. So this isn't-- we were in the area just anterior to the frontal eye fields. And that's an area that has some similar properties to the frontal eye field. It also has spatially tuned cells.
And it has strong projections with the frontal eye field. So what we did here is we looked at how neurons in LIP and the prefrontal cortex encode targets and distractors. So again, we use sort of a variant of this delayed saccade paradigm. The monkey was fixating. We flashed a visual stimulus indicating a location that the monkey had to remember, and then make an eye movement to it at the end of the trial.
And during the delay period, we flashed a distractor. Again, a light that was totally irrelevant, but highly salient. So in a sense, this was similar to the experiment that you've seen before that was done by Bisley and Goldberg. And we played some distractors could be near the target, some were farther away. And we also varied the asynchrony, the timing relative to the target when the distractor was flashed.
So here's what we found in LIP and dorsolateral prefrontal cortex. So first, if we look at the responses to the target in the receptive field. On some trials, the target was in the receptive field. And after a delay, the monkey made his eye movement to the target. And these are the trials shown here in gray.
And you can see that on these trials, LIP and prefrontal have very similar responses. And again, this is the visual, delay, and saccade related activity that I've shown you before. And people have been looking at this, getting these sort of responses for a long time, and concluding that these areas are sort of functionally similar.
But if you look at the responses to the distractor, so then, now, we look at trials in which the target was outside of the receptive field. It doesn't activate our cell, but we flash distractor in the receptive field. And what we see here is that LIP neurons have these very strong responses to the distractors, just like I've shown you in the Bisley and Goldberg test. The neuron responds and then comes back.
But in prefrontal cortex, the story is entirely different. And what we see in this case is that the moment the target appears somewhere else, the responses at other locations begin to be suppressed. And it's actually quite a strong inhibition that we see. The baseline responses are suppressed. And then when the distractor arrives, it evokes much, much weaker responses. In many cells, we found no response, which is really an amazing degree of suppression, given that distractor is highly salient and smack in the receptive field of our cells.
And we did a number of analyses that showed that the monkeys made a good number of errors. They were captured by the distractor on many trials. So they remembered the wrong location. And we found that the earliest predictor of this distractor capture was a failure of suppression in the frontal-- in the PFC. And LIP neurons only showed a difference between correct and error trials much later than the prefrontal cells.
And finally, we did a direct manipulation experiment in which we injected small amounts of muscimol, which is GABAA agonist and temporarily silences neural activity. And we found that-- so this is the change in error rate. So how many more mistakes in saccades to the distractor did the monkey make? Tiny amounts of muscimol in the prefrontal cortex increased error rates dramatically, I mean, around 50% or 60%.
Whereas, much larger effects in LIP had much smaller effects on error rates. So what this suggests, then, is that we should really be thinking of two types of selection. And these systems can function somewhat in a partially independently in our brain. We have a frontal system that does sort of goal-oriented action. So it tells you what should I do. Should I make an eye movement or not?
But we have a posterior system that keeps track of interesting things in our world, even if we don't act on them. So they're two different things. One I select what to attend to. And the other is I select what to do about it. And it's very important that we keep those-- of course, they have to be coordinated. But the fact that we have some degree of separation is really important for behavioral flexibility.
All right, and I should also add, again, repeat what I said before, you shouldn't really think of this as a bottom-up system. So maybe the reflex, jumping to conclusion here would be that LIP is bottom up and just respond to all salient objects. But I don't-- it's not really true. This exploratory or monitoring function could be, actually, quite a high level, top-down, controlled exploration type response.
So LIP does visual selection. And I told you a little bit about the dorsolateral prefrontal that does more action-oriented selection. Now, in the frontal eye field, so just posterior to dorsolateral prefrontal is actually where most studies of visual and eye movement responses were carried out. And there, people also see-- so again, this basic response pattern.
But there are cells that have visual delay and saccade activity to different degrees. So they're visual, visuomotor, motor cells. And there, this seems to be quite neat continuum of roles. So this is a task in which the monkeys use covert attention. That is they monitored the stimulus covertly in the visual field, but never made eye movements to it. And in visual cells, really modulated, and so they responded if the attention was in the receptive field, but not if it was away.
The visual movement cells, so cells that have visual and delay activity, again, modulated with covert attention. But movement cells, those are cells that only respond to an movement, as you might predict, did not tell you whether attention is covertly deployed. And these movement cells are really tightly linked to eye movements.
So here's an example of one of them. Here, the visual stimulus flashes at this time. You can see the cell has no response. But it ramps up sharply before the eye movement. And these cells really behave like a rice to threshold mechanism. So this cell, for example, when it reaches 99 spikes per second, or 97 spikes per second, that's when the eye movement is triggered.
Sometimes, it takes-- it ramps up a bit more slowly. And then the eye movement has a longer latency. Sometimes, it ramps up faster. And the eye movement has a shortened latency. But I mention this because you will hear a lot of work on decision making that tries-- that suggests that LIP neurons ramp to threshold when a saccade is made.
And you might be able to tease this out from the data. But you have to remember that those LIP neurons that seems to ramp to threshold also have huge visual responses when eye movements are not made. So the fact that neuron ramps to threshold doesn't really predict the eye movement. The best example of a ramp-to-threshold decision process is these neurons, the movement neurons in the frontal eye field. But we don't know what these guys do in a decision scenario, and how they integrate evidence, and stuff.
And also from microstimulation, this is, in fact, the way that the frontal eye fields were initially discovered and defined is by low current and microstimulation. And if you pass small amounts of currents, you can force the eye to go in different directions. And then there is an orderly topographic map. Large saccades are more medially. Smaller saccades are represented more ventrally, along the sulcus.
And the thresholds here are very, very small, usually 50 microamps or below. You can evoke saccades from FEF with as little as 5 microamps. Whereas, if you went to LIP, you need more than 200. So FEF, it has visual selection. But it also has-- it also has this dedicated circuitry for saccade selection. And it's closer to the brainstem and superior colliculus, saccade generating circuitry.
All right, so I think, now, we kind of came full circle. And I've shown you all the physiological evidence that leads to this picture, leads to calling LIP and FEF, this priority map, and claiming that it controls or directs visual attention. So at this point, then you might say, well, we're done. That's it. We figured it out. There's nothing more to-- what else is there to find out?
But that's not the case. We actually-- there's actually a huge question that is hidden here. And that question is hidden in plain sight. And it is right in this picture in front of us. And the question is this. So if you look at this attention field for long enough, you realize that it's kind of a funny entity.
It's a box that has an output. It has an effect that we know about. But it doesn't have any inputs. It's just hand coded into the model. So what's going on? All right, so what does this mean? So this means that at this point in our scientific life, we know that these neurons encode an attention field, attentional command signal.
But we have no clue how they come to derive it, how they compute it. How do they know to attend to the right? So what controls the controller is a huge open question. And this is-- I mean, this is a question that, of course, has been in plain view for many, many years. And yet, our field has-- we sort have ignored it. We have a huge scotoma when it comes to it.
When we started to think about this question in my lab, I even had a problem convincing my fellow, Mickey Goldberg, my postdoctoral advisor that this is even an interesting question. So I would ask him, well, why do you make an eye movement? And he'd say, well, because the neurons fire in LIP and FEF. But why do the neurons fire? It's like, oh, they just fire.
Why is that not an interesting question? I don't know. All right, well, I think it's an interesting question. So one road you can take is you can say, OK, well, just God put that priority response there and be done with it. But of course, if you want to be a scientist, you may want something better. So how does our field sort of attempt to deal with this gap?
And has done it so far, and the first response is what we humans do whenever things get difficult. And that is avoid the question. So the standard answer that we have for the priority map, you've heard about it before, is the idea of saliency maps. Is the idea that it's all bottom-up. So this is the Itti and Koch model. You can simply take the visual features, and analyze them, and extract contrast, and that's what determines the priority response.
So we already know that this doesn't explain a lot. So in most task situations, the bottom-up input has very weak effect on our behavior. I mean, you're not steering to this bright light on the ceiling all the time. The second way, which is a little better way, I think we've heard some of it from Gabriel this morning, is to sort of tackle a reduced version of the question.
And that is, well, assume that the top-down signal exists, hand code it into the model. And then ask how it influences visual processing. So for example, Gabriel said, OK, here, assume that I know the properties of the hat. And then I'm biasing my attention towards locations that, they kind of look like the hat. And then also assume that we have a feature based biasing system that kind of works through LIP and FEF.
But still, we're not addressing the question, because we just had hand coded that top-down signal there. And we haven't explained how it comes about. And so to really fully answer this question, I think we have to really change our way of thinking about vision and really appreciate the fact that our visual system is there to serve our behavior goals. And it is a motivated system in the sense that it has to be coordinated with what we are trying to accomplish.
Sometimes you hear this question mentioned. But I think that we really haven't-- we're not even close to starting to dig into all its implications. So if you think about all this visual circuitry, it doesn't just happen in a vacuum. It happens because the monkeys in these tasks have to make a decision. So for example, here, you say OK attend to this Gabor and release a bar. If it's tilted to the right, release the right bar. If it's tilted to the left, release the left bar.
So you have to decide which bar to release. And really, your life depends on this, because you're going to get a reward at the end. So then, there has to be some mechanism by which the needs of this decision and your desire for the reward feeds back and tells the brain to what to attend. The stimulus is only interesting because it's relevant for guiding my actions and eventually getting a reward.
So we need to think about this. And this way of thinking leads, really, to very different kinds of paradigms than we have been using so far. And I think that it's very, very early in our exploration of this question. So that's why it's so interesting for you guys who are just starting out. This is like a minefield of questions and topics to work on.
But the first thing that we need to change a little bit is to think about decision-making not as a single step task. Here I am, and I do something, I release the right or left bar. But as it is at least a two-step task. So you can see here then in the circuit, we really have two kinds of decisions.
We have first the decision of which stimulus to attend, do I attend to the right or to the left? And then the decision of what to do about this, do I release the right or left bar? And this is really how our behavior works. When you cross the street, you look at the traffic light. And then you decide what to do about it. All the time, we have this sort of attend, act, reward sequence.
So now, behaviorally, we have plenty of evidence that there is a very tight coordination between decision making and attention allocation. So in any task context, attention or eye movements are very, very tightly correlated with what we are about to do. And we've-- so this is one of my favorite studies of hand-eye coordination. So here, what people have to do is grab this little bar, and move it past this sharp point, and move it here, and press a little button that was on top.
And here are the eye movements that people made. You can see that they are very, very tightly allocated to the contact points in the task, the contact where you grab the bar, the contact of the bar with a table here, this danger point, and finally, the target. And this is highly reproducible from subject to subject and within individual subjects across days, so very, very tight coordination.
So this is in a sensorimotor task. The same thing in a cognitive task, this might be an example that you're more familiar with. This was shown from the earliest recordings of eye movements by Yarbas. Who simply asked people to look at paintings. And then he asked them different questions.
So sometimes, an observer might be asked how wealthy is the family. And in that case, he looked at the furniture, and the clothing, and so on. But then, if you ask him how old are the people, you can see that the gaze pattern changes. And now, he's looking much more closely at the faces of the people.
So then we need to kind of think about attention as a form of decision. It's a decision that selects the evidence that informs our actions. And the question is then how is that optimized to best serve our actions. What are the value functions that tell us to what to attend?
OK, so as I said, very, very early days. But I can give you a preview of some findings from physiology and from behavior that get us started on this question. So one hint goes back to Gabriel's question is that in contrast with that simplified model, in which priority is unidimensional, so the neurons just tell you attend to the right, and just seem to magically acquire that signal. In fact, we find that the priority response, this has been mostly shown in LIP, is actually much more differentiated and contains information about the task and about the action that you're going to do after you attend, after you direct attention.
And the second thing that I'll talk about after this is that these neurons are-- we have reason to think that the neurons will be sensitive to estimates of reward and uncertainty reduction. And that those may shape the priority response. OK, so let's look first at the content.
So here, all right so this is what I just said. Instead of a unidimensional response, we actually have modulations by-- and different, there's evidence out there from different papers that features of the stimuli, the task rules and context modulate this spatial selection response. And here, I'll just show you one example of such an effect from our studies, in which the modulating factor was a manual action, a simple manual bar release.
So here's what we did in this task. The monkeys had to attend to a visual display that was initially homogeneous, but then changed to reveal unique shapes at every location. One of these shapes was a target. So there was this E stimulus. And the E could be facing to the left, or it could be facing to the right. All the other stimuli were irrelevant.
And if the monkey-- if the E was facing to the left, the monkey had to release a bar that he was holding in the left paw. And if the E was facing to the right, he had to release the right bar. So in a sense, it's very similar to the attention tasks that are done in V4, just reporting the orientation of a stimulus.
But the point of this task is to really more clearly dissociate the different operations that are going on here. So one operation is-- OK, so one operation is simply finding the target in the visual display, finding that E. The E could appear anywhere. And the first thing is to find it and direct attention to it. And this was without an eye movement. The monkeys had to keep fixated in the center.
And the second stage was, of course, to decide what to do, release the right or left bar. And we designed the task so that we really did our best to ensure that the bar release would not activate LIP cells. The monkeys could not see their hands. And anyway, the neurons do not respond to limb-- they don't have limb motor responses. And the hands were out of sight. So we saw no reason why these neurons would respond-- would have anything to do with the bar release.
And this is-- so in the classical thinking, remember, that target location, shape discrimination, and bar release happen in different areas. The target will be localized in this attention area, LIP, FEF, maybe the superior colliculus. The shape discrimination we would say goes on in V2, V4, IT. And the release of the bar will be driven by some premotor areas somewhere else.
What we found, however, much to our surprise, is that these LIP neurons do have a signal of target location, visual location. But it's modulated by what the monkey is doing with it. It's modulated by the manual release.
So let me actually show you-- let me just jump to this. So this is an example of such a cell. So what you see here are trials in which the E was in the receptive field of the cell. Here is the time when the visual-- the search display appeared. And the black dots are the times when the monkey released the bar.
So if the E is in the receptive field of the cell, the neuron begins responding. You can see there is a latency here, which correlates with the fact that search is not-- this is a difficult search task. And if you put more distractors, it takes longer time. Nevertheless, the neurons find the E after a little while. If there's a distractor in the receptive field, the firing rate goes down.
So the cells clearly tell you it's on the right or it's on the left. But in addition, the cells discriminate, this particular cell responds more strongly if it's a left facing E requiring a left bar release and if it's a right facing E requiring a right bar release. And we found that this was the case-- some neurons only-- some neurons did not show this modulation. But this modulation was present in about 50% of our cells. So it's a pretty significant signal.
We determined that this is not due to the shape of the E. So you can say this is shape selectivity. But it's not really straight shape selectivity, because we trained the monkey on a second set of stimuli, it was a U pointing up or down. And the neurons maintained a constant hand preference. So if a neuron liked the left bar release here for the E, it also like the left bar release for the U. And that's very likely to happen just from-- unlikely to happen just from a pure shape selectivity.
We also found that-- sorry, we also found that this is not related to the location of the limb in space. Because we trained the monkeys to do the task with their hands crossed. And that didn't make any difference. So if a neuron liked the left hand on the left side of space, it also like the left hand on the right side of space. So it's really effector specific.
And so, wow, what's going on here? So are we missing something really big? Is LIP somewhat involved in hand movements as well? We think that, no, that's not the case. We don't think it's involved in hand movements. But we think that it receives information about hand movement that somehow modulates a visual selection response.
And you can see that here. This is actually a very important point. This hand selectivity is a modulatory response. It's not really a motor, a primary motor response. So one way you can see this is here. In these trials, the monkey's attending outside of the receptive field. But he's making the same hand movements. He's making the left and right bar release. But our neuron doesn't tell you about the hand movement in these trials.
So it's somewhat similar to what I was showing you before. Sometimes the monkey makes eye movements and the LIP neurons don't know about his eye movements, the same for hands. And also, if you look at the responses across the population, what I'm showing you here is these are three different set sizes and more distractors on the screen. They reduce the response.
But anyway, in every case, you can see that the response is increased. If the E is in the receptive field, they go down at the distractors in the receptive field. But notice that the peak response is here early. And by the time the monkey's making the eye movement, this area is sort of going offline.
The responses are already going down. So we think that what's happening here is that in this period, motor planning, and some other hand related area probably increases and drives the hand response. And finally, we looked at this with inactivation. So we inactivated LIP. And we looked at whether-- and what we found is that the inactivation reduces performance, so reduces percent correct, and increases the reaction times if the E is in-- if the E is in the contralateral hemifield.
So if the E is in the-- so if when you put muscimol in one side of the brain in LIP, you produce some slight visual neglect in the contralateral side. So if the E was in that contralateral inactivated side, we see that there is a decrement in performance with inactivation relative to control. But if the E was not-- was in the good hemifield, not inactivated hemifield, nothing happened. And in particular, there was no effect that was specific to the limb. So the contralateral hand was not affected. In general, hand movements were not affected at all.
So this is, then-- so then we don't really have to change our story about these neurons encoding visual selection. That is their basic function. But the selection response doesn't come in a vacuum. In fact, it contains information about the action associations of the stimuli that are being selected. And whether and how this has to do-- feeds back to modulate visual representations, I think it's a fascinating question.
So we think that this sort of feedback is part of a puzzle of how-- in order to direct attention in a task related way, you obviously have to have a lot of knowledge about what stimuli mean. When you cross the street, you wouldn't look at the traffic light unless you knew what it means for your future actions. So there's a lot of learning of stimulus action associations that is required to orient attention appropriately.
But this sort of knowledge base is not sufficient to guide your attentional control. It's necessary, but is not sufficient. Because on a moment by moment basis, you also have to determine whether you have the need for that information. And that is really crucial. So you might know the associations of a traffic light. But you're not going to look at it unless you need it at that moment in time.
And so it seems from all-- if you think about it, you might say that, well, what we need is some sort of signal-- some sort of signal of the rewards that are associated with your task, how much you want to accomplish something. And also, very importantly, a signal of the uncertainty that you have in your decision, because after all, if you know what you're going to do, you don't need to pay attention to anything. You just go ahead and do it.
So what did I want to say? What did I want to say here? I don't remember. All right, so let's just move to rewards. So there have been a lot of studies in the eye movement system that look at effects of rewards. And they find very reliable effects. This is both at the behavioral level and also in LIP cells.
So these are studies of decision making. And usually what happens in these studies, the monkey see two stimuli. And the stimuli have different reward associations. And then one-- and the monkeys are allowed to choose between them with their eye movements. One is in the receptive field. The other is not.
And here is the tip-- one representative result from such studies. So what you see here is a free choice task. This is from the lab of Bill Newsome. A blue stimulus and a green stimulus, and they have-- each one has probabilistic rewards. And the probabilities change in a dynamical fashion. So sometimes, the blue stimulus may have higher probability, sometimes the green. And it varies across several levels of probability.
And the monkey's track these probabilities. And they choose what they estimate is the best at the current point in time. So here are trials in blue-- in blue are trials in which the eye movement is directed to the blue stimulus. And that stimulus is in the receptive field. And in green, the monkey makes his eye movement to the green stimulus, which is outside of the receptive field.
So you see, first of all, you see that the cells give their target selection response. That is all of the blue traces are above all of the green traces, because the eye goes to the receptive field in blue and away from the receptive field in green. But if you divide it by reward probability, you see that there's a clear scaling by reward probability as well.
If the saccade to the blue is expected to have a very high reward probability, the responses are very high. If it's expected to have a lower probability, the responses are lower. And the opposite happens for when the eye movement goes away. So the very reliable effects of reward that are found-- and these are replicated in many different studies.
And you can say, well, of course, this makes sense, because this could be a mechanism for coordinating attention and decision making. If you are going to get information, and then do something, and receive a reward, that reward can assign value to everything, everything in this decision chain, including the sampling of the stimulus. So this could be a useful mechanism.
But we have to remember that when it comes to sampling information, really, the role of-- we really are asking about assigning value to a photon on the retina. And the main function of that photon on the retina cannot really give us biological rewards in and of itself. It can't really sustain life in and of itself. What it can do for us is reduce uncertainty. So that is really the specific function of that photon on the retina is give us information or reduce uncertainty.
And so this leads to a very complicated set of questions about what is the role of reward versus uncertainty reduction in information sampling. And it's, again, it's complex because the two quantities are correlated in natural behavior. And because also, we have no studies at the physiological level.
So I am trying-- nevertheless, so what we do know so far is from modeling behavior is that uncertainty does seem to play a role above and beyond-- above and beyond increasing reward probability. So let me first illustrate this in an intuitive way. If you remember this example here that I've shown you before.
So all of these, there are a lot of things here that are rewarded and associated with reward. If you imagine what the hand does, so the whole-- the block moves, and the hand motion, of course, is rewarding, and so on. So all this trajectory here is associated with reward, because it's necessary. If you put in a reinforcement learning model, you would assign value to all the movements and all the trajectory.
But notice that nobody is tracking their own hand movements. Nobody is looking at this thing as it moves through space. Even though that is a rewarding trajectory, it has value, it has instrumental value. So why is that? How do you explain that? Well, because that's probably redundant. And it's not informative. So you have little uncertainty about that trajectory.
The points of uncertainty are here, the points of contact. So this is sort of a general intuition of why you may need another quantity in your models in addition to reward to explain eye movements. How is uncertainty-- is uncertainty coded in the brain, and how? And this is a big, big, big, big question.
If you ask economists, it's actually interesting. If you ask economists or if you ask people in finance, they give you two different answers to this. So in economics, they say we do not need a separate parameter. We do not need to encode, to specifically use a measure of the variance of a distribution.
Why? Because we can get uncertainty preference or risk preference just with nonlinear utility functions. So all we have to assume is that people have some preference or utility for an income. So here, if your income increases, your utility would increase. But this doesn't have to be linear. And if you have a convex utility function, then imagine that you are in an uncertain situation, where you can have outcome A of 10 or outcome C of 30.
So imagine that situation versus you have a fixed outcome B of 20. Which is exactly average. If you have a convex utility function, you were you will value this risky lottery more. You would say, I'd rather get A or C, because its average value is somewhere here. Because of the convexity, it's superadditive. It's nonlinear.
So the value of this uncertain lottery they can pay 10 or 30 will be higher than the value of the safe lottery, which has the same expected value. So convex utility function is the classical way, in economics, to produce any sort of uncertainty preference. And you don't have to have an explicit measure of uncertainty. It turns out, that people who do finance models, they do put explicit terms for uncertainty in their models. I don't know why. But this is the more sort of academically established way of thinking about it.
And the similar debate goes on now in reinforcement learning. Reinforcement learning models, I'm sure some of you know this better than I do, try to do everything with rewards. And they can do amazing things with just rewards. Some information sampling data, though, these models have a lot of trouble with.
And they also-- people working in this field also bend over backwards to avoid introducing another term related to uncertainty. But you have to come up with more and more convoluted reward functions to explain the data. What do we know about this in neurons and behavior?
So the little data that are coming out now suggests that there are neurons in the brain that explicitly code uncertainty. But this is very often coupled with an additional signal of reward on top of it. So here is one example of this. This was in a group of neurons in the basal forebrain. So they're neurons deep in the brain that are close. They're not cholinergic neurons. But they're close to nuclei that contain acetylcholine.
So what the monkeys-- the task here was very simple. The monkeys just were fixating. And they saw different cues that signaled to them what is the probability of receiving a reward on this trial. So here is-- so here are the cues. So for example, this cue told the monkey, 0% reward, 25%, 50%, 75%, or 100% probability of a fixed drop of juice.
And then there were also cues that told them the signal different amounts of reward with 100% probability. And these were chosen so that the average expected value of these pairs of cues were equal to each other. And this is what the neurons did. So the neurons showed a function like this. On the x-axis here, for the black trace, is the probability of reward. And for the gray trace is the magnitude of reward. So such that the same points here are matched for expected value.
So you can see two things. First of all, there is a linear trend here. Responses kind of go up as a function of probability. And that happens even for these cues that have, they are shown in gray, that have fixed the 100% probability. So this linear trend is driven purely by the magnitude of the drop that is being promised.
But then there's a second component. And there is this inverted U shaped component, which peaks at 50%, which is the point of maximal uncertainty. And here, if you subtract those two curves, you can get the pure uncertainty response. So this sort of linear combination of uncertainty and expected value seems to come up in various places. And I'm going to end with showing you this very similar effect that we found in eye movements in monkeys who are sampling information-- who are searching for information.
These types of studies are done by people who are interested in value, and motivation, and subcortical structures, and so on, and so forth. It's not really attention related. But they are really the type of value functions that I would like to have in my attention system in order to tell it where to sample, and when to sample. So we did a study in monkeys, first to see whether uncertainty-- how uncertainty and reward bias eye movements.
So I'll show you the behavioral results. We're doing the neural recordings now. So maybe next year or later, I can talk about the recordings. But I'm not going to talk about them now.
So in this study, I cut out the details of the task. But here's just the logic of the task. What happened is we told-- we first signal to the monkeys the probability of reward on that trial using some cues that they were trained on. So some of these abstract patterns, and some patterns signal no reward on the trial, 0% chance. Some said maybe 50% chance. And some said 100% chance of reward.
So the monkey saw the cue the very beginning of the trial. And then there was a delay period. And then they received the reward according to that probability that was signaled. And there was really not-- this was not an [INAUDIBLE] task. There was nothing that the monkeys had to do in order to get the reward. It is more Pavlovian. It's just the signal and then the reward arrives.
But during the delay period, we gave the monkey a display containing several placeholders, like white patches. And if the monkeys wanted, they could search through the display to get additional information. So there were three white patches. And if the monkey looked at one of them and kept their eyes on it, the patch flipped and revealed what was underneath.
And one of the three patches had an additional cue. This is what we call cue 2. And cue 2 always gave perfect information. So cue 2 could tell you either it's 100% chance or a 0% chance of reward. Now, the monkeys, they didn't have to look for cue 2. And in fact, you'll see, we had a monkey who decided that's not worth his time. And he didn't look very much.
Wanted to get, really, to how curious are they. Do they really want to get more information? And we reasoned that if this wasn't-- if we were designing an information sampling system, like an eye movement, that system should only kick in here. If the first [AUDIO OUT] uncertainty, then the is getting information by searching for that, because he can now be certain of what he's going to get.
But if the first cue already is informative, the second clue is redundant. So whether it says 100% or it's a 0%, the second cue is redundant. And you shouldn't bother sampling. So what we predict is a perfect inverted U shape with sampling only at 50%.
So here's what they did. In fact, I can show you-- So what I'm going to show you here is on the x-axis is the prior probabilities, signaled by cue 1. Then I'm going to show you the number of samples, the number of placeholders that the monkeys look at and the probability of finding cue 2. And we're going to see this for two monkeys, M1 and M2. So number of samples and probability of finding the cue are highly correlated.
But here is our inverted then skewed U shaped function. So what you see here is the highest sampling is, indeed, at 50%. So that's cool. That's reassuring. The amazing thing is that we also have high sampling at 100%. So in other words, if I tell you for sure you're going to get a reward, you're still willing to work to get a perfectly redundant signal just because it's cool, it's good, it's good news.
Notice that here, I told you also for sure something, but it's bad news. And you're not willing to get a redundant signal if it's bad news. So that curve, I mean, you can see, it looks just like those subcortical neurons, some combination of uncertainty and reward.
You can see that these are the sessions. You can see that this pattern is very stable. Highest sampling at 50%, 100% and 0% is the lowest. So here's one monkey. You're asking about looking off the screen. This monkey sampled very little.
You can see this monkey sampled up to two placeholders on average. This guy started at one and was downhill from there. So he figured out, this is a perfectly rational strategy, he figured out that he's not required to sample, so why should he? So in most trials, he looked up, up to the upper left of the screen, which is fine. But luckily for us, when he sampled, he showed the most sampling at 50%, then 100%, and then 0%.
So one thing that I wanted to note is-- you to note, though, is this seems to me that this reward effect on information sampling seems to me, specifically this point here, the fact that this point is so high, it's not optimal. And this thing is going to get you into trouble. Because think about it, you have an organism that devotes resources to useless stuff.
This cue 2 here is entirely useless, because it brings no information, and it brings no more reward. The reward is the same anyway. So if you have an organism that is willing-- that is sort of biased in its information sampling in this way, that's willing to expend effort on signals that are just pleasurable, but not really useful, you can get into trouble.
So anyway, all right, so on that note, to understand attention, I think we really need to start thinking of vision as an active, motivated process, tie the loop together with decision making. And then I've shown you a couple of steps toward that. Information sampling is a knowledge driven process. We have to understand how the meaning of stimuli is related to our ability to select them and attend to them.
It's partly motivated by uncertainty reduction, but also shows some funky biases due to reward. And I think that these kind of biases, I think, are really going to be important in understanding all kinds of decision biases and strange behaviors that we seem to show. Yeah? Yes.
[APPLAUSE]
have an interactive transcript feature enabled, which appears below the video when playing. Viewers can search for keywords in the video or click on any word in the transcript to jump to that point in the video. When searching, a dark bar with white vertical lines appears below the video frame. Each white line is an occurrence of the searched term and can be clicked on to jump to that spot in the video.
