PRESENTER 1: Welcome back, everyone. It's a great pleasure to introduce Professor James DiCarlo. He's the head of the Department of Brain and Cognitive Sciences. He was trained by Kenneth Johnson and John Maunsell.

And he contributes what I think is a perfect example of the marriage of the three levels of analysis that Tomy Poggio was alluding to-- combining computational work, investigating the neural circuits at the neurophysiological level, as well as building computational models and studying behavior and cognition. I don't want to take any time away from James-- so please, Jim, go ahead.

JAMES DICARLO: OK. Thank you, Gabriel. And thank you, Tommy, and the other organizers for organizing this summer school virtually and for giving me a chance to speak a little bit about our science but also to try to give you a sense, as Gabriel said, of how the spirit of Brains, Minds, and Machines from CBMM-- I think all those three ares can be brought together to the benefit of all of them.

Again, I think that's what's quite exciting about CBMM. And I hope that's why many of you are listening today. To that point, my first slide-- why are you here to listen to my talk? Or why are you here at CBMM?

I'm going to let our first department head for Brain and Cognitive Sciences, Hans-Lukas Teuber, motivate you a bit-- and what motivated me into the field. So you're going to listen to some audio here. This is Hans-Lukas Teuber in 1974 motivating a question that I'm going to talk a bit about today.

HANS-LUKAS TEUBER: In a way, we could say that the first step in any new field is to know what to be astonished at-- what to wonder at. A very, very gifted theoretical physicist by the name of Murray Gelman who was here on a visit from that strange place on the west coast, Caltech-- you know, our competitors. We tried very hard to persuade him not to go back there, but we failed.

Gelman said to me, without any provocation on my part, that he thought ours was a weak-kneed department, rather than being unorthodox, as I claim, because it had in it neuroanatomy and neurophysiology and linguistics and other things that are usually left out. It was not bold enough, because we were not studying those issues in psychology that were really important. I said, which ones are important?

And he said-- and he absolutely floored me by saying that, because I think he's bright. I know he's bright. He said, well, you shouldn't study perception. You should study extra-sensory perception. You should not study language, but thought transference and hypnosis.

There was one third thing, which I have forgotten or repressed, as some people say. It might come back under hypnosis, maybe. But in any case, I was astounded and shocked because he said all this in front of our then-provost-- fortunately, a very good friend of mine.

I tried to convey to him the idea that we think to wonder about is perception-- that if extra-sensory perception did exist, I would feel somewhat let down. I'm not against its existing, but I would feel deflated, because it would be much easier to understand.

Perception is the great riddle. How do I see a line as a line and a face as a face? And as that face amongst the 1,000 faces? Where emotion and memory enter-- that's a much greater riddle.

JAMES DICARLO: OK, so I'm in Teuber's shoes now, as the Head of Brain and Cognitive Sciences. But even back then, motivating what seemed to be something almost, at the time, seeming simple-- which is the problem of perception-- is the thing I wanted you to take from that clip. That is one of the things that we should be wondering and then be astonished about. Indeed, visual intelligence and visual perception is one of the most astonishing things that the human brain is able to accomplish. And I'm going to tell you a bit about one aspect of visual intelligence today.

And again, Teuber mentioned other things-- language and other things as well. And I think you'll hear about those in other talks. But today, my talk is going to focus on an aspect of visual intelligence, which is really motivated by answering the kind of nouns that are out there in the world, which is things like, what is in this scene? Where are the cars? Where are the people?

So these are objects of the world. And being able to pick those out of pixels and to say they're either there or not there. And where are they, exactly, in the scene? Those are fundamental questions in visual perception and computer vision. And we call them and put them under the umbrella of visual object recognition.

There are more complicated questions, like, what will happen next? And where is it safe to walk? And so forth. And these are questions that you'll hear from other speakers during this course. But I'm going to tell you about the first three-- these core visual object recognition questions. Just basically, what's out there?

OK, before I do that, I wanted to-- since the CBMM talk-- to orient all of you, that many of you are coming from computer science or AI backgrounds or you're interested in a an AI approach to things. So what you're trying to do, I think-- and I think I'd put myself in this category-- is we'd all like to build some silicon-based what we call computational intelligence that meets or exceeds the biological or human intelligence. We'd ideally like that to be a good model of the system-- or maybe even, again, exceed the system. So you have lots of choices if you're going to take that as your goal. And many of you out in the audience I think might take this as your goal.

First of all, you could decide, as a strategy, to ignore brain sciences and just proceed as straight-up engineering and do your best. And I don't criticize that approach. But I don't think you'd be here at this Brains, Minds, and Machines course if you thought that that was the right way to go. But I want to point out there are other things you could do.

You could ignore the brain sciences and what you actually do day-to-day, but you could talk about your systems as brain-inspired or brain-like. I'm not a big fan of that approach. It's really good for advertising and PR, but it's like taking the branding of the brain and the mind, but not actually using the data to constrain what you're doing. So I encourage you to think beyond just stamping things with a brain-like seal of approval and move beyond that to actual data that are measured from either behavior or neurons or other brain activity.

You could use human performance as a benchmark to report your progress. That is a bit better, because you're using the constraints of human performance. And so you're moving in the right direction as you do that. And many of you may be thinking about that as you think about that work you want to do beyond the CBMM course.

If you're a scientist-- this is an opposite approach. Coming from this background, you might say, well, let's start with some very simple reduced neural system and hope that some principles of intelligence will be discovered. And that's motivated by the codes of DNA being first worked out in simpler systems, like viruses. And then the idea that you're going to scale up later, but first you're going to work on principles.

Again, a lot of people take that approach. And I don't think that's a bad idea. But I don't think it's going to be enough for what our project is here.

Now, this is, again, my opinion-- not facts. But I'm just giving you lay of land here. I'm going to contextualize what we do as what we call forward engineering systems within wisely-chosen brain science measurements. And again, looking at the data to test deviations from those measurements as you build those systems. And adjust your system-building if the deviations are growing rather than shrinking.

So we're trying to build better and better approximations of what's actually going on in the biology to accomplish the intelligence tasks that interest-- in our case, visual intelligence tasks. And I refer to this broadly as reverse engineering, as a theme of approach. And a bonus of this approach, relative to some of the approaches above, is that it enables advances in human health, education, and brain machine interfaces and other applications, to name a few.

And by the way-- and I think Tommy knows we'll be first to point this out-- there's a small bonus that if you work on this, this is one of the greatest open science problems in the history of our species. How the biology gives rise to intelligence and consciousness and so forth. And that's really a motivating thing beyond just building good, say, AI systems.

OK, so that is, to me, under this umbrella of reverse engineering, which I also sometimes describe this way-- is that our goal is to account for the ability of the mind-- say, visual intelligent behavior-- using systems of components in the brain-- for instance, connected neurons-- in the language of engineering and predictive-built system.

So this is where science and engineering meet around what we'll call algorithms, which to us are specific in-silicone neural networks that actually work. So they can do tasks that engineers would like them to do and they can explain the measurements of both behavior and neurons that you find within the mind and the brain. And they're motivated by both engineering and science. And their building is to the benefit of both fields, both as hypotheses for brain and cognitive sciences and as actually applications for engineering.

So it's quite exciting right now that engineering and science are coming together. And again, CBMM represents that intersection. And I'm going to give you an example of how this science merging with engineering has really been blossoming in the area of visual intelligence-- or really, visual recognition.

And when I say we, I'm mostly referring to work from some of these folks down here. This is my current lab. And I'll try to highlight them along the way. But they deserve all the credit for things I'll show you. I'm just an ambassador of their great accomplishments.

So when I started building a lab at MIT around 2002, we said, let's set about trying to reverse-engineer some aspects of human visual intelligence. And these are some of the aspects that I've already referred to in the earlier slide-- what's out there in the scene. And to do that, we said, let's reduce the problem to something manageable. So rather than how you absorb that whole scene and think about that whole scene as a scene understanding problem, we focused on how the primate brain extracts object identity from that scene.

And the first thing we applied is the notion that you, as a primate, do not just absorb the whole scene at once. You have high acuity at the center of the gaze-- so if this red dot is your fixation point, this is illustrating, say, the central 10 degrees when you have high acuity vision. And that's the portion of the visual field that we've been analyzing. So that part of your retina that processes the central 10 degrees, which a lot of your brain is devoted to. And the way you make these analyses of the larger scene as a whole is by making eye movements around the scene with fixation durations shown by the dots that are on the order of a couple hundred milliseconds.

So just a split second at each of these dot locations, as an example-- so-called scan path. I'm telling you all this, because now, instead of us thinking about a whole scene analysis problem, we can think about what you do in each of those glimpses. And I'm going to now show you those short fixation glimpses here as a sequence of frames. Now, if you just fixate in the middle here while I play these glimpses for you. Those are 10 degree cut-out glimpses of the scene. I'll play that one more time.

I hope that you can realize, as I show those, that you're not identifying all problems of visual intelligence in those brief glimpses. But you're able to identify things like, what is out there? Cars, signs, people. You're also able to identify things about the pose and position of those things within those glimpses as well.

So this reduced problem focuses on question. You can answer questions like, is a car here? Is a person here? And so forth. This reduced problem we refer to as core object recognition.

And again, that's just an operational definition of a 10-degree viewing window with about a 200-millisecond viewing duration. So this, as you see, is a building block for vision and visual intelligence, but it's not all the visual intelligence. But that's what I'm going to focus on today so that we can make progress on our understanding.

OK, the way we test this behaviorally is that you might be brought into the lab and you might look at a scene like this. And you're asked to fixate the dot. And then an image might come up like that one. And I hope that each of you can recognize that what was in that image was more likely a car than a bird. And you would indicate to the right a choice of a car.

Now, this is a little bit of a tangent here to remind you that the reason the problem of "recognition" in general is hard is because the light coming off the car never strikes your eyes in exactly the same way. So you have what's called identity-preserving image variation. So the cars can be in different positions, sizes, poses, illuminations-- and have background clutter and so forth. All of that stuff that you have to deal with or be tolerant to or so-called invariant to is what the visual system is robustly able to generalize to. It's been a long-known core problem that the visual system has solved.

And so I'm just pointing that out to you to say, when we test you or other test subjects in this-- or animal species, like primates, that I'll show you in a minute-- we're careful to test not just very central objects but also wide degrees of variation. And again, the position, size, pose, and also the background.

And we test both synthetic images, like the ones I showed [INAUDIBLE], when you have 3D-rendered objects on complex backgrounds, as well as naturalistic image photographs taken from the real world. And we test both of these along the way. So I'm saying that we have context to give you a sense of how we do experiments. And then we can test subjects like this again, so here you go.

What did you think? That was a synthetic object-- more likely a face than a car. Here's another one-- OK, more likely a bird than an elephant. These are pretty easy examples.

OK, there is a natural image. That's a photograph-- more likely a bird than an elephant. So these are the kind of tests that we do. We do them, as I mentioned, on humans.

Many humans that we test online to get your ability to do these-- and you're not perfect at it. You're good, but not perfect. And your pattern of errors is actually quite useful to guiding us. But you are much better than machines-- at least machines were around 2010 or so.

And that's what's shown on this plot. Your performance on the y-axis and your ability to report the object identity accurately as performance as a function of the variation in the object. So you're very good, even when you have high uncertainty in the object, [INAUDIBLE] shown to the right side of this plot-- and much better than machines were at the time. And that regime there, where you're doing good generalization across these variations, is what we call core object recognition.

So again, this is all by way of set-up to say, we study this problem of visual recognition in primates and humans. And in my lab, we study it also in rhesus monkeys, because we can gain access to the neural hardware at a much finer grain than we can in humans. Here is a monkey doing the task that I was just asking you to do a few slides ago. The monkey is triggering the screen to present a test image that you see here in the middle.

That 100-millisecond presentation is followed by a choice of one or two choice objects. You see there's multiple different types of choice objects coming up. This animal knows on the order of 30 objects or so. It takes them about a day to learn each new object.

And they're doing it in high variation conditions to the center of the animal's gaze, as I set up for you. Black means he gets it wrong-- so he doesn't get them all right. Green is he's got the trial right. So you see he runs a lot of trials this way. And we can collect a lot of data about the monkey's behavior-- and as I mentioned, the human behavior.

And just as a shortcut, I'm going to tell you that those studies basically reveal that you, as a primate, are equivalent to the rhesus monkey as a primate in your ability to discriminate among objects in the conditions that I just described for you as core recognition conditions. That doesn't mean monkeys know what a car is in the sense of being able to drive it away or that they can use it to get from point A to point B. It just means that they can distinguish a car from a truck as well as you can distinguish a car from a truck. And they make the same kinds of errors that you do. So this means that we have a very good, almost quantitatively accurate model inside the monkey brain of our own brains in doing this kind of intelligence test.

And again, by the way, these primate systems are much better than computer vision systems, certainly back in 2010-- and they're still actually better than computer vision systems at doing these things. And so we still have a lot to learn from the primate biology about how it's doing better than computer vision system. But that big picture-- primates, equal to each other, greater than computer vision systems-- is what I'd like you to take from this.

OK, as I mentioned, we studied the rhesus monkey as a primate model because we can access that brain and we know much more about it. So decades of neuroscience have provided measurements of the macro and meso architecture of the non-human primate brain. And I'll show you some of those areas here.

So now we're switching exclusively to the rhesus monkey. These areas in color are what we refer to as the ventral visual stream. It's a series of cortical processing areas that I'll outline for you. The highest one of those purely visual areas-- it's mostly visual-- is called infratemporal cortex, or IT cortex. And we're especially interested in IT cortex in the ventral stream that leads to IT cortex, because it's long been known that lesions in this part of the brain cause deficits in tasks like core recognition.

Just to orient you, this is not processing of visual data just to go nowhere. The parts of the brain the ventral stream project to areas involved in decision and action and also to regions in the medial temporal lobe involved in memory and value judgments. We think the ventral stream is processing the visual data to good forms to be able to explicitly encode things like objects to guide, again, decision and action and to lay down long-term memories-- just to orient your other talks that you'll hear in the CBMM course. So here's this ventral visual stream. It's a series of visual processing areas in the cortex.

Remember, it's preceded by the eyes, of course, which are capturing the data in the back of the eyes, in the retinal ganglion cells-- down here is RGC. The thalamus, which lives in the middle of the brain that I'm not showing on the slide-- this is the lateral geniculate nucleus, the LGN. And then these series of cortical areas-- V1, V2, V4, and IT.

And each of these dots within each area is meant to signify a neuron. There's, of course, millions of neurons in each of these areas. And I'm just showing a schematic here to give you a feel for things.

There is both feedforward connections and feedback connections, as well as recurrent connections. And this is a very simplified ventral visual stream diagram, just to orient you to the basic dominant neuroanatomy of the ventral stream. And so when you think about the function of the ventral stream, the way we usually start off conceptualizing this is you have an image like this one, which is a pattern of light that might be striking, again, your central 10 degrees of your retina. And then that is then converted from light photon energy into patterns of firing in the back of your eyes and your retinal ganglion cells. So now spiking neurons in the retinal ganglion that transmit various spike rates.

And here I'm showing high-firing neurons in black and low-firing ones in white as an example, just to schematize this. Transmit this down your optic nerve, to your lateral geniculate, which then transmits it to V1, to V2, to V3, and to IT. And you have this cascaded flow of neural activity with different neurons firing at each level of the system, indicated here by colored dots. So you have an image shown on the eyes that produces a pattern of neural firing in each of these areas, up to this IT cortex that I outlined for you.

It takes about 100 milliseconds to go from the image to the firing at IT cortex. And the key thing is that the images on the retina, they're basically nice photographic copies of the image. They're like what your camera would capture on your smartphone. But then your brain is transforming these so that the neural pattern up in IT cortex looks nothing like a photograph anymore. It's some special pattern of neural activity that has made the object identity much more explicit than is on the retinal cells themselves. And I'll show you more about that in a minute.

But you'll get the basic idea that you now have a new image leads to a new pattern in IT cortex. And I can go back to the old image and it evokes basically the same pattern of activity in IT as when I showed it the first time. So these patterns are reproducible when you study them with electrode recordings that I'll show you in a moment. But also I want to point out that your visual system can easily follow along at timescales like this one.

These are images shown about one per 100 millisecond. And you can already notice, if you look at this image yourself, this movie that you can identify one or more objects in that movie as it comes by, similar to what I showed you at the beginning, with the natural scene. And your visual system, your ventral stream can follow along quite nicely-- again, with a lag of about 100 milliseconds-- producing a new pattern for each image at the top in IT cortex.

So since this is one of your first CBMM talks, I want to remind you just a little bit of neuroanatomy. This is a rhesus monkey. As I've been showing you some behavioral data and the lay of the land, neurally-- this is what a monkey brain looks like after it's been fixed and taken out of the skull. So you can get a feel for what a monkey brain looks like. And I'm showing you that if you then cut this brain and look at the neurons, this is what you might see-- the layers of cortex all folded up here.

This is in the visual area of V1. I'm not going to do a neuroanatomy lesson here, other than to remind you that there are multiple layers within the cortex. Each of these little dots that you're seeing here stained with this missile stain-- is an actual neuron within the cortex.

If you stain it in different ways-- this is a Golgi stain. And you see all the dendritic and axonal processes associated with a subset of those neurons. So now you see the neurons as these pyramidal bodies here, with all these dendrites and axons coming off of them. And what we do in our experiments that neurophysiology in general does is lower microelectrodes into the tissue where they're close enough to some of individual neurons that they can record the spiking patterns from individual neurons. And that is the workhorse method of neurophysiology.

And the basic idea, the reason that we'd really like to record these spikes is that these spikes are what transmits the information through the neural network from one set of neurons-- say, in a visual area, V1-- to another set in V2. So these spiking level activities of neurons are a very privileged level of information, because it's the only way that neurons can communicate at the speeds that you can do things, like vision. And so even though there's many things you can measure in neurons-- calcium molecules, other things-- those things aren't fast enough to do vision. So the spikes are what we're most interested in modeling-- what the spike activity looks like within each cortical area leading up to behavior. That's why it's a special level of information.

OK, so that's a bit of neuroanatomy and context for why we study spikes inside the monkey brain. Here's back to the visual ventral stream. Remember, you're getting different images producing different patterns of activity up in IT cortex. And as I mentioned, you can lower electrodes and record from individual neurons in IT cortex.

This is an example recording from IT. And these are patterns of neural spikes out of this one individual neurorecording site within the monkey brain, in response to four images that you see on the top. Each tick mark is a firing of an action potential out of the neuron and each row is a different trial where we're representing the image. So just to orient you to tell you what's going on here-- when we present this first image, you can see that it always gives rise to an increased firing rate with a lag. You see the timescale at the bottom here of about 100 milliseconds after the image comes on at time zero.

The fire rate of the neuron increases from a background firing rate that you see with these sparse fights to this higher firing rate. And it does that on every one of these 10 trials we've shown here. Here's another-- the second image, you see the same thing. An increased firing in a response to this image. So we sometimes say the neuron likes these two images and it doesn't like these other two images as much-- the firing rate doesn't elevate as much.

So this is just an example so you can get a feel for the elemental neural data. Here's another site in IT cortex. It doesn't like this first image, but it likes the second one. It's the second, the third, and the fourth. And here's a third site which doesn't like either of the first two, but likes the third and the fourth.

And I'm showing you this to remind you that there's a diversity of neural response types within IT. And these are examples of just random recordings within IT cortex and showing you how the neurons respond to different images. And I'll show you in a minute-- how the neurons respond to these images is actually one of the great mysteries, I think, we've been working to solve. But I'm just giving you a feel for the kind of data that you get when you record from monkey IT.

OK, now instead of us tracking these individual spikes out of neurons, what we are doing here is we make an approximation. So we average the spiking activity in a time window-- say, in this case, I'm showing you a time window that is on the order of 100 milliseconds long. And we average over the repetitions here, different presentations of the image-- assuming things that are roughly the same. So we're averaging out the trial-by-trial variability and we're averaging across time, ignoring any spine temporal dynamics. Later, we return to exposing some of those interest points.

But for now, I'm just telling you how to summarize the data so that we end up with one number out of each image. So we can say this site in IT likes this image this amount-- 60 spikes per second are the units of these measurements. The average spiking rate of this neuron to this image-- average spiking rate of this neuron to this image is 71 spikes per second here at 25-- here's 7, as examples. So now you can understand how we go from the elemental recordings to what we actually are going to tell you about modeling when we model systems like this one.

And we don't record just neurons one at a time-- we're typically in our lab recording around 300 simultaneous sites by implanting chronic arrays, like these ones shown here, which have about 100 electrodes each. And they live in the animal's tissue, in his cortex, for multiple months so that we record thousands of images over those time frames and stitch together large data sets this way.

OK, now I'm going to ask you to do a little bit of a mind flip in how you think about these data. Instead of us thinking about one neuron responding to many images, I'm now asking you to think about many neurons in IT cortex responding to one image. So here's this one image at the top, on the order-- this is actually about 168 neurons shown here. And the green color indicates the spike rate out of each neuron to this particular image.

So again, the units are spiked per second. And you have this big, long response vector. That's the length of the number of neurons that you were recording. And this is meant to be a sample estimate of what IT as a whole is doing sampling from IT.

Again, we're not recording the millions of neurons in IT. We're recording on the orders of hundreds and trying to make inferences of how the system works by recording this in IT or in other areas, like V4 or the other visual areas. OK, so this is one so-called response vector, because it's a population response where we keep track of the individual responses of each neuron.

Here's eight different images. And we don't record just eight but, as I mentioned, thousands of images. And we get large data sets that look like this one. We have thousands of images and hundreds to thousands of neurons.

And by the way, if you're interested in modeling these kind of data in your CBMM projects, we have some online and we're happy to share these with you and share more with you to do projects in this space. So one of the main messages I'd like you to get from the talk at this point, beyond the background of the monkey visual system, is that these patterns in IT, as I mentioned earlier-- these population patterns that we can measure, they're very, very special.

And the reason that they're special is that if you apply linear decoders onto these populations and you test those decoders and their ability to generalize to new tasks, like I trained you to identify a dog and then I test you on new images of dogs-- they generalize quite well. In fact, they generalize just as well as the animal generalizes-- the animal being either the human or the monkey-- which, remember I said they're equivalent in the recognition performance.

So when you just put linear decoders on these kind of patterns-- and we've shown this on a number of papers, some of which are listed here-- these are quite powerful, in the sense that they generalize across those hard image variation problems where I showed you the car varying earlier in the talk. And so this is all by way of background, in some sense, to tell you how we think about vision in the monkey for object recognition. And from a computer vision or AI point of view, the fact that you and I have these codes computed in our head that are in IT cortex that we can measure in a monkey, that's why our brains are better than machine systems at being able to do tasks like recognition-- because we compute these codes. Because once these codes are computed, linear decoders explain the rest of our behavior.

So in some sense, you can think of shifting the problem away from explaining your behavior to explaining this almost magical code up in IT cortex. As what these kind of studies taught us early on is that there's very special codes up there that are very privileged in that sense that linear decoders can be applied to them.

So this is a point where I want to pause and take questions, because that is basically a big set of background on vision to get you to the newer stuff over the last decade or so that I'd like to tell you about, if we have time. So Chris, if you see questions or if someone can field for me, that would be helpful. I should be about halfway through the talk now, maybe a little more.

PRESENTER 2: Sounds good. We have a question from [INAUDIBLE]. The question is, how do you derive the recurrent network architecture in the visual system? How do you find the evidence?

JAMES DICARLO: So I think that's a question about neuroanatomy. So at the bottom, I'm showing this feedforward and feed-recurrent network. And that anatomy, I didn't unfortunately have time to put that into our talk. The connections between the different areas are established by making micro-injections in animal brains, euthanizing those animals, and then tracing by various methods how one area connects to the other. And those tracings for the rough ventral stream are at the level of giving you area-by-area connectivity, not neuron-by-neuron connectivity.

How we draw diagrams like this tells you one area is, on average, strongly connected to another without giving you the exact precise wiring diagram-- which is still a topic of great investigation at the moment. So I think that's my short answer to the question. It includes both recurrent and feedforward. I think the question was about recurrent.

And I should say, by the way, what I've been telling you about is a feedforward view of the ventral stream. The recurrence is very important to the processing. And if we have time, we'll talk about that at the end. But you've got to start with the first order model, which is the feedforward model. And then you can start to add in the recurrence.

That's how we think about approaching the problem. But good question. Thank you. Others, please.

PRESENTER 2: Great, thanks. We have one from Nathan Corneal. The question is, do you think spikes as units of information transmission can be useful from an engineering perspective for machine learning, where currently floating point numbers are usually the unit?

JAMES DICARLO: Yeah, so that's another interesting question. So as I mentioned, when we model this, we take the spikes and we convert them to mean fire rate. So now they're analog numbers. I think there are lots of deep questions that I'm not going to talk about here today about why spikes might be useful from an energetics perspective, relative to current compute [INAUDIBLE]. And I think that's the spirit of the question.

And maybe if the questioner is interested, we can have some more of that discussion offline and I can connect him to some references. But for the talk today, we're going to ignore the individual spike timings and just focus on predicting the mean firing rates. And one thing I'll add to that is when I mentioned that the IT codes explain the behavior, that means you can take the mean firing rates without regard to the individual timings of those fights and explain the behavior with linear mechanisms at that point.

So that's part of the reason that we believe that mean firings is a good level approximation of the information processing. But the question contains hints of it and elements of not just information, but also about energetics rather just information carrying that I think are interesting with regard to spikes. But I think that's all I say at this point.

PRESENTER 2: Great, thanks. The next question is from Guy Gasiv. You have shown reproducible responses to images in IT. After how many trials does adaptation occur? How is adaptation avoided or how is it addressed?

JAMES DICARLO: Yeah, so that's another good question-- though I did say the responses are quite stable. I hope you noticed when I showed you these examples of the responses, they're not stable at the individual spike level. So the mean rates you can think of as being stable is the way we model it with stochasticity on top of it. Sorry, Chris. Can you say-- there was something more to that question that I think I was going to get to.

PRESENTER 2: Sure. So the two main parts were, after how many trials does adaptation occur? And then how is adaptation avoided? Or how is it addressed?

JAMES DICARLO: Yes, I forgot the word adaptation. So adaptation generally means a change in response over some timescale. So there's many timescales of adaptation here. And when they're called long enough, then they're referred to more as learning. And of course, throughout the brain, you can find all of the timescales.

And the way we study IT is we generally ignore the short-term adaptations that you'll get trial-by-trial that are reduced when you go from a very dark room, for instance, to a light room. Just the light transient will cause that. And the way we wash that out is we average over many presentations that average out those short-term effects.

And again, basically we can check if the population vectors look similar at the beginning of the experiment versus the end of the experiment. And to the extent that they do, we say the system hasn't adapted. And so we treat it as a non-adapting system that at that point. And that's generally what you see, unless you apply very specialized statistics or other kinds of learning paradigms that can change the IT responses.

So here you should think of these IT responses as essentially stable, with adaptation effects that we washed out for the moment. That's a short answer to a very complicated question. And I hope that's enough for now. Anymore, Chris? I should maybe move on to get through the rest of the talk.

PRESENTER 2: Yes. So we have about 24 minutes left. And plenty of questions that we can get to at the end if you have time.

JAMES DICARLO: I hope all the meat of this talk is maybe you're now motivated by the ventral stream is an interesting area, a place to do projects for CBMM projects. Data are available if you want to work on many of these kind of questions, many of which are still open. But I want to orient you to the things we've been working on the last decade or so is, even though I told you that IT has these powerful, almost-- I use the word magical-- codes, that of course shouldn't satisfy you if you're an engineer. Because it just says we can measure them in the brain.

And of course, they need to live in the brain somewhere because the brain supports behavior. But how they're computed from the pixels is the really interesting question to us. So how do you go from the pixels up to what we measure in IT? Or how is the IT code computed? Or that is, how are the neural responses derived from the pixels themselves?

And I've already given you a hint of that. Of course they don't just go from the pixels up to IT, but they go through all this neuroanatomy. So there's intermediate transformation stages that need to be understood to get up to IT cortex. So that's another way of phrasing the same kind of question. What are the population patterns of activity along the way, along the ventral stream?

And here, I'm speaking of these questions all at the mean rate level-- the timescale's 100 milliseconds or so, as I introduced. In our lab, we considered a bit of a breakthrough that we had around 2013 or so. And I'm going to tell you about that today, just to give you a spirit of how that worked out and what we think it tells us and what it doesn't tell us yet. And just so you know context here-- as neurophysiologists, as I mentioned, we've been recording for decades in these areas, like V4 and IT.

My lab's not the first to make these measurements. But it's always been mysterious as to what makes these neurons do what they do-- that is, what aspects of the images drive those neural responses? And so you can do things like this-- you could plot the neural response here to a bunch of images that I'm showing here.

Here, they're organized by category. This is the mean firing rate, as I mentioned earlier-- over a 10o-millisecond timescale. And you see these fluctuations in the response to different images.

Just to be clear, this is not time on the x-axis-- these are just different images, grouped by category. And you see this neuron for some reason tends to like images of chairs-- some of the images of chairs, but not all chairs. So it would be a mistake to call it a chair neuron, but it has this average preference for chairs.

But it's really hard to eyeball these data and say, what makes these neurons do what they do? You need a mathematical or computational model to try to take this on. Same thing where these are famous neurons in IT called face neurons, because they respond more to faces, images of faces, than to other, non-face objects. But you can see, they don't respond to all images of faces and they sometimes respond to images of non-faces higher than some images of faces. So it's more complicated than just saying it's a face neuron.

And so I've been here in area V4-- remember, which is a mid-level visual area. The responses even look less categorical. So here, you see some of the top firing images of this neuron. And here, you wouldn't even dare to call it a car neuron or a chair neuron. It's some kind of image features that are driving these neurons.

You have a complicated pattern of responses, because this is just one example E formula. And I would say to all of you that if you want to say you've understood a system, that we need to explain the neural mechanisms of, we must at least be able to predict these behavioral responses. That doesn't mean that the only thing we need to be able to do, but we need models that can at least accurately predict these responses. And that's what we've been after is trying to find neurally mechanistic models-- that is, neural networks that can capture these neural responses. Not as the end of understanding, but as the next step in understanding.

And we've been following a long line of very important work. And I think Tommy probably introduced some of this in his introduction, because his lab was a key driver of a lot of this background work-- is that there's a lot of background from brain science that led to this style of modeling where you have this feedforward cascade models with local processing copied over the visual field and a stacked-like system. They build a feedforward approximation of what's going on from the brain data, the type that I alluded to, or outlined for you earlier. And that led us over decades to many types of models.

Some of them I've listed here-- models of V1, which you might know as [INAUDIBLE] filter-like models. Models of the whole ventral stream-- there's HMAC family models built by Tommy and his collaborators. My lab built some of these models-- we called Klaus O9 models. These are examples of all-- you can see feedforward cascade-like models motivated by brain science.

The trouble we were having is not that we didn't believe this family of models. It's just that none of them were really adequate as scientific hypothesis because they couldn't explain the data well. So they all failed to accurately predict those complicated V4 and IT responses that I just took you through. So we were inspired by this class of models, but they weren't quite right.

So the reason we knew that is we could take those V4 neurons like I showed you and we could ask, how well can you predict the responses to new images? That's what I'm showing on the y-axis. This is for cortical area, V4. Predicting for new images-- and this is noise-corrected, so you've already adjusted for the reproducibility of the data.

And you can see that it's really pretty poor. This red line shows a model trying to predict the responses of the neuron in black that I showed you a minute ago. And you see, it doesn't fit very well. You can see that that eye is just quantified on the left-- that we're not doing very well. Here in IT, also we're doing even slightly worse with some of these models.

So would you call this a failure? We're not there. We didn't throw out the hypothesis class. The problem, we thought, was that there's a lot of parameters in these feedforward models that are not determined by the existing science results-- the neuroscience results. As I mentioned from the anatomy question, we don't know all the details about how things are wired up and what the weight values are on the synapses and so forth.