Functional Imaging of the Human Brain: A Window into the Organization of the Human Mind
Date Posted:
July 27, 2020
Date Recorded:
June 17, 2020
CBMM Speaker(s):
Nancy Kanwisher All Captioned Videos CBMM Summer Lecture Series
Loading your interactive content...
Description:
[The video is missing the first few minutes of the talk due to technical difficulties.]
Nancy Kanwisher, MIT
NANCY KANWISHER: So you see a higher response to faces than anything else right there. That's the fusiform face area that responds selectively to faces. OK. Right next door is this little region in green that responds for the same set of stimuli very strongly to scenes, not at all to faces, and weakly to everything else. That's the parahippocampal place area that likes places. OK.
Over in the left hemisphere as this particularly cool little patch in yellow. This region responds strongly when you look at visually presented words and letter strings, and not when you look at-- or much less when you look at faces and bodies and scenes and objects, and other stuff.
And so what's cool about that is that humans have only been reading for a few thousand years, and that's not enough for natural selection to have crafted a special circuit for just that process of reading. And that means that since we find this specialized response here in the bottom of the brain, that bit of brain must have been wired up by this person's individual experience learning to read, not by the ancestral, evolutionary experience of their ancestors. So that suggests that this person's experiences have wired up that selectivity. Doesn't tell us how these bits got wired up, a question we will return to later in this talk.
OK. Out on the lateral surface of the brain, we have a lot of other good stuff. So over here, that little purple bit responds more to images of bodies than anything else. And over here on the left temporal lobe, that orange bit responds to the sounds of speech.
And importantly, one of these bars is when you're listening to your native language and the other bar is when you're listening to a foreign language you do not understand. So this region's not interested in understanding the meaning of a sentence. It's interested in processing the sounds of speech, like the difference between a ba and a pa, OK?
So right now, you're doing both those things. You're analyzing the sounds you're hearing, and you're also, let's hope, understanding the meaning of the sentences that I'm uttering. That little orange bit is just for processing the sounds of speech.
OK. So all of these things I've talked about so far are involved in analyzing different perceptual problems for recognizing faces visually, scenes, bodies, digital words, and hearing speech sounds. So you might think, maybe we only have specialized machinery in the brain for aspects of perception. Do we also have them for higher level cognition?
Well, Ev Fedorenko, who's a faculty member in our department and was my postdoc a bunch of years ago decided to ask that question for the case of language. So she figured out how to identify these language regions that I showed you actually in the first slide. They've been known for 200 years from studies of brain damage, but only very blurrily.
She showed with functional MRI you can see my lab tech David-- that's his brain here-- you can see these red bits right here on his temporal lobe and his frontal lobe that responds selectively when you read or hear sentences compared to everything else. OK. So what Ev wanted to know is, are those regions really selected for processing language, just as the face, place, and body regions are selected for processing faces, places, and bodies?
So to find out, she scanned subjects while they did a whole bunch of different tasks. And the details don't matter. She included basically all the tasks that people had thought might overlap with language in the brain like arithmetic, like music, like holding information in working memory, and like a whole bunch of other things.
And so she scanned people. She had already identified their language regions here by scanning them on language tasks. And then she measured the response of those language regions to all these other tasks. And the question is, do those language regions do all that other stuff or not?
And the answer's pretty simple. Here's the region, a language region in the back of the temporal lobe, sometimes known as Wernicke's area. Here's its response when you read sentences and in light gray when you read strings of non-words. That's meaningless things that look like they could be words, but they're not actual words.
And here's the response on all the other tasks. Big bunch of nothing. So what that shows is that this region is very selectively involved in processing language information.
AUDIENCE: How come some of them are upside down?
NANCY KANWISHER: What are-- oh, the bars?
AUDIENCE: Yeah.
NANCY KANWISHER: That just means they go down. So it's a good question. What does zero mean? So with functional MRI, a given number, you scan somebody, you get some MRI magnitude number from some voxel, some 3D pixel in their brain, and it'll be 647. And that's meaningless. Doesn't mean anything at all.
So the only way to interpret MRI data is to compare two conditions, two or more conditions, and say, which one is higher? So in these experiments, we usually have a rest condition, which is ridiculous, because what would that mean? You tell people, OK. Don't think now. Turn off your brain. For the next 20 seconds, no cognition. Nothing. We can't do that.
So instead, we just had people stare at a dot, and we just let them rest. And whatever happens, happens. But at least they're not focusing on one particular task we're asking them to do.
AUDIENCE: So the baseline is staring at a dot?
NANCY KANWISHER: Yeah. So zero is staring at a dot. And this blue thing just means that-- what is that? That's a spatial working memory-- a difficult spatial working memory task. It says that maybe you have slightly less activity in that region when doing the difficult spatial working memory task than when staring at a dot, OK?
Now you can also see the error bars are huge, so that's not even a significant difference. But if it were to become significant, that's what it would mean. OK? Does that makes sense?
AUDIENCE: Yeah.
NANCY KANWISHER: That's a good question. I'm glad you asked. OK. So does everybody get how this is showing really pretty extreme selectivity of that region for language? It's not engaged in all these other things. And I will say that lots of other brain regions respond to all of these things, so we know that they can strongly drive the brain. It's not that they're crappy tasks. They just don't activate these regions.
Similarly, if we look up here in the frontal lobe, this is basically an area called Broca's area. And it does the same thing, higher response to sentences than non-words, and pretty much almost no significant response to any other condition.
So what's cool to me about these data is it's not just, oh, we get to put another selective blob on the brain. That's always fun. I do like that. But I think this is a particularly fun blob, because it tells us that language and thought are not the same thing.
And probably every one of you has wondered about this at some point. Like, do we think in language? Do we need language for thinking? What is up with that? And I think here, we have a very concrete answer from the brain. It's telling us that the parts of your brain that are engaged when you understand the meaning of a sentence are just not engaged when you do all kinds of other thinking.
Like arithmetic and holding stuff in memory and appreciating music, and lots of other kinds of thinking just don't use those regions at all. So language and thought are not the same thing in the brain.
But maybe the coolest, wildest, freakiest part of the brain is that little pink bit right there that's been studied most extensively by our own Rebecca Saxe. And she showed that that region is selectively engaged when you think about what other people are thinking. Now if that seems nuts to you, reflect on the fact that you do this all the time. Every time you have a conversation with somebody, you're taking into account what they already know so you can decide what you need to tell them.
Right now, as I'm lecturing to you, I'm trying to figure out, OK. These guys have lots of technical background. Do I need to say this other stuff? Do they already know that? What's in their heads? Are they following me? All this kind of stuff. And that is kind of what it means to be a human being, is to just spend a lot of your time thinking about what other people are thinking.
It is the essential ingredient in novels. We don't have novels that are about mountains or inanimate objects, not just because they don't do much. You can have a novel about a tornado, but it wouldn't be very interesting if there weren't people involved. What we care about is what is going on inside other people's heads. That's what human beings care most about. And so it turns out we have a specialized region of the brain for processing just that.
OK. So what's cool about Rebecca's work is she has spent lots of effort showing that that region is not engaged in processing what other people look like or even their bodily sensations like thirst or hunger or pain. It's very specifically involved in thinking about the contents of their thoughts, which is remarkable.
OK. Having told you all this stuff, I want to say that I don't think this is true of every patch of the cortex. Not every little patch is doing something hyper specific like the regions I just described. In fact, those white bits up there are sometimes called the multiple demand regions because they respond to almost any different kind of difficulty or demand. So pretty much any difficult task you do, those white regions kick in.
And so here's some data showing that. Here's a whole bunch of tasks where they filled in bars with the difficult version of the task and the light bars are the easy version of that task. So basically showing that across a wide swath of tasks, those regions are turned on more when you do something harder.
AUDIENCE: So I was wondering, when can fMRIs show inhibition as opposed to just the activation of neurons? And is there some kind of baseline that you'd base that off of?
NANCY KANWISHER: Right. It's a great question. So I skipped over what functional MRI's actually measuring, so let's do just 30 seconds on that. So functional MRI is actually looking at changes of the oxygenation of hemoglobin in the blood in your brain. And because we have fine capillaries all over the brain with a really dense blood flow supply, we can look at that.
And it turns out that if you use a little piece of your brain, like some little piece up there, you turn on the neurons there because you're doing some task using those neurons, turns out that the blood flow increases to just that part of the brain. Just like if you go running, blood flow has to increase to your calf muscles to supply them, metabolically because it's metabolically costly to run. It's also metabolically costly to have neurons fire.
So blood flow increases to that region, and that changes the concentrations of oxygenated and deoxygenated hemoglobin, OK? And so those two are magnetically different in a way than an MRI machine can detect. So that's the basic signal.
So now if you think about it, it's actually looking at metabolic cost measured by way of blood flow. OK? So because it's metabolic cost, it's possible that inhibition is also metabolically costly. And so it may be that the activation we're seeing is actually suppression of some neurons in those regions, right?
So there are many, many ambiguities in functional MRI data, and an early researcher in the field pointed out mostly as a joke, but it's technically true, that that face area, that could be the part that really sucks at face recognition, right? That's why the neurons are there, going, err! You know? Right? Or it could be the part that gets shut off where there's active suppression of those neurons.
Now I'll show you other data to suggest that that's not the case here, but your point is exactly right, that when you see a response with functional MRI, it's very ambiguous. And in some cases, it could reflect inhibition rather than excitation. OK? You had a second question, which I forgot now.
AUDIENCE: Yes. So this is just more for a curiosity, but you talked about a region that tries to process other people's thoughts. And I was wondering, you know, in people with things like anxiety or mental disorders that are known for quote, unquote, "overthinking," have you seen maybe like an overactivation of these areas?
NANCY KANWISHER: Yeah, that's a good question. I don't know the answer to that question. I'm not sure people have looked at that. But another thing you might wonder is, what does that region look like in people with autism who famously have relatively specific deficits in understanding other people? Yeah?
And so the surprising result is-- so Rebecca Saxe and many other people have looked at this. And first of all, you have to scan quite-- to answer the question, you have to scan quite high-functioning people with autism, because if they're non-verbal, they're not going to cooperate with your task. You're not going to be able to explain to them what to do. So she needs people who can basically do her tasks that require thinking about other people's thoughts. And that's already a subset of people with autism.
But if you scan those people and say, OK. Yes, they gain those abilities much later in development than typically developing kids, and maybe they're a little worse at it, but they can basically do it. Now do they use the same machinery, or have they solved that problem in some totally different way?
The surprising answer is they use the same damn machinery. They get exactly the same magnitude of response. It's in the same place. It looks the same in every possible way except for one very obscure difference, which you could ask me about at the end of the lecture if you like. But surprisingly, it looks really similar.
And so there must be-- but again, what we see with functional MRI is just that it's there, and just basically, when the neurons there are firing. That leaves wide open the question of what exactly the neurons are doing and what they're computing and what they're representing and how they're doing it, and all of that. But at least the same basic architecture is in place.
AUDIENCE: Thank you.
NANCY KANWISHER: Sure. So these multiple demand regions, they just turn on whenever you do anything difficult, unlike all the other regions that are very particularly and only turned on when you do one little thing. All right. So that's my whirlwind overview, highly idiosyncratic featuring lines of research I've been involved in and am a fan of.
But what you can see even from this overview is that we've made real progress. I really like this picture, because this picture of this set of functionally distinct regions that are present in every one of us that includes regions that are highly specialized for abstract, uniquely human functions like language and thinking about each other's thoughts. I like to think of this diagram as a kind of initial sketch of who we are as human beings, who we are as thinkers.
But at the same time, this is just the barest beginning. This diagram is like a thumbnail of a theory of brain organization, not an actual theory, because it leaves out a huge number of questions. And the way I like to think of it is just a roadmap for the questions we ask next.
And so what I'm going to do is consider a few of those questions. What is the causal role of these regions in behavior? What other specializations might exist in the brain? How do these things get wired up in development? And then there's a million other questions, but I'm going to focus on these three, starting with this one here. All right? And I probably won't get through all three, but we'll just do what we do.
All right. So what do I mean by saying, what is a causal role? Well, first, let me remind you that in science, causality is of the essence, right? Scientific theories don't just say stuff like, well, when this happens, that other thing tends to happen at the same time. Like, no. That's no good. We want to know, did this thing make that thing happen? Did this thing make that thing happen? Did something else make them both happen? We need to know what is causing what in science. That's the essence of the scientific theory.
And here's the bad news. Functional MRI will never answer that question. It's a drag, but we should face reality. It's not going to answer that question. So that means that if we want to know not just, OK, these regions tend to turn on when you do these tasks, we want to know, are those regions necessary for those tasks? And to answer that question, we need different methods. OK?
So a few years ago, I got a phone call from a colleague. And he told me that this gentleman here was awaiting neurosurgery in Japan and had already been implanted with electrodes all over the bottom of his brain for clinical reasons so that the neurosurgeons could plan the surgery they planned to treat his intractable epilepsy.
And so they said, we have electrodes all over the bottom of the brain, and the patient has agreed to allow us to record data from those electrodes in the few days while he's awaiting neurosurgery. Did I want to collaborate? Like, duh. Of course, I wanted to collaborate.
So for comparison, here's the bottom of my brain showing you in red my face selective regions, in green my place selected regions, in purple are regions of my brain that respond more to color information than grayscale. And you can see by comparison that it looks like those electrodes are going right over some of the interesting stuff back here.
So we sent some stimuli to Japan quickly, and they sent us back data from this strip of electrodes here. So each little graph is the response of one of those electrodes over time. This is two seconds on the x-axis here. And this is a kind of measure of mean neural activity measured electrically at each point.
And so what you see-- oh, and the colors are different kinds of stimuli. So you see a bunch of electrodes here that respond a whole lot to faces and pretty much not at all to anything else.
Look at that selectivity. Much better than the selectivity I showed you with functional MRI before, because functional MRI is based on blood flow, which is pretty good, but blurs together adjacent brain regions. And here, we have electrical responses from just one little two-millimeter patch of brain, and it's extremely selective. So the first cool thing about this is that these data validate what we see before with functional MRI.
But the cooler thing is the neurosurgeons wanted to know about the causal role of those regions. I think one of these neurosurgeons had accidentally cut out that bit and made the patient unable to recognize faces. So I think he was very wary of that and wanted to make sure he knew which of those regions were causally involved.
So they said, OK. They're going to stimulate right there electrically through those same electrodes and see what the patient says. So let's see what happens. Whoops. Here we go. OK. So you're going to see-- here's the patient looking at the neurosurgeon, and this shows you that that [INAUDIBLE].
[VIDEO PLAYBACK]
- [SPEAKING JAPANESE]
- One more time.
NANCY KANWISHER: OK, so this guy has no idea where [INAUDIBLE] right there, right in this [INAUDIBLE].
- [SPEAKING JAPANESE]
NANCY KANWISHER: OK, so that showed that that region's [INAUDIBLE]. We're asking, did that region causally [INAUDIBLE]?
- [SPEAKING JAPANESE]
- No change.
- [SPEAKING JAPANESE]
NANCY KANWISHER: [INAUDIBLE]
- [SPEAKING JAPANESE]
- One more time.
- [SPEAKING JAPANESE]
[END PLAYBACK]
NANCY KANWISHER: All right. You get the idea. He doesn't know where he's being stimulated. He's just asked to report if anything changes. He doesn't even know that there's a fusiform face area. He doesn't know where the electrodes are. He's just looking at those things and reporting what he sees, OK?
So the point of all of this is that shows us that that region is causally involved in face perception. It doesn't just turn on when you look at faces. If you disrupt it, you can actually create a percept of a face, right?
And further, it shows us that it's selectively involved. When he's looking at a box or a ball or a character, it doesn't change the shape of the box or the ball or the character. It just puts a face on top of it.
So now our next question is, OK, but maybe anything you do, maybe you hit your head or something. It makes you see a face. Maybe anything you do makes a person see a face. So what if you stimulated right next door in a region that prefers colors? We show that both in his brain, and you can see it's also true in mine. So what happens when he's zapped right there? Well, let's see.
[VIDEO PLAYBACK]
NANCY KANWISHER: Again, he just asked [INAUDIBLE].
[END PLAYBACK]
Oops. I screwed it up. Let's start it again.
[VIDEO PLAYBACK]
- [SPEAKING JAPANESE]
[END PLAYBACK]
NANCY KANWISHER: Pretty wild, huh? Scientifically, I mean, it's just wild to see. Scientifically, what's important about this is it shows that those regions are not only activated by but causally involved in the relevant function. And those causal roles are very specific. You zap one region, you get a face percept. You zap the color preferring region, you get a rainbow. It's like I wrote the script, but I didn't.
[LAUGH]
AUDIENCE: I have a question. So I was confused in the first video. So when he is looking at a face and then they zap him, he sees a distorted face? But when he's looking at something that doesn't have a face, he sees a face?
NANCY KANWISHER: Exactly. Exactly. And so we wouldn't have necessarily predicted exactly that, because we don't know-- basically, when you stimulate, you're basically activating probably tens of thousands, maybe hundreds of thousands of neurons all at once in some area. So that's a very crude sledgehammer. You're taking that whole chunk of brain and saying, OK, guys. Turn on a whole bunch of stuff here.
So you can imagine how, if you had a representation of the face and the pattern of response across those neurons before you stimulated, then you throw in this big signal, you disrupt it. You change what the face looks like. Whereas if you're looking at something else, you make a net face signal, right? Didn't have to be, but given that that's what happens, it's consistent with the idea that those neurons are-- activation of those neurons is sufficient to create a face person. OK?
AUDIENCE: I have a question. So I don't know if I completely understand the stimulus itself, but was there a lot of research into quantifying if different stimuli have different effects on what the patient saw?
NANCY KANWISHER: Yeah. You mean different kinds of electrical stimulation.
AUDIENCE: Yeah. Or like, if it was in the same region, maybe if we stimulated by 20% more, perhaps it could have a different effect.
NANCY KANWISHER: Yes. That is possible. And in this study, they stimulated at just two intensities. I think it was four milliamps and eight milliamps. And with four milliamps, there was just a weak version of the same thing. With eight milliamps, that those were the data I showed you, right?
So there's plenty more room to ask all kinds of more sophisticated questions, like suppose you stimulate at different frequencies or in slightly different locations. And with this case, when you get data like these, these are extremely rare and precious to get these kind of data with humans. Monkey labs, you just go in and do this to a monkey anytime you like. It's up to you, right? But here, we only get to piggyback on what the neurosurgeons have decided to do for clinical reasons anyway.
And so it will be wonderful to explore all that stuff in rich detail, but it's kind of unlikely anybody will get to do that. Yeah? So it's kind of wide open, the degree to which it depends on all kinds of other things, because we just don't get enough data to find out.
And you can try stuff like this on monkeys, but the real advantage of doing this in humans-- with a monkey, you would have to train the monkey to do a task like, did it change? Or does it look like this? You could ask the monkey a specific question. You spend months and months training the monkey to be able to answer this question.
But then if something else happens, the monkey's going to say, well, you know, this thing you asked me didn't happen, but this other cool thing happened you might want to know about. Monkey's not going to tell you that. A person will tell you all of that, right? So there's a real advantage to doing this in people, which you can see just watching the video.
AUDIENCE: I had a question. Thank you. So do you think then that this sort of appropriate direction of using fMRI with its own limits, like, affect the connectivity being really difficult to do, that it would be appropriate to use fMRI to find that functional segregation of areas like the fusiform, and then use things like that where you actually go to that human part of the brain and then stimulate it to see? Like, you use function connectivity to sort of localize where you think that is more likely to be the case where it would be a face area, and then you stimulate it?
So then my question would be then-- because that seems like a really good path, but obviously, doing that on humans is just really, really rare. I mean, this time it was possible because of that specific scenario. Do you think that would be more common, or do you think it's still going to be very rare to use--
[INTERPOSING VOICES]
NANCY KANWISHER: Well, OK. So me--
AUDIENCE: Because it seems like a really good way of doing things, but obviously--
NANCY KANWISHER: It's awesome. In the very rare cases where we get to do this, we're like, oh my god.
[LAUGH]
But it's very rare. So I think many people in studying the human brain have realized that these intracranial studies are where it's at. It's a very delicate business, because you don't want to go overboard and pressure patients. Here they are in dire straits. We're really lucky when they're generous enough to let us run one or two experiments and record from their brains. But they're not feeling good and they're undergoing a really scary and horrible situation. And they're very vulnerable, so you have to be very careful not to pressure them.
But many people actually, I think, really appreciate the opportunity to do something scientifically useful. And after all, they're hanging out in the hospital for a week anyway, basically waiting to have a seizure with nothing much else to do. It's not that awful to look at our stimuli while they're at it, right?
So there are many efforts to increase access to these kind of data. And we're very lucky at MIT, because we have a new collaboration that's just been started with a new neurosurgeon at Mass General Hospital, Mark Richardson. He's awesome. He's the nicest guy ever, and really excited to collaborate with us.
And so he is now working with us, and we think we'll have much increased opportunity to get these kind of data. At least the recording data, if not the stimulation data. Stimulation data are even more rare. But we'll get some of that.
So I think with this kind of work, you just have to be patient. And so you develop many other lines of research, and then in my lab, if there's a patient with electrodes in their brain-- usually we hear about it the day before-- everybody drops everything and focuses on that, because it's just so exciting and so special.
AUDIENCE: OK. Thank you. And so is transcranial magnetic stimulation too big of a-- it's not as good as that electric?
NANCY KANWISHER: Yeah. So there are many, many methods in human cognitive neuroscience. And if you guys want to learn more about this, go to my brain site, Nancy's Brain Talks. I have loads and loads of lectures there on all the different methods of cognitive neuroscience.
But the simple story is there's lots of them, and each of them has a slightly different niche, right? And so the ideal thing to do is just for any scientific question you're asking, you want to figure out, what is the right method to answer that question?
So TMS, Transcranial Magnetic Stimulation, is great, because it's a causal test, right? Like electrical stimulation, like studying patients with brain damage. But unlike functional MRI, TMS is a causal test. But it has many other limitations, like it's not very spatially specific. It's not terrible. It's good for maybe two-centimeter resolution. Something like that. So it's very useful.
But it has tiny effect sizes. It can't reach deep in the brain. And it has other limitations. So basically, to answer any question fully, you need all the methods in human cognitive neuroscience.
AUDIENCE: Thank you.
NANCY KANWISHER: Sure. OK. So we're going to run out of time, but that's OK. I mean, actually, if I get through a little bit of this next part, then any of you want to hear more can watch the talk, can email me. I'll send you the Zoom link to the talk I gave a few weeks go, and that will actually just build right from here, so that will work great. OK.
An obvious other question when you look at a picture like this is, OK, what else? We've got speech and visual motion and bodies and reaching and words, and what else, right?
So one of the things we do in my lab and that lots of people do in other labs is to ask that question. And so one paper we published a couple of years ago that I'll just mentioned the results-- if you're interested, I can send you the paper-- is my then postdoc Leyla Isik said, well, one of the things we humans do is not just look at other people's faces or listen to their language. We look at groups of people, pairs of people, and we try to figure out, are they interacting with each other?
So she scanned people and showed that a patch right about there in that kind of dark teal, turquoisey color, responds selectively when you see two people interacting with each other compared to two people acting independently. So that seems to be specific for perceiving third party social interactions. That's pretty wild and cool.
Another thing we've been looking at is these set of brain regions here in purple approximately. This is work in collaboration with Josh Tenenbaum. And Josh has been saying for a number of years now-- and it took me a while to get what he was talking about. Often, I go through this trajectory with Josh where he'll be making a whole big point, and I think, what planet is this guy on? And then after a while, I hear it a third, fourth time, it's like, oh. Oh, he's really onto something there. And that happened here.
And the point is that when we go around in the world, we do lots of interacting with other people. And so a lot of these brain regions are social. They're involved in seeing or hearing or understanding other people.
But we also have to understand the physical world. You can't take a single action on the world-- you can't pick up a pen or dial something on your iPhone or get up from the chair or walk across the floor without having some kind of mental idea of the physical basis of the world you're acting on. How heavy is the pen? How frictiony is its surface? If I grab this phone, will it slide right out of my fingers? Is it squishy? No, it's not. You know, how exactly will this floor support my weight? All that kind of stuff.
So we published a paper a few years ago in collaboration with Josh with preliminary evidence that these brain regions here may be involved in understanding the physical world. It's very early days in that line of work. We're doing other things to try to build on it. Anyway, it's ongoing work.
But something actually I'd love to fit in if I can before I run out of time is some studies that I did a few years ago with Josh McDermott on auditory cortex. So let me tell you a little bit about that, because they're quite different from the other things I've talked about.
So here's Josh. This was done by Sam Norman-Haignere, who was our joint graduate student. And when we started this work, almost nothing was known about the organization of auditory cortex. It was known that all this stuff on the top of the temporal lobes here is involved in processing auditory information. But beyond that, we just didn't really know much.
So we said, OK. Let's take a totally different approach to trying to understand how auditory cortex is organized. Instead of sitting around and making up hypotheses like, OK, do we have a bit that's good for voice recognition? Let's test that. Do we have a bit that does this? Do we have a bit that does that? I mean, there's nothing wrong with that. I've been doing that for a few decades now.
But who's to say that the things that we think up are the things that are going to be true in the brain? Maybe the brain has its own ideas about how to organize itself that are things we would never think of. So this is a study that tries to use the data. You collect a lot of data, and you try to let the data tell you what the structure is instead of going at the brain with a specific hypothesis.
OK. So we scanned people while they listen to lots of different sounds. Basically, all the basic categories of sounds you hear frequently in daily life. Stuff like this, OK? 165 different two-second sound clips. We scanned people listening to each of these.
We then measure the response of each 3D pixel or voxel in the brain to each of those 165 sounds, and that gives us a big old data matrix like this. OK? So now, here are our actual data. Very compressed, so it kind of blurred out. But here's our 165 sounds. And across 10 subjects, we have voxels appear in the auditory cortex of roughly 1,000 per subject, and we just stick them all together in this data matrix. So the brightness of each pixel tells you how strongly one of those voxels responds to one of those sounds, OK?
So now, we have this big matrix. And now, we just do some math to basically say, what is the structure inherent in this matrix? And if any of you have had any linear algebra, this will be basic to you. This is a variant of principal components analysis called independent components analysis. It's actually a variant of independent component analysis, but that's close enough.
And if you don't know what that is, don't worry. It's just a bunch of math that enables you to take a matrix like this and say, tell me what its basic structure is. What is the structure hidden in here. OK?
And so what's cool about this is it's completely neutral to any hypothesis about the locations of particular things in the brain. This analysis doesn't even know which voxel is where. They're just all thrown together in the matrix. And it doesn't know which sound is which. It's just math. Pure math, OK?
And so what we do is we find that just six components can explain 80% of the replicable variance in this matrix. So what that means is we have just six response profiles of the sounds, each of them paired with a set of weights across the cortex. And if you take those six, you can reconstruct basically all of the structure in this matrix that's replicable.
And that doesn't mean there are only six kinds of neurons in the auditory cortex. In part, it's a statement about how impoverished functional MRI is. But it's pretty cool. The data are just telling us there's basic structure.
And so now we can say, what are those components? And to find out, what we can do is we can stick the labels back on and look at them. And the first four that we see are kind of expected properties of auditory cortex. For example, one of them is a selective response to high frequencies. Another is a selective response to low frequencies. And that replicates the long-known tonotopic organization of auditory cortex.
Basically, there's a map of auditory frequency laid out across the auditory cortex, and we rediscover that long-known map with our hypothesis-neutral methods. And that's great, because it tells us these methods can discover things we already know to be true. That's like check, check. Good. And that makes us more likely to believe the new stuff it discovers.
OK. So there were two new components that were not like anything we predicted, and here's one of them. So this is now the response of one of those components inferred mathematically from that matrix to each of the 165 sounds. Now those sounds are color-coded by the basic category of sound over here.
So if you look at that, you say, hmm. Look at those green things. If we then average within a category, you can see that that component reflects high responses to speech sounds, whether English or foreign.
So just like I told you at the beginning of the lecture, there's this patch of brain that isn't about processing the meaning of language, because this region responds the same to both English and foreign speech. Rather, it's responding to the sounds of speech. OK, what's this blue bar? Oh, that's vocal music. So there's also speech sounds in there, OK?
So this component that emerges from the math and the data, without our going in with a hypothesis is showing us that there's a specialized aspect of the brain that respond selectively to speech. So that's pretty awesome. There was actually a little bit of prior evidence that this might be true, but our evidence is much stronger with all these sounds here. We can show how strongly selective this response is. OK. So that's cool. OK. I just said that.
OK. Then there was component six. And component six was a real surprise. So if you look at component six, you say, oh, there's all this blue and purple stuff. What is that? Well, if you average by category, you see that component six is selective for music.
The dark blues up here are the instrumental music sounds. And you can see they're all interweaved with the music sounds that have vocals in blue. And you can see there's almost like a drop off a cliff. Like, all the music sounds are high, and then you go off a cliff. And actually, some of those orange ones there are things like cell phone ringtone, which was categorized as not music, but actually kind of is.
[LAUGH]
OK. So this is also-- wait, sorry. Before I say how awesome that is, each component also has a set of weights in the particular voxels in the brain. So we can also say, OK. Where are those components primarily located in the brain?
And this one, this is a kind of close-up of the top of the temporal lobe. This is a temporal pole, back of the head. The white, black and white are high-low-high frequencies. That's primary auditory cortex. This is that tonotopic map I mentioned before.
And what you see here is the speech component is just below it. OK? OK. The music component is just in front and behind that part of the auditory cortex. So this is awesome, because now we have what's known as a double association of speech and music in the brain. That means we have a part that does speech, not music, and another part that does music, not speech. And that shows that those things are really separate in the brain.
And that's great, because a lot of people-- why we have music in the first place has been this longstanding mystery in the field. Nobody knows. It's clear why we have to be able to recognize faces or understand language. We need those things to survive. But why do we have music? Why does every human culture have music? Nobody knows.
One of the theories has been that natural selection has built in speech and music machinery in the brain. But once you have that, you can then co-opt it and do other stuff with it and use your speech machinery to process music. And this says, nope. That story doesn't work, because the speech and the music machinery are completely non-overlapping in the brain. OK?
I'm almost out of time. OK. So when you find an awesome wild result like this, your first reaction should be, really? Let's try to find it another way. Just as somebody asked a little while ago, when do we use different methods? We use all the methods we can get access to to try to reinforce what we're finding.
And so we were very lucky to be able to work with [INAUDIBLE] here to collect intracranial data-- just recording data, not stimulation data-- from the temporal lobe in patients undergoing neurosurgery while they listened to our sounds. And so what we found is without doing any of the fancy math of factorizing that matrix, just looking at the responses of individual electrodes, here's, for example, one electrode.
This is time here, because now with intracranial recording, it's electrical, so we have time information. You can see this electrode responds selectively to speech. The two green curves are the time course of the response to native and foreign speech. And pink, I believe, is vocal music. So that's a speech selective electrode, consistent with what we found in functional MRI.
And here is a music selective electrode, vocal and non-vocal music, consistent with what we inferred from functional MRI. But with intracranial recording, we actually also found something new that we had not seen with functional MRI, and that is song selective electrodes. So these are electrodes that are so specific that they don't respond to any music. They respond only to vocal music. Isn't that awesome?
OK. So I am running out of time. So what I'm going to do is skip the whole second part of the talk and go to my last slide and say that we've argued that some of these regions are causally specifically involved in different aspects of cognition. We're still discovering new bits like this music bit here.
And the part you didn't hear about development shows that we know there's a role of experience in development. For example, that visual word form area I described, it's only selective for visually presented words after you learn to read, not before. And in data I didn't show you, we think it lands in the same location across subjects because of preexisting, long-range connectivity of that region to other parts of the brain.
And there's just a whole host of other wide open questions that weren't part of this lecture and that are wide open, like, how does all this stuff, all this intricate structure, the same in every one of us, get built up over evolution? How did it evolve? What exactly is represented and computed in each region?
As a little advertisement, I used to think this was the coolest question and mostly out of reach. But now I think with the advent of deep neural networks and the ability to compare them to brains, we're starting to make new progress here. I'm almost done. What is the exact connectivity of each region to the rest of the brain, and how do they work together to produce intelligent cognition and behavior?
And my favorite question that's also in my recent talk if you're interested is, why is this a good way to design a brain in the first place? If you were an engineer, would you build a computer with all these specialized bits? Well, we've been using deep nets to ask that question, and ask whether the particular specializations we see in the brain are for mental functions that have to have their own machinery, because if you try to get one network to do all those things together, it can't do all of them.
have an interactive transcript feature enabled, which appears below the video when playing. Viewers can search for keywords in the video or click on any word in the transcript to jump to that point in the video. When searching, a dark bar with white vertical lines appears below the video frame. Each white line is an occurrence of the searched term and can be clicked on to jump to that spot in the video.
