The inferior temporal cortex is a potential cortical precursor of orthographic processing in untrained monkeys [video]
Date Posted:
August 6, 2020
Date Recorded:
August 6, 2020
CBMM Speaker(s):
James DiCarlo ,
Kohitij Kar Speaker(s):
Rishi Rajalingham, Stan Dehaene
All Captioned Videos Publication Releases
Loading your interactive content...
Description:
Rishi Rajalingham, the primary author, and others from the team that published the paper in Nature Communications describe their findings and the impact of their research.
[MUSIC PLAYING] RISHI RAJALINGHAM: When you and I read, we look at these tiny scratches on a page, and we make a lot of sense out of it. The recognition of letters and words from these tiny scratches on a page-- we refer to that as orthographic processing. Orthographic processing is the first critical step of reading as a whole.
Interestingly, reading and writing are relatively recent cultural phenomena. Humans have only had a writing system for a couple of thousand years. The majority of human beings have only been literate for a couple hundred years.
And so it's very unlikely that the human brain has evolved completely new mechanisms in order to support this relatively recent cultural phenomenon. And so what we think must have happened is that existing representations, the existing mechanisms that supported other behaviors, may have been co-opted or recycled in order to support reading and orthographic processing.
STANISLAS DEHAENE: In my lab, in the last two decades, we've performed a whole series of brain imaging experiments in order to map out the brain circuits for reading. We tested the theory that when you learn to read, you actually reconvert or recycle areas of the brain that are involved in object recognition in all primates but that become specialized for recognizing visual words. And in order to test this idea, we've scanned illiterate versus literate subjects, and we've mapped out all the circuits that change when you learn to read. We found that the acquisition of reading is accompanied by the specialization of a particular region, actually, in the left hemisphere, here in the left occipitotemporal gyrus. In illiterate people, this region cares about objects and faces, but in people who have learned to read, this region becomes specialized for written words.
However, given the limitations of brain imaging in human subjects, it's not been possible so far to understand how this region encodes, at the single neuron level, the written word. And, of course, this is due to the fact that we cannot access single neurons except in very rare cases of patients with epilepsy. So we were inspired by behavioral research which suggests that you can study reading particularly in non-human primates.
There was a beautiful study in baboons that showed that they too, just like humans, can learn to accurately discriminate written strings of letters and discriminate words from pseudo-words in English. Of course, these monkeys have zero understanding of the English words, so this discrimination ability is purely based on the shapes of letters and the statistics of letters. But it does suggest that it's not completely inappropriate to ask if monkeys could provide an opportunity to study the neural code for letters and letter strings and to compare that to-- with human behavioral results. So this is what we did. We looked for the neural coding of letter strings in the monkey brain.
KOHITIJ KAR: To test this IT precursor hypothesis, we performed large-scale neurophysiological measurement in rhesus macaque monkeys. We implanted four monkeys with two to three multi-electrode arrays across the ventral visual cortex, specifically areas IT and V4. From each area, in principle, we can record from around 96 neural sites while the monkey fixate at a central dot and we present hundreds of orthographic stimuli on the screen. By averaging each site's spiking activity over a fixed temporal window across multiple image repetitions, we can generate what we call the population response vectors per image. Now, essentially this becomes a neural data set that we have analyzed further.
RISHI RAJALINGHAM: With this neural data set in hand, we could ask, do these neural populations in IT and V4 cortex contain explicit information about word forms? And to test this, we used linear decoders. These are simple models that learn patterns of activity over this neural population in order to support a task, for example, to discriminate between words and non-words.
We found that decoders trained on IT cortex activity were able to achieve high performance on many of these tasks. In fact, they were able to match the performance of baboons on the discrimination of words and pseudo-words. In addition to matching the overall performance of baboons in this task, IT-based decoders could also capture the pattern of errors over different images. In other words, the images that were most difficult for baboons were also the ones most difficult for IT-based decoders.
We then repeated all these analyses in an upstream cortical area, V4, and we found that V4 neither matched the overall performance nor the pattern of errors of baboons on this task. And so this is something selective to IT cortex. So within IT, we found that the neural code that supported many of these tasks was highly distributed over the tissue, rather than specialized to specific letter coding modules. And so what this suggests is that the set of complex invariant visual features that we've studied for so long in the domain of object recognition might actually form a common substrate for learning to read.
JAMES DICARLO: This work is a collaboration in the truest sense. Stan Dehaene and I began talking about the underlying question several years ago at a scientific conference. And it was Stan who really inspired me to start thinking about how our work on the neural mechanisms of visual processing of objects might be relevant to the neural mechanisms of how humans learn to read written text. Without that motivation, this work would not have happened.
But that was still not enough. What really pushed us along was a Gordon conference I attended with then PHD student Rishi Rajalingham. Rishi and I connected with Stan to talk further about these ideas, and he jumped at the opportunity to carry them forward. Rishi took the ideas and ran with them in our laboratory, enlisting the help of further collaborators to help execute the experiments.
And the results are quite exciting to me, as I open up a potential linkage between our rapidly developing understanding of the neural mechanisms of visual processing and an important primate behavior, human reading. The results also speak to the importance of open science and collaboration to push these new frontiers and, really, to the amazing ability of graduate students like Rishi to take risks and to try new things.
Associated Research Module:
have an interactive transcript feature enabled, which appears below the video when playing. Viewers can search for keywords in the video or click on any word in the transcript to jump to that point in the video. When searching, a dark bar with white vertical lines appears below the video frame. Each white line is an occurrence of the searched term and can be clicked on to jump to that spot in the video.
