Abstract:
The ventral visual stream underlies key human visual object recognition abilities. However, neural encoding in the higher areas of the ventral stream remains poorly understood. Here, we describe a modeling approach that yields a quantitatively accurate model of inferior temporal (IT) cortex, the highest ventral cortical area. Our key idea is to leverage recent advances in high-performance computing to optimize neural networks for object recognition performance, and then use these high-performing networks as the basis of neural models. We found that, across a wide class of Hierarchical Convolutional Neural Networks (HCNNs), there is a strong correlation between a model’s categorization performance and its ability to predict IT neural response data.

Pursuing this idea further, we then identified an HCNN that matches human performance on a range of recognition tasks. Critically, even though we did not constrain this model to match neural data, its top output layer turns out to be highly predictive of IT spiking responses to complex naturalistic images at both the single site and population levels. The model’s intermediate layers are highly predictive of neural responses in the V4 cortex, a midlevel visual area that provides the dominant cortical input to IT. Moreover, lower layers are highly predictive of voxel responses from fMRI data in V1 and V2. These results show that performance optimization — applied in a biologically appropriate model class — can be used to build quantitative predictive models of neural processing.

If time allows, I will also discuss recent (experimental and modeling) extensions of these results to tasks outside of categorization, shedding light on how cortex jointly represents the categorical and non-categorical visual properties that together underlie general scene parsing.