Leveraging the Allen Brain Observatories
Date Posted:
September 23, 2024
Date Recorded:
August 12, 2024
Speaker(s):
Saskia de Vries, Allen Institute for Neural Dynamics
All Captioned Videos Brains, Minds and Machines Summer Course 2024
SASKIA DE VRIES: Well, thank you for the introduction and for inviting me here. I always love to come here and to tell you about some of the things that we have that we've done at the Allen Institute and some of the new things that are starting to happen, as things always continue to grow and evolve.
So I'm going to be telling you about the Allen Brain Observatory, which was our first in vivo physiology open data set. And this was launched-- ooh, that's very bright. This was launched about 12 years ago in 2012 when the Allen Institute for Brain Science initiated a new project called the Mindscope project. Prior to this, the Allen Institute had created the Allen Mouse Brain Atlas and other types of brain atlases that were looking at gene expression throughout the brain.
And what they wanted to do, what they set out to do was to create an observatory of the mind where they would focus a lot of different survey-style data sets on a single part of the brain. And the idea for these types of survey data sets is to create, and to collect and create a data set that is collected in a really standardized manner so that the data is very consistent. It creates a volume of data that is usually kind of unachievable with standard experimental techniques in academic labs.
And it's collected without a specific hypothesis. So it's designed to be a broad survey that can be used in a lot of different ways rather than to answer one single specific question.
And while the previous work had been on things like gene expression and they started doing-- they had begun a connectivity atlas looking at how different brain regions were connected, the challenges with in vivo physiology is that it's so much more dynamic. There's so much more variability, both within subjects and across subjects, that you need to collect even more data than you would for an atlas in order to be able to make sense of that variability.
And so they chose to focus this observatory on the visual cortex of the mouse. And to contextualize that a little bit, I want to jump back. And I assume this is somewhat familiar to many of you, so I'm going to be somewhat brief. But if there's questions, of course, raise your hand and interrupt me.
But the visual cortex, if we-- has been widely studied in neuroscience because going back to the late '50s and early '60s when Hubel and Wiesel began using tungsten electrodes to record from single cells. And so this was a technique, a technology that they were developing in the '50s of having these wires that they could insert into the brain and really just nestle up close to a cell, and that they could record the currents around that cell to record the spiking activity.
And in their early-- in their work, they focused on the visual system in the cat, recording either in the primary visual cortex or in the lateral geniculate nucleus, which is part of the thalamus. That's part of the visual pathway coming from the eye to the thalamus and then back to the cortex.
And so-- ooh, I don't know if I set up audio here. We're going to make this work. We can-- if you've never seen the Hubel and Wiesel movies, they're really useful and give you a good intuition of what these experiments were like.
So what you're hearing are the spikes of one of these single cells. And what you're looking at is what's being projected in front of the cat on a projector-- on a screen very similar to this screen, probably a little bit smaller.
And they move this beam of light around until they hear spikes. And they're going in literally by hand, drawing on the screen these X's whenever the light is in a place where it drives activity from the cell, right? And they can move this light around, and there's places where the light drives the cells, places where it suppresses the cells. So they're putting X's where it drives it, and they put these triangles on these flanking regions where you see that the light is suppressing the activity from these cells.
So over here, they kind of realized it's a slightly wider region, and you hear that rebound when the light goes off. And so they go in. Takes them a while, but they put in the triangles and they do this on the other side as well.
[SCRATCHING]
Get these triangles. And then they can move this beam of light around. And what you're going to see in one second, they're going to reshape it a little bit. And when the light passes across the space, it's not going to fire unless it's hitting more X's than O's. And what that means is that when it starts to be oriented at this angle, it's going to-- where it hits those X's exclusively, that's when you hear the firing.
And so we have this orientation-selective cell that only responds when the bar is oriented in a particular-- at this very precise angle. And so even if it's-- as we saw, even when the bar comes down from the side, from the top, you don't hear a response, right? And so this was one of the types of receptive fields that they saw, the receptive field being the region in space and the pattern of light that is needed to drive a response from this cell.
And so they used their tungsten electrodes to record from a lot of different cells. And so here are some of the examples that they saw. This is essentially what we just looked at, where you've got these X's and the triangles flanking it on either side.
When they had the electrode in the thalamus-- so this is earlier in the visual circuit-- they had different shapes. They had these concentric receptive fields, again, with X's and triangles. Sometimes the X's are in the middle with the triangles on the outside. Sometimes it's inverted where it responds to darkness in that center and bright on the outside. But you've got these concentric-shaped receptive fields in the LGN.
A colleague of theirs, Stephen Kuffler, saw similar receptive fields in the retina. So this is in the back of the eye with-- his paper has a different style of figures, but there's a central region that's driven by the light and a broader region around it that's suppressed by the light, and in the same kind of concentric shape. But then in the cortex, as the example that we saw, you'd see these elongated, orientation-selective receptive fields where you can kind of figure out how much the cell is going to respond based on how much light is hitting X's compared to triangles.
But they also saw a different type of receptive field in the cortex where it's still-- the cells still responded to these oriented edges. But in this case, you can position that edge in different locations, right?
So with an example that we saw when they put the light over the triangles, you didn't get a response. But in these cells, you would get a response kind of regardless of the position. These are four different positions of these bars, and you can see the spiking response of the cell is pretty strong for all four of these different positions.
But if they orient the light at a different orientation so it's still going through the same region of space, this same amount of light in that region of space, you don't see a response, or you see a much weaker response. So the cell's still orientation-selective, but it's now phase invariant. Its position can vary within this broad region.
And so one of the questions that came up from this small collection of receptive field patterns was, how do you go from these types of concentric receptive fields in the thalamus to the elongated, oriented receptive fields of the simple cells? And then from there, how do you get those complex phase-invariant responses?
And they postulated in their original paper that you could imagine a situation where if you had a small number of thalamic neurons, all with their concentric receptive fields that happened to be arranged in space along a particular direction, that if all of them are converging onto a single simple cell, you could have something like an and operation, that all four of these need to be firing in order for the simple cell to drive a response from that simple cell. And so that's how you could build from these [INAUDIBLE] spots a response to an oriented edge.
And they suggested another type of configuration going from simple cells to complex cells where you could have an assortment of simple cells at different positions with different phases. And that as long as at least one of them is responding, it will drive the response of that complex cell, but it doesn't necessarily matter which of these three is being driven.
This, of course, was just speculation, but there's been a lot of work since they suggested this-- [SIGH] I can't do the math anymore-- like, 60 years ago that has been able to show circuits from the thalamus converging just like is suggested here. And this type of feed-forward hierarchy has been really-- is now really widespread in our understanding of neural networks and feature selectivity.
So these examples of receptive fields that I've shown you, I showed you one little example from the retina, the concentric receptive fields in the LGN of the thalamus, and then going up to the simple and complex cells in the cortex. These are really just the tip of the iceberg of the visual system. So while this is a beautiful drawing and it is a really impressive circuit, it is, in fact, just the beginning of the visual system.
This is what's known as the subway map, which shows that based off of connectivity of different cortical regions in cortex that are involved, thought to be involved in visual processing. And so it starts from the retina down here. V1 is this purplish color, going all the way up to the end of the entorhinal cortex and hippocampus. And so there's a lot of different regions in cortex that are all connected to each other in these complex ways.
And so one of the big questions that we've had in the field is, what's going on in all of these different parts of the visual circuit? What types of features are they responding to? How do we go from these spots and edges to object recognition, facial recognition, motion selectivity? How do all of those types of things get computed through these circuits? And how-- yeah, how do we build these types of responses?
And in order to understand that, one of the key things that we'd like is to be able to have a model that can help us predict the responses of a cell. It's one thing to map a receptive field and be like, ooh, this cell likes pictures of flowers in the field. But how can you actually predict that response based off of arbitrary stimuli in order to know whether you actually understand that cell's response?
And so what emerged was a model, again, going just in this early part of the circuit of the retina, the thalamus, and the primary visual cortex, that's kind of referred to as the standard model for this part. And it consists of a spatial and temporal filter. And the figure only shows the spatial filters, but there's a temporal component as well. And for the retinal ganglion cells or LGNs, these look like our concentric, spatial, receptive fields that we mapped with X's and triangles that becomes these elongated Gabor patches for simple and complex cells.
But you have this-- the spatial temporal feature that is convolved with the stimulus and then passed through a nonlinearity, right? And early on, you see this rectified response so that if the stimulus matches this filter, you get a response. But if you have the opposite, like if it's out of phase, the response is suppressed. And then those complex cells where we have the phase invariance, you've got this quadratic nonlinearity.
All right. And so from this, we have-- the intuition for this is if this is one of our simple cells spatial features, if we were to show the animal a grating stimulus that aligns at that right orientation with the right width and frequent spatial frequency, we'd expect to drive a response from that cell. But if we were to show this picture of a flower with a ladybug on it where the stem now aligns in that same location, at that same orientation, we'd also expect to see a nice, robust response.
And so this has been kind of the standard model in the field and how we've thought about this part of the circuit for many years. And about almost, I guess, 20 years ago now, there was a book that came out called 23 Problems in Systems Neuroscience.
I just-- I always like to bring up this book because if you're ever looking for project ideas, there's 23 project ideas. Actually, probably more project ideas, but 23 really interesting, big, open questions in the field. Even though it's 20 years old, I think they're all still open problems. I should really check that, though. I always say that, but maybe it's not true. Maybe it's only 21 problems.
But one of the chapters in this book is called "What's the Other 85% of V1 Doing?" And the authors basically point out that this standard model does a very poor job of actually predicting the activity of V1, which is to say that there are cells that are well predicted for some types of stimuli.
Part of the problem that they point out is that there's a lot of bias in the way we record from neurons, right? So the recording techniques, especially those single cell recordings, are-- listen, you're not going to put an electrode in the brain and record a really unreliable response, a really quiescent cell. You're not going to spend your time doing that.
So a lot of the recordings are from cells with big spikes and high firing rates. And so there's a strong bias, particularly like Layer 5 neurons. Those are big cells with high firing rates. So there's already kind of a bias for cells that already are responding the way we expect them to.
There's other biases as well where we go in using gratings. That's what we think we're going to see, so we go in waving these bars around, listening for these responses. And so then we're finding the cells that are responding to the stimuli that we're showing to it.
There's other types of biases as well that they call out. Some of the biases have to do with the types of theories of how we think about what's going on. But one of the things that they point out is that there really is a failure to be able to go from the standard model and to predict responses to naturalistic stimuli.
And so they show this example in their chapter. So this is-- in blue is the response of a cell to a natural movie clip that they showed to the animal. I think this is a recording from a primate, but I'm not 100% sure. And then the red trace is the response that they predict based off of the measured receptive field where they measure its orientation, its spatial frequency, all of the tuning properties of the cell.
They compute-- they create that spatial temporal filter. And when they convolve the movie with it, the predicted response is what's shown here in red. And so you can see this is just a failure to map between the prediction and the actual response.
They even show another illustration of this where the red trace here is the same as the blue trace. These are three cells that all have the same receptive field properties. So the size, the location, the orientation, the spatial frequency, temporal frequency. All of the properties of the receptive fields are really closely the same for these three different cells.
But when you look at their responses to this natural movie clip, they have different responses. Like, there's some places where they overlap a little bit, but they have a lot of differences across these three different cells. So this ability to predict responses to naturalistic stimuli and not just grating stimuli and things like that is a key failure of the standard model.
And so in their book chapter, in their conclusion, they have this paragraph where they ask, what would it actually take to understand V1? And they say, there's at least three ingredients required. One is an unbiased sample of neurons of all types, firing rates, and layers of V1. Second is the ability to observe simultaneously the activities of hundreds of neurons in the population. And then the third is the ability to predict, at least qualitatively model the responses of the population under natural viewing conditions for these more naturalistic types of stimuli.
And so in many ways, this little paragraph kind of exemplifies the goals that we had in creating this survey of visual activity in the visual cortex, was to try and create this type of unbiased sample of neurons. Nothing can be completely unbiased, but let's say a reduced bias sampling of neurons of all different types, of all different layers in V1.
And to record the activities-- wrong button-- of hundreds of neurons in a population, right? Not going in one by one, but trying to record as many cells at a time as we can.
So the work on this pipeline and this data set, it's called the Allen Brain Observatory Visual Coding Two-Photon Dataset. You can see our paper from a few years ago. And this work was work that I did, along with Jerome and Michael, as well as a large team of technicians and scientists who helped us to build the pipeline, collect the data, process the data, analyze the data, and all of those things.
And so what this survey is is we wanted to be able to compare responses. We're working in the mouse visual cortex, and we wanted to be able to look at responses in the primary visual cortex as well as some of the higher visual areas. So if you think of the-- if we go back to the subway map that I showed you where V1 was down here, we're trying to compare V1 to some of the subway stations kind of higher up in that map.
So this is the surface of the mouse's cortex. You have the barrel cortex up there and auditory cortex over there. Here's our primary visual cortex, and then these are higher visual areas. And so we collected data from six different visual areas.
And we leveraged a number of transgenic tools that are accessible in the mouse in order to collect data from specific types of neurons across different layers of cortex, as well as being able to collect data from excitatory cells as well as defined inhibitory cell types.
And so we're using calcium imaging, and I'll show an example of what this looks like in a second. But what we can do within the mouse is use transgenic tools to derive the expression of our calcium indicator in particular populations of cells.
And sometimes we use a broad driver. So these are what are called pan excitatory lines like SLC17 where it drives the expression of our indicator, our calcium indicator in any excitatory cell. And it's expressed across all of the different layers.
And so we can image at different depths in order to collect data from different layers, but there's a limit to how deep we can image with these lines because of the optics of imaging deep, right? You get a lot of scatter, and so you have low signal to noise the deeper you go.
Alternatively, we can target the expression of our indicator to specific types of cells that are found in different layers, or even in subtypes of neurons within the layer. So in Layer 5, we have populations of cells that are corticothalamic projecting and other cells that are cortical projecting neurons.
And so these are different types of cells both in Layer 5. And we can limit the expression of our indicator just to those cells, which means that we don't have the problem of the optics-- of expression in higher layers, so we can get good recordings from Layer 5 and Layer 6 neurons.
And as I mentioned, we can also target these three major classes of inhibitory cells. Parvalbumin, somatostatin, and VIP expressing interneurons. And so these have very different responses than we'd see in the excitatory cells.
And then finally, we wanted to use a standard set of stimuli so that when we're comparing the responses of these different types of neurons in these different parts of the cortical circuit, we're comparing apples to apples. We're looking at responses to the same types of stimuli across these different cell types and brain regions.
So our stimuli included some of these grating stimuli, similar to what's been used kind of historically, drifting gratings that move at different directions and speeds, static gratings that are flashed and they have different orientations and spatial frequencies. So the width of the grating.
We use a sparse noise stimulus to map the spatial receptive field. And then we've got natural images, flash natural images, as well as some natural movie clips. And then of course, we always have some period of no contrast stimuli that allows us to look at the spontaneous activity of the cells.
All right. And so this allows us to-- like I said before, there's no single hypothesis here, but it allows us to ask questions about how the visual information is represented and transformed through this circuit. Are there differences across cortical areas, different cortical layers and cell types? And whether the stimulus statistics can affect the encoding properties of the neurons.
All right. So this is an example of one experiment. And so we're using calcium imaging, as I mentioned. So this panel here is where you see what we're seeing through our two-photon microscope. So we have optical access to the cortex. We are expressing a calcium indicator in a particular population of cells. And what this means is that whenever a cell fires a spike, calcium floods into the cell, and then the indicator lights up. It's a fluorescent indicator.
And so we have a mouse that's head fixed on a wheel underneath this objective. It's an awake mouse. And you can see sometimes it runs. Sometimes it stands still. All of that is just by its own free choice. There's no particular task that we're asking the mouse to perform.
And so you can see different cells light up at different times. And we can start to ask questions about, what types of stimuli are we showing when this particular cell or that particular cell fires up? So we could do the same Hubel and Wiesel analysis for each cell in this field of view, and there's probably a few hundred cells in this field of view. You can't see all of them at the same time, but it's a pretty dense population.
So awake mouse, you saw some of the stimuli that are shown here. Tracking the activity, we've got a camera on the eye that gives us pupil area as well as the eye position.
Mice are not foveal the way that humans and primates and cats are, so their eye movements are-- they don't make as many eye movements as we do, and they don't play as big a role as they do for humans and primates, but we still want to be able to capture that. We have information about when the mouse runs, how fast it runs, things like that as well.
And so I mentioned this is done with a pipeline. So we've got teams that are doing each of these steps of creating the surgical window. We map out the different visual areas so that we can go in and target them for our calcium imaging. We get the mouse comfortable being handled, being head fixed, being on a running wheel.
Then we collect data. For each field of view, we return to that field of view across three different days because our entire stimulus set takes three full hours for us to show it. And so we image for one hour at a time. And so we come back to that same group of cells on three different days.
And then for a given mouse, we might image from a different field of view. So the first day we might image from V1. We come back for two more days using all of our stimulus, and then we want to collect data from Area LM, which is a different visual area, and we'll do three days in LM.
After we're done imaging, the mouse is euthanized and we do histology, which allows us to, A, make sure that there's no problems with the brain. Sometimes having the window or other parts of the surgery can have adverse effects. It's rare, but it can happen. But it also can confirm that the expression pattern for the cell line that we used was, in fact, what we expected to see.
And because we're doing this in a standardized way, it allows us to set up quality control metrics at every stage of the process so that we can make sure that the data that we're collecting is high quality. And a lot of this that you see has to do with the animal health and the procedure at different stages of the surgery.
But there's also a lot of quality controls that go into the actual session data that gets collected to make sure there's not too much motion artifacts, or that everything-- all of the hardware worked as it appropriately did. So one of-- actually, I think, one of the things that often goes unrecognized is how important this type of quality control is for these type of large-scale standardized data sets.
So we used this pipeline and collected over 1,300 hours of data from over 250 mice. And so you can see many fields of view here from lots of different experiments. And one of the things that you might notice is that some of these are really bright with a lot, a lot of cells in them. Others are very dim and you only see a handful of cells.
And so this poses another challenge, right? One challenge is our standardized data collection. The other challenge is our standardized data processing where we go from these raw movies, we do motion correction, we do cell identification and segmentation, we extract our fluorescent signals, we correct for neuropil, we do all of these processing steps. And we want to do this with as little manual intervention as possible because we have 1,300 hours of this data.
And so one of the big things is developing that type of data processing pipeline that can handle this type-- this range of variability with some ease. And so we processed all of this data. This is a table that just kind of gives you a summary. In total, I don't know if you can see it, but there's over 63,000 neurons in the data set from 456 different experiment containers, each container being three sessions.
And you can see how this is divided. These are all of our different transgenic lines that we used. And you can see which layers they were imaged in. And so many of these target, as I said before, different populations in particular layers. These are the inhibitory lines at the bottom.
And then you can see how these were surveyed across our six different visual areas, the primary visual cortex, V1, and then these five higher visual areas. So not every line was imaged in every area, but we got a broad survey of several lines across all of the areas. And then in two of the visual areas, we got all of the different cell types within those two areas.
So of these 63,000 neurons, I will show you one. I won't show you 63,000, because that would take a very long time. So we can look at the types of responses that we see.
So this is an example cell. This is an excitatory cell in Layer 5 of V1. And we can look at its responses to the drifting gratings. So these are sinusoidal gratings that move in eight different directions and at five different temporal frequencies or speeds.
And we represent the response here on this plot where each arm represents a different direction of motion and each ring is one of the temporal frequencies, the slowest one in the center, the fastest one on the outside. And then you see these red dots. Each dot is a single trial, and the color of the dot corresponds to the strength of that trial's response.
So where you see this group of dark red dots over here is where the cell had a reliable response or a strong, reliable response to that stimulus condition. Whereas over here and over there where you don't see the dots because the response is so-- there is no response, that's where the cell isn't responding.
And so we can look at the cell and we can figure out pretty quickly that the cell is responding to horizontally oriented bars that are moving upwards. It has a weaker response to horizontally oriented bars that are moving downwards, right? There's a little response over there and kind of these intermediate speeds.
We also showed static gratings. So these are, again, sinusoidal gratings that are flashed at different orientations. And so each arm here is a different orientation. And then at different spatial frequencies. That's the period of the grating. So the low spatial frequency. So the wide bars are here in the middle and the high spatial frequency is out there.
And again, these are large numbers of red dots, the dots showing the strength of the response. And so we can say that it's responding to this condition here as well as that condition there. So these are horizontal, maybe slightly angled gratings at, again, intermediate spatial frequencies.
We can look at the responses to the sparse noise. So these are black and white spots that are flashed in different positions. And we mapped separately the responses to the white spots and the responses to the black spots.
This cell in particular only had significant responses to the white spots, kind of on this edge of the screen. So if this was the screen the mouse is looking at, it would kind of over here.
We showed 119 different natural images. These were selected from a couple of different image data sets. Each image was shown 50 times. And so on this plot, each ray is a different image and each dot is a single trial. And so these long rays are where the cell, again, responds reliably and repeatedly across many trials. So there's about four different images that the cell responds to.
And then we showed some different natural movie clips. And so this is a 30-second movie clip. It gets repeated 10 times. And so this is essentially just a raster plot of the cell's activity. You've got 10 trials down here, and just loops around. The outer ring is just the average of those 10 trials. And so again, you'll see there's a couple of points in here where across all 10 trials we're seeing activity from this cell.
So here's this one example. We can put metrics to this. We can talk about the area of the receptive field. We can talk about the direction selectivity, how strongly the cell prefers the gratings moving in one direction compared to the opposite direction. Likewise, orientation selectivity. We can put a variety of different metrics on this. And if you go and look in the paper, you'll see a lot of plots comparing these types of metrics across different areas and cell types, and all of that.
We can also try and make sense of all of these different features that we just saw. So we know that the cell responded to light things kind of over on the far edge. It responded to horizontal, slightly off-angle gratings, particularly if they're moving upwards. There were these four images that it responded to. These are the four that it had strong responses to.
And you can start to do a little bit of hand-wavy explanation about what's going on here, right? So this hill has a bit of an edge kind of right over here that might be at the same angle. Maybe the top of the tiger is also having that effect. The edge of this wing could be driving something.
We can look at the movie clip when the cell is-- one of the times where the cell responds. The movies are all taken from the opening scene of Touch of Evil. There's this three-minute-long scene with no camera cuts, but a lot of different types of motion. I know the first three minutes of Touch of Evil really well, and I've never seen anything after that, so no spoilers, please. Yeah?
AUDIENCE: Can you convolve the line with the rest of the images?
SASKIA DE VRIES: Can we convolve this line?
AUDIENCE: As in, look for that line in all the--
SASKIA DE VRIES: In all of these images. Well, so that's kind of by eye, right? That's what we're kind of doing. And if we look at this movie clip, you see that the response is right at this point where this bright-- it's a street organ kind of comes into the scene. And so we can start to hypothesize that this cell is being driven by bright edges, particularly on this side of the visual image. You can kind of pull all these things together, right?
And so the question that is essentially what I think you're asking is, can we come up with a model for-- the standard model for this cell that will predict the cell's response to these different stimuli? And so here we go.
So this is one of the things that we did with this data set, was to implement the standard model. And the way that we did this was we used a wavelet projection, a basis set. So we've got 3D wavelet-- so it's both spatial and temporal-- that span all of the spatial and temporal features that are within the mouse's visual space. I can't tell if you have a hand up. Yeah?
AUDIENCE: Yeah. I have a question about getting the images and the videos you used. Are they controlled for spatial frequency or context?
SASKIA DE VRIES: So are the images controlled for spatial frequency is what you're asking? Yeah. So that's a great question. And they are. So we spent some time looking in the data and the literature about what the range of spatial and temporal frequency tuning that we expect in the mouse based off of behavioral assays and previous studies.
And so one of the things that we made sure is that the images that we were using had strong power in that range, and they all do. Yeah. So we don't want to show them images that have really, really high spatial frequency that they can't possibly see. Like, who cares?
So we have a bunch of different basis functions. They cover different spatial frequencies, different-- so you have big ones and small ones. They cover different orientations, different temporal frequencies that span the whole visual space. They're followed by either linear or quadratic nonlinearities so that we can have both simple and complex responses going on. And then they all get weighted, right?
And so we can train and test this data either using our natural stimuli, the images and movies, or using the artificial stimuli, the gratings and noise. And we do held-out cross validation. So we're using a part of the data to train the weights on all of these different projections and nonlinearities in order to see how well we can predict it. We're also adding in the mouse's running speed because we know that that also plays a factor in the responses.
And so we can ask, how well does the cell that we looked at, how well can we predict its responses? And so here, what I'm plotting is the correlation between the predicted response to the measured response, either for the natural stimuli or for the artificial stimuli.
And we actually do pretty decently. We get a little bit above 0.4 for each of them. And if you go and look in the literature for studies that have used these types of standard models, that's pretty good for what you see out there. It's definitely not 100%, but it's considered a good performance.
All right. So this is one cell out of our 63,000. What about the others? The others don't look as great, right? So there's our one cell up here. This is a density plot. So the color is telling you how many dots are in each of these places.
And really, the bulk of our data is right down here, very close to 0, 0, right? We have our example cell. It's not even our best cell. Like, I could have picked this cell out here and we'd have seen even better performance. So there's a handful of really good cells, but the vast majority of our data is not well predicted at all. So what can be going on?
One of the things-- and this is kind of something that jumps out really quickly, but not all cells respond to all of the visual stimuli, right? The example cell that I showed you, it had reliable responses to each of the types of stimuli that I showed you, all of the gratings, all of the-- like, there was some feature that across trial after trial, it responded to.
But not all cells respond to all stimuli, and this is something that really just jumps out at you when you just look at some of the data without doing very much analysis. So this is-- I've plotted 50 cells from one experiment. This is across the entire imaging session, so it's a little over an hour.
And I've just highlighted the different stimulus epochs. So these are about 10-minute blocks where we're showing different static gratings. And then for 10 minutes, we show different natural scenes. Then we've got five minutes of spontaneous activity, and then we show more natural scenes, et cetera, et cetera.
And you can very quickly see some examples of cells that are really, really active during the static gratings that suddenly are much less active when we switch to the natural scenes. And it's not that the cell died or it fell out of plane of the imaging plane, because when we come back to the static gratings later on in the experiment, we again see really robust activity from that cell. So it seems to be somewhat selective for the static grating stimulus.
You see that also here with a couple of cells where for the natural movies, this movie gets repeated 10 times. And so you have a number of cells here that have really nice periodic responses where you see these kind of 10 peaks at some feature of the movie that drives it. But then when it switches over to the natural scenes, the flash natural images, the cell's not responding with a high reliability or a high response rate.
And so we see this selectivity. So we wanted to dive into this a little bit and look if a cell responds to one stimulus, reliably to one stimulus, is it going to respond reliably to another stimulus?
And our first pass was just to do a pairwise comparison where we looked at the reliability of the cell's response to its preferred condition of one stimulus compared to the reliability of a response to another stimulus. So what that means is consider for the natural scenes, there's one image that's going to drive the cell that's across-- average across all of the trials is going to drive the cell the best, right?
So you find out, of those 50 trials that we showed that image, what fraction of those cells had a significant response? And that's comparing each response to a null distribution that was taken during the spontaneous activity.
So let's say for one cell, it's a picture of the tiger in the water that drives a cell-- that drives the strongest mean response. And it turns out that 52% of the trials that we showed that image, it responded to.
Now we want to compare that with those-- again, for the drifting grating, there's going to be one direction and frequency of grating that drives the cell the most. Of the trials of that grating, how many did it respond to? And can we-- what's the correlation between those reliabilities, right? So that's what we're looking at here.
And so we've got our different stimuli, static gratings, drifting gratings, natural scenes. And we've got natural movies. We've got Natural Movie 1A, B, and C. And that's the same movie clip that gets repeated in each of the three sessions that we image from those same cells.
And so the correlation across these, across 1A with 1B and 1C, is the variability of the same cell's response to the same stimulus across different days. And so that gives us a bit of a baseline to think about that day-to-day variability for the same cell in the same animal to the same stimulus.
And then there's two other natural movies that are different movie clips, but are similar, taken from the same overall movie. Similar statistics and all of that.
And what we see is that by and large, there's very low-- very low correlation between the reliability of a cell's response to one stimulus to its response to another stimulus. What that means is if a cell responds reliably to our drifting grating stimulus, I don't know if it's going to respond reliably to a natural movie or not. A correlation close to 0 doesn't mean that it won't respond reliably. It means, I don't know. It could respond reliably. It could not.
The only places where we see some correlation is between static gratings and natural scenes, and then this correlation that we talk about of the same cells to the same stimulus across multiple days. But by and large, knowing that a cell responds to one stimulus doesn't tell us whether it's going to respond to another stimulus.
We took this a little bit further. This is just doing a pairwise correlation. But we wanted to ask this a little bit more broadly, and we did a clustering of this reliability across stimulus types, right? So now what we've done is we're looking at drifting gratings, static gratings, natural scenes, and we're looking at the reliability for natural movies.
And you've already seen that there's five different natural movie stimuli that get shown across all of these things. We're just taking the largest reliability for any one of those natural movie clips for a given cell. And we want to-- we cluster these reliabilities for each cell, and these are the clusters that we get.
So there's 30 different clusters. This is just a zoom in on one of these where we see high reliability for drifting gratings and natural movies and low reliability for static gratings and natural scenes.
Now, we've colored these on this heat map. And the way that we set the scale on this heat map is that we knew that there's-- we know that there's some cells that don't respond to our stimuli, right? Just going back to that raster plot that I showed, if you look closely, there are some cells that just aren't super active.
So we know there's got to be one cluster of cells that aren't very responsive to these stimuli. So we find the cluster up here that has the lowest mean reliability across each of the stimuli, and we use that to set the threshold. And we took the mean of these four reliabilities plus one standard deviation, and that's how we set the midpoint of our color scale. And that was about 27%. So a little over a quarter of trials differentiated the cells that were in this unresponsive group.
And so from there, we then grouped these clusters into classes based off of the patterns that we see in terms of, which of the stimuli are red? So they're more reliable. And which of the ones are blue? Just based off of this threshold that we've set.
And so now we can group these. And so for instance, there's about six different clusters here where you've got reliable responses to drifting gratings and natural movies and not to the static gratings and natural scenes.
Now again, as we saw just a couple slides ago, the correlation between drifting gratings and natural movies itself is 0. So you still see a lot of variability in how reliable those responses are for those two things, but both of them are just above our threshold of 27%.
So now we've got these different classes. So there's some cells that respond reliably to drifting gratings and natural movies. There's some cells that respond reliably just to natural movies. There are four different stimulus categories here, so there should be 16 different classes that we have.
But we don't actually find cells in all 16. So this is just looking at the percentage of cells in our data set in all of the possible groups that we've done. We iterate-- we did this with different seed conditions in order to test the robustness of these results. Not all of these 16 categories have cells.
Interestingly-- and this was kind a bit of a surprise to us-- our largest class was our none cells. 35% of the cells in our data set don't respond reliably to these stimuli.
We then see big classes here for the drifting gratings and natural movies where we see-- we believe that the motion, the temporal correlations of the stimulus are really important so that these cells are driven by some of these motion features. We see a big category for the natural scenes and natural movies where we think that the spatial statistics are really important.
But there's only 10% of cells in our data set that have this-- that respond reliably to all four of these classes of stimuli that we show, right? That are these kind of all cells like the one example that I showed you.
And so if we come back to our model performance, and if we look at this and we break it down by our different classes, our none cells are all down here at 0, 0. If a cell's not responding reliably to any of the stimuli that we're showing, we're not going to be able to predict its responses. You can't predict unreliable responses.
The cells that respond reliably to the naturalistic stimuli, they fall up here where we're seeing not terrible predictions for the naturalistic stimuli, but not good predictions for the artificial stimuli. And again, they don't have a reliable response for these types of stimuli, so we're not going to be able to predict that, but we can get OK predictions on that axis.
And if we look at the 10% that are in our all cell category, these-- well, they have pretty broad. You can see that they're well kind of matched across both natural and artificial stimuli. Their median is close to 0.4, 0.4. I told you before in the literature this would be a pretty good performance for the model.
And so we can then look at these results. These are just kind of lumping all of the cells together across all of these different types and areas. This is the same plot I showed you before. I've just rotated it on its side so we can look at how this breaks down across areas.
So here's V1 and then the five higher areas. And so in V1, for instance, it's only 20% of our cells that are these none cells. And it's RL that's almost entirely made up of the none cells. So this 35%, you need to take that with a grain of salt because that's really somewhat in the higher visual areas and less a thing in V1. We've got about 15% of V1 that are these all cells like we saw. But some of the things that you'll notice, so the fraction of none cells increases.
Another thing that I find really interesting is that the fraction of cells until you get to RL-- but before that, the fraction of cells that are in these blue categories, which are the natural movie, drifting grating, or just natural movie, but kind of the motion-sensitive cells, that remains pretty constant, even though the fraction of visually responsive cells at all is going down. So this suggests that in these areas, AM and PM, that there's kind of a condensing of motion feature selectivity compared to some of these earlier areas. Yeah?
AUDIENCE: So [INAUDIBLE]. I'm not familiar with different areas. Is it like a hierarchy or--
SASKIA DE VRIES: Yeah.
AUDIENCE: [INAUDIBLE]
SASKIA DE VRIES: Yeah. So this is a great question. So I don't know if everyone else heard this. How are these higher visual areas connected? Is there a hierarchy, or is there something else going on?
It's a bit of an open question. Some people argue that it's a hierarchy. Others argue that there's some parallel tracks going on. There's a concept of-- I don't know if you've learned about dorsal and ventral streams in the visual system in primates. There is thought to be those types of streams in mouse.
What I will tell you, though, is that we're imaging the surface of the cortex, and so most of these areas would fall into what we think is the dorsal stream. You could argue that LM might be part of the ventral stream, but most of the ventral areas are hard to access optically.
So personally, if there's a hierarchy, it's a weak hierarchy. It's a relatively flat hierarchy. And I think there's more of a parallel processing type of thing going on. But other people in the field disagree with me. So-- yeah?
AUDIENCE: So when you're looking-- or this is a plot of cells that are responding reliably to a stimulus as a whole.
SASKIA DE VRIES: Yes.
AUDIENCE: And so is there-- like for example, with their dynamic movies, is there a subset of the stimulus that it does respond reliably for, but not the whole movie?
SASKIA DE VRIES: Yes. So for each of these cells, there's going to be something-- if it's responding reliably, say, to the drifting gratings, there's going to be a particular frequency and direction that it responds to and not others, and that's going to be different for all of these cells. And you can then go and look and say, for cells in this area, the temporal frequency tends to be higher than cells in that area, or something like that. Yeah.
All right. So the question is-- the previous question was, how do these clusters go across layers? And so this is showing just in V1. And so you'll see we've organized it by layer, and then within layers by these different cell types.
And the big takeaway here is for the excitatory cells, there's not a ton of difference in the functional clusters across the different layers with a few exceptions. But generally, they look pretty much the same.
The big differences are with the inhibitory types. And so mixed in here you'll see this is somatostatin inhibitory cells. And one thing you'll notice with somatostatin is that there's almost no none cells. Somatostatin cells have really, really reliable responses, and almost half of them are the all cells, that they respond to all of the different stimuli that we show.
The VIP cells, they're also interesting in that they have this typical 20% are none cells. But they have a really big fraction of cells-- this is another VIP and superficial layers-- a really big fraction that respond to the natural scenes, natural movies. So they're driven really well by the naturalistic stimuli.
They're actually suppressed by the gratings that we show. These high-contrast gratings will actually kind of often suppress them below their baseline firing rate. So you don't see much in terms of these blue-- the dark blue that has the drifting gratings in there because they don't respond to those.
So those are the biggest differences we see across cell types, and likewise across layer. There's not much-- there aren't really big differences that jump out among these features.
All right. So a couple of conclusions that we take from this. As we pointed out, only about 10% of these neurons are these kind of textbook all cells where you see a pretty nice match between their responses to artificial and natural stimuli where we can pretty well predict what they're going to do using the standard model.
But instead, there's a lot of neurons that aren't responding to these stimuli. And this raises a lot of questions in terms of, what are they doing? Are they responding to more complex visual features? Or is it that they are involved in, say, the integration of different types of modalities?
And things that I can say that speak to this-- so I think if any of you have paid attention to mouse physiology in recent years, one of the things we see a lot of is the influence of motor activity on neural activity, even in these sensory areas. So if the mouse is running, it increases the activity of many cells and increases their visual responses as well. So for instance, we could ask, and one of the things that we looked at is whether cells that are none cells are more correlated with the mouse's running behavior, and thus less involved in the visual features.
And so you can see here, this is a plot of the correlation of cells' activities with the mouse's running speed. So just ignoring visual stimuli, this is just in V1, broken down by the different cell types and the different layers.
But when we look at this across our different cell classes, we actually see that the none cells, their correlation and the median is really close to 0. Our all cells actually have a somewhat stronger correlation with running than the none cells do, in fact. So it's not that they're driven by the running behavior.
We can also look at the eye movements. I mentioned that mice make eye movements. Their eye movements are less frequent, and they basically move just, like, back and forth. You can see that in this plot. There's very little vertical motion. It's just along the x-axis.
And we did a study looking at cells' activities relative to these saccades. And you can see examples here of cells that have increases of activity with saccades, other cells that are suppressed. Their activity is suppressed across saccades. And then also cells that have a direction selectivity. Like, they respond to saccades in the temporal direction, but they don't respond to saccades in the nasal direction. So those are-- here's an example of one and here's an example of the ones that respond in the opposite direction.
And so we asked again whether these-- whether these saccade responsive cells are more likely to be among our none category or the other ones. And we actually saw there was no difference in our distribution of-- in the fraction of saccade responsive cells as well as the distribution across these different types of responses across the different categories of cells that we have.
So it's a little bit above 10% of all cells are saccade responsive, and that's pretty even across our different categories. The slight differences you see here really just are sample size differences that come from that.
So we don't have any reason to think that these none cells are motor-driven and aren't part of the-- in a way that makes some kind of a separate type of thing. But I think there's a lot more space to be explored about all of-- this number, this fraction of cells that aren't engaged with the types of visual stimuli that we've used.
And then another question. So pointing out these things, but one of the big questions is, can we develop better models to capture the neural responses that might be able to account for some of the things that the standard model is failing to capture? And I feel like this is-- if you're looking for me to tell you, yes, here it is, you're going to be sorely disappointed because this is, I feel like, the challenge in our field right now.
And it kind of comes back to, as I mentioned, the 85%-- what is the other 85% of V1 doing? There are these biases in a lot of our theories, as well as the biases in the types of stimuli and the types of recording methods that it's possible that we're approaching this kind of in the wrong angle, and that the types of models that we're using are kind of stuck in an archaic way of thinking, and we need to break out of those models.
And this kind of, I think, leads us into where-- some of the challenges in our field. And I think a lot of why-- I'll get you in a second-- why I think-- what I want you guys to pick up and run along with is really trying to push new directions in our understanding and our thinking of these neural activities.
And I want to point you to this review article from a couple years ago, the title being "Large-scale neural recordings call for new insights to link brain and behavior." And it's a great review article that points out that a lot of our efforts in recent years where we've increased the numbers of cells that we record from-- this is what's called a Stevenson cording plot showing the number of simultaneously recorded neurons as a function of year.
And it was for-- for a while, this was doubling every seven years. I think that's now-- I think we've broken that with new technologies that have been developed through the Brain Initiative for dense electrophysiology and for optical imaging methods where we're really starting to deviate from that line.
But we're able to collect these really large-scale data sets. And when we apply a lot of the analytical methods that were developed when we were recording one cell at a time or four cells at a time, a lot of those ideas are starting to fall apart.
And so the challenge in the field is really to push further and to use new analytical methods to approach these data from new vantage points. And so I put this to you as, I'm looking to you to help answer these questions because the methods I learned, I learned back-- back over here when we were looking at a dozen cells at a time. And I can't do the math for thousands of cells at a time. That's just outside-- I'm too old to learn new tricks.
So I'll give you the data, and you guys come with new analyses. But I think that this is then-- actually, before I shift, you had a question a second ago.
AUDIENCE: On the none cells, are the neurons in the higher visual areas less responsive in general?
SASKIA DE VRIES: So the neurons in the higher visual areas-- so the final visual area I showed you, RL, that had 85% of none cells, I would put a big grain of salt next to that area. It's a really kind of weird area, and it's hard to map it on the surface of the cortex. So it's possible we didn't target it very well.
And so I'm always like-- when people are digging into the higher visual areas, I'm often like, maybe ignore RL in our calcium data set. There's some peculiar challenges with that particular area.
The other areas-- I think I can jump back real quick. Yeah, these other areas, there's still 60% of cells that are responding to something among these stimuli. And I think there's, again, big questions about, what are these other 40% of the cells doing? And like I said, I think that's a big open question. Yeah?
AUDIENCE: First, thanks for the very nice set. It's amazing to see so much data about [INAUDIBLE]. I have a question that regards the shifting paradigm that it seems that these data propose.
Like, we began from [INAUDIBLE] 15 years ago from a tuning based [INAUDIBLE] where we are in search for this nice tuning [INAUDIBLE]. So these neurons describing [INAUDIBLE], but now it's not working anymore. Some people say that populations are the way forward, but the population seems like a sort of weird term. What do you think about this?
SASKIA DE VRIES: Sorry. That population-- population recordings did you say?
AUDIENCE: In analysis.
SASKIA DE VRIES: Population analysis to understand the--
AUDIENCE: Instead of analyzing from a single neuron, you analyze a population of neurons [INAUDIBLE] that behavior.
SASKIA DE VRIES: Yeah. No, I mean, I think that that's a big thing. And this is where I think that the new technologies, I think, are really kind of pushing us into a new space because we now can have really large populations that aren't a dozen cells, that are hundreds and sometimes thousands of cells, often from different brain regions simultaneously.
And I think that that's really going to open a lot of new avenues for analysis and new kind of questions that we can-- can really drive our understanding further forward. So I absolutely think that the population level is the place to look.
I do think at some point, you need to bring that back to single cells a little bit and understand how single cells are part of that. But this is a question-- and it kind goes to a slightly more philosophical space of, at what level do we need to understand how the brain works? And is it at the cell? Is it at the molecule? Like, do we need to get down to the atoms?
And so personally, I like to be in that cell to circuit range. That's kind of where I'm most comfortable. But I feel like maybe we need to shift and be looking at the circuits and the populations, and then in 15 years come back and understand how the cell is part of those circuits a little bit better. Or we have people answering different parts of those questions simultaneously. My comfort space is I want to understand how a cell is part of a circuit, but I don't know that that's the only place we can answer any of these questions at. Yeah?
AUDIENCE: I work on human data, so I'm not exactly sure if, I guess, functional connectivity is, like, as much--
SASKIA DE VRIES: It's a huge-- it's a huge thing.
AUDIENCE: And so I guess my question is with these, I guess, selective responses within each of these areas, do those are these selectivities then functionally connected across the layers of the hierarchy, or is it that these cells are-- it's just layer selective?
SASKIA DE VRIES: Sorry. Rephrase that just slightly for me.
AUDIENCE: So is it that if cells in V1 that are selective to the static gratings are more preferentially connected to the gratings in LM?
SASKIA DE VRIES: No, great. That's a great question. And this is a thing where this data set isn't really well poised to answer that question because we image each area separately. And I can jump real quickly. The big challenge is-- OK. So I'll come back to your question in one second.
So this is one data set that we collected, but there is a couple of related data sets to it. And so one is a data set that uses essentially the exact same stimulus, some slight modifications, but now we're using Neuropixel electrodes. So these are silicone probes that have hundreds of sites along this shank, and so you can record thousands of neurons along a single-- along a single Neuropixels probes.
And then we used six Neuropixels probes. So we're hitting each of those visual areas that we did our calcium imaging in simultaneously. And then we're getting the cortical areas as well as subcortical areas simultaneously.
And so this is the type of data set that might allow you to ask that question a lot better, where you can look for cells that are selective for certain types of features or certain types of properties, and then look for correlations of those cells with cells in different areas. Whereas in ours, we're just one layer, one area at a time.
And so yeah. So this is-- and this kind of points out why having these different modalities can be really valuable. Like, here you don't get the cell type resolution that we had for the calcium, but you get this simultaneous area. You get spike time resolution, which you don't get with calcium imaging.
We have two other data sets. Oh, here's just a snippet of some of the things that they saw with this data set is asking the question about the hierarchy. And so here, they analyzed these data looking at certain features. This is the latency of the first spike, and showing a progression of that latency as you move across these different visual areas. I don't know if you see down there. But suggesting that there is kind of a linear hierarchy among these areas.
We have two other data sets that some of my colleagues collected, again, leveraging the same pipeline. So the same infrastructure, the same surgeries, the same recording hardware. And here, rather than showing these passive visual stimuli, the mice were engaged in a visual change detection task.
So they flash a sequence of images, and then it's the same image that gets flashed repeatedly, and then it switches to a different image. And the mouse's job is to report every time that a new-- there's a change from one image to the next.
And so these experiments were done again, either using calcium imaging or using the Neuropixels recordings. And one of the things that they see is they train the animals with a particular set of images. There's, like, eight images that they switch between. And then when they're actually doing the recordings, they'll record once with the images that they've been trained with, and then they'll switch to a novel set of images that they've never seen before.
And one of the things that they've noticed is that the response to a changed image is much larger in excitatory cells when those images are novel than when they're familiar. And they're working out some interesting things in the inhibitory microcircuit that they think kind of caused this change or this novelty response that comes out of it. You have a question?
AUDIENCE: Yeah. What about the anesthetized animals?
SASKIA DE VRIES: Yeah.
AUDIENCE: OK. But how do these statistics relate close to that kind of condition?
SASKIA DE VRIES: Yeah. So all of the data sets that I've shown you have been done in awake animals, partly because we want to understand the awake circuit, right? And there are some noted differences with anesthetized. And we did have a study very early on when we were kind of getting all of these things built together where they did extracellular electrophysiology. Not Neuropixels, per se, but just extracellular probes in mice when they were awake and when they were anesthetized, I think sometimes in the same mouse, but I'm not 100% sure on that.
And there are some subtle differences, mostly like in the temporal frequency tuning, but the differences are kind of subtle. You have to dig into it. But there is effects of arousal on responses, both in terms of overall activity, in terms of the strength of the responses to visual stimuli.
And this is where the running comes into play because mice are more visually responsive and more alert when they're running than when they're standing still. It's kind of a state of stupor a little bit often when they're stationary. And so there are effects of that arousal on a lot of these processes, and that's one of the reasons that we do this with the running and taking all of those things into account. Yeah?
AUDIENCE: For the novel stimuli, is it that more cells are being activated at that time? Or is it the same roughly number of cells that are being activated in the familiar case, but at a larger--
SASKIA DE VRIES: So there's two things if I recall correctly. There's two things going on. And one is that-- and I think this is an example of a cell that responds to familiar images and responds more to novel images. But there's also cells that will respond to the familiar images and not to the novel, and vice versa. And so there's kind of a combination of things going on.
But the effects with the inhibitory circuit-- with the novel-- cells that respond only to the novel, it could just be the images that are chosen, right? Whereas with the inhibitory cells, they respond to pretty much all of the images, but they respond a lot differently when it's a familiar versus novel. And so that's where we see that there's something going on that's affecting the gain in that circuit because of the novelty situation.
AUDIENCE: So [INAUDIBLE] for the stimulus classes. Did you show like blockwise manner? And also, what's [INAUDIBLE]?
SASKIA DE VRIES: I just can't hear you.
AUDIENCE: Oh, sorry. So when you show the different stimulus classes, can you show the blockwise or [INAUDIBLE]?
SASKIA DE VRIES: Yeah, block-- so it'd be, like, 10 minutes of images, 10 minutes of gratings, five minutes of movies. They were in pretty large blocks.
AUDIENCE: [INAUDIBLE]
SASKIA DE VRIES: Say it again.
AUDIENCE: What's the interstimulus interval?
SASKIA DE VRIES: So it varies. So for the drifting gratings, you have a grating that-- each grating condition is shown for two seconds, and then there's one second of mean luminance in between them. But with the flashed images, there's actually no intertrial interval, so it just goes from one image to the next to the next, which is different from the behavior configuration, but in the way that we did it. And same with the static gratings. It just went from one grating to another. Yeah?
AUDIENCE: When you do your recording and you're measuring reliability, is all the recording done on the same day?
SASKIA DE VRIES: So there's three days of imaging. And the one stimulus that's repeated across the three days is one of the natural movies.
AUDIENCE: Got you.
SASKIA DE VRIES: Yeah.
AUDIENCE: I guess my question then is if you analyze it within day versus across day, do you see a higher amount of selectivity within a day? And then does that change across the days?
SASKIA DE VRIES: So if I jump back-- yeah, I jump back to this, right? So the real place where you can ask that question, again, is with this natural movie where you can see that day-to-day variability. The things that I'll point out, for instance, is that the static readings and the natural scenes, those are on the same day, right? And so I do think that that's why they're slightly high-- there's a slightly higher correlation.
But the drifting gratings-- the drifting gratings are on the same day as Natural Movie 1A and Natural Movie 3. And there, we don't see that because they're on the same day. They all respond together type of thing. So I don't think that's a huge component, but I think it's something that warrants more attention.
All right. I want to just to tell you real quickly, because we're pretty much out of time, but I wanted to tell you just a little bit about all of this is work that has been done. These data sets are available to access. They're on the cloud. You can download them. We've got a software kit that helps you find and work with the data.
But I want to tell you a little bit about where things are going now and what's happening. And this was-- as I mentioned at the start, this was a project that started in the Allen Institute for Brain Science. And when it started, the Allen Institute in science was the institute.
And since then, there's now-- I can't count-- seven different institutes within the Allen Institute. So there's the Allen Institute, which has the Allen Institute for Brain Science and the Allen Institute for Cell Science, the Allen Institute for Immunology. And just two years ago, we started the Allen Institute for Neural Dynamics.
So this is a new brain science institute within the umbrella. We collaborate and work very closely with our colleagues in Brain Science. Brain Science is really focused mostly right now on questions of transcriptomics. They've done a lot of spatial transcriptomics and identifying real fine resolution cell typing across the entire structure of the brain so you're no longer focused just on the cortex.
And so we've now all moved into the Institute for Neural Dynamics. And I want to tell you just a little bit, just to give you a taste of what's happening, because the institute has a bunch of different teams that are focused on a lot of things.
But we still have a lot of this type of high-density electrophysiology, Neuropixels probes, a lot of calcium imaging and optical physiology. Not just calcium imaging, but using fiber photometry to image different types of indicators and transmitters deeper in the brain. We're doing some calcium imaging that gets combined with spatial transcriptomics so we can get finer resolution on cell details, as well as some molecular anatomy teams that are working to map some of the brain-wide circuits with much more resolution.
So for instance, we use what's called an expansion-assisted light sheet microscope where we can sparsely label cells and then clear and expand the brain. So we can image the brain intact, and we can get full brain reconstructions of individual neurons. And so this gives us really high, detailed information about the anatomy of individual cells and allows us to start thinking about the circuits.
A lot of the connectivity work, like when I showed you that subway map of the different brain regions, is done on bulk, right? We put a bunch of indicator into one brain region and we look to where that goes. But you don't necessarily get a view of how individual cells-- like, does every cell in Region A all project to Region B in the same way, or do we have a subset that goes to Region B and a subset that goes to Region C?
And this type of morphology allows us to pull apart more discrete circuitry of the brain. And so we're collecting a lot of this type of morphology data to give us these single cell, brain-wide reconstructions. You can see there's a lot of-- cells project to a lot of great regions.
And we can use this to now map out regions that are connected into specific circuits that we can now target for either electrophysiology or optical physiology where we're no longer restricted to the cortex. We're no longer focused on the visual system, but really looking brain-wide.
And we can combine this with optogenetics. We can combine it with a bunch of behavioral paradigms that are focused a bit more on decision making. And so working on foraging tasks where mice are being asked to choose where to forage for rewards.
So these experiments are kind of up and just-- right now, kind of up and running. And we're starting to collect a large amount of data both in terms of the morphology, but also with this brain-wide anatomy.
And we're taking a slightly different approach to how we share this data. So rather than packaging everything up into a single data set with a bow on it and a set of software tools, our data is open as we collect it and in our bucket. And as I said, we're kind of just getting started, so you're welcome to go see what's in the bucket, but you're probably not going to be able to make heads or tails of it until things are kind of a little further along.
But I just kind of wanted to put that on your radar so you could get excited about more of these types of observatory-style data sets that are coming out in the future. All right. So I will wrap up there. I just saw a question over here, if you still have it.
AUDIENCE: Single cell morphologies.
SASKIA DE VRIES: Yeah.
AUDIENCE: Did you say you could sparse label neurons and then do expansion microscopy? Was that what you were showing?
SASKIA DE VRIES: Yeah. So it's sparsely labeled neurons. And again, this comes down to a lot of molecular biology where there's a couple of different approaches that go on. Part of it is we've got these promoters, and you can kind of do intersectional promotion strategies so it kind of reduces-- like, I can have a promoter for all of the Layer 4 cells in cortex. But I can cross that with another promoter that's more selective so it will reduce it.
There's also tricks that-- and I don't-- I'm not-- I have a degree in molecular biology, but I'm not a molecular biologist, so don't tell my college that. But there's techniques that basically just make the expression more sporadic. So for a given promoter, it will reduce it to maybe less than 10% of the cells actually expressed.
And so we use strategies like that to just-- yeah, instead of expressing it in thousands of neurons in the brain, we're expressing it in, like, 100. I might be off by a factor, but enough-- it's reduced enough that you can actually trace individual neurons in the cortex. Right.
AUDIENCE: Do you trace manually?
SASKIA DE VRIES: Yeah. So the annotation is done manually. And so there's a lot of steps of stitching the images and registering them. We register it into a common coordinate framework so that we can actually say which brain regions all of the structures are in.
And then we have manual annotation. And this is actually probably starting in a few months will be something that if you're like, I really care about brain structure x, we're going to have a tool where you can come out and-- not come out-- and get on your computer and pick cells in that brain region that haven't been annotated and contribute that annotation into the data set so that we can-- if you're waiting for us to do the annotation and we don't care about your brain regions as much as you do, you might be waiting a while.
Whereas if you're like, I'm really motivated to do this annotation, we'd love to have you contribute and collaborate in that way. So we're developing tools to make that a possibility. All right. Other questions? Yeah?
AUDIENCE: Is there any future plan to move to another species for your-- other species? Monkeys?
SASKIA DE VRIES: So we don't have plans to work in other species. We're really focused on the mouse, and yeah. The brain-- The Institute for Brain Science doing this spatial transcriptomics, they're doing some cross-species comparison work. But on the physiology and behavior side, we just work in mouse here. Yeah?
AUDIENCE: So just a shout-out. This is the most-- I would say the Allen Institute and their efforts are the most remarkable achievements in neuroscience [INAUDIBLE] so really great job. It really is incredibly impressive. If you haven't seen any like this [INAUDIBLE] transcriptomics, cell type library is just-- it's mind blowing, really.
And also, shout-out. They put up-- you had some data from Josh Siegel up there. [INAUDIBLE] some of this data. And one of the points I was making in my talk was the imperative for embracing kind of ethology, allowing rats to do what they do. He discovered place cells by basically letting-- taking the rodents out of the [INAUDIBLE] allowing them to move freely.
So thinking about the importance of naturalistic behavior, I'm just wondering how you're thinking about incorporating that. In our own experiments, I showed some of our early visual cortical data where you see spatial [INAUDIBLE] and animals. They can actually move and navigate.
And in our own experience with doing visual mapping, when we took what I refer to as a primate-centric view, displaying front [INAUDIBLE] forward facing distal cues. Similarly, that very kind of [INAUDIBLE]. But as soon as we put the cues down on the floor-- that is, the lower visual field where a lot of the sort of group visual action is, dramatic change in visual responsiveness. That's where you see the spatial specificity.
So I'm just sort of wondering how you're thinking about embracing [INAUDIBLE] and naturalistic behavior. You pointed to foraging, and that's kind of good. Any other thoughts about how you might add that kind of behavioral complexity to understand the real function beyond the standard model, which is sort of a coding model? How do you represent objects? But in fact, the real model is, how do I use visual information to guide behavior?
SASKIA DE VRIES: Yeah. No, that's a great question. And I think the things that we have going on are-- so these foraging behaviors are much more flexible, right? Like, you can ask questions about foraging in a lot of different-- it can-- the types of cues that are used to provide information can kind of span different modalities.
And so one of the things that's going on there is that we've got a couple variants on these tasks that can use different types of stimuli as information. Some of them really don't have any stimuli as information. Some of them are more virtual reality-based where it's kind of simulating a mouse exploring an environment where there's different cues and different spaces rather than just kind of you've got left and right choices.
And there are preliminary plans to move into open field types of experiments, which I think will happen in the behavior before we get the physiology in there. But I think we're hoping-- like, our plan is to get the physiology, doing some open field physiology as well.
So we definitely are moving in that direction, but it's a direction-- like, we're well over time, so I have lots of thoughts and feelings about ethological behavior. And it needs to start on the behavior side. You really need to start these experiments behaviorally before then trying to go in and do the physiology and trying to understand what all these different things mean.
And that's how we're doing it now, in large part because the technology is just more complex. But yeah, but that is a really important part of all of this because all of these behaviors are really kind of-- are pretty arbitrary for the mice, so yeah. Yeah?
AUDIENCE: Just a quick question. So is it a standard practice to do functional recordings of any kind of electrophysiology [INAUDIBLE] before you do structural studies on the same mice?
SASKIA DE VRIES: Yeah. So we do-- we do have a couple of data sets where we've done calcium imaging followed by EM. And so one is part of what's called the MICrONS Consortium where actually, we didn't do the calcium imaging. A team at Baylor did the calcium imaging, and then we did the EM, and then Sebastian Seung's group at Princeton did a lot of the image-- the image processing kind of part of it.
And we have a second data set. That's the same thing, but all done in-house where we did calcium imaging with very similar stimuli as what we used here. Not the full set, but a subset. And we imaged a cubic millimeter of V1. And so that takes, like, 30 days of imaging to get all of that. And then did EM on that tissue.
And these data sets are monumental to collect, and then the effort in collecting it is heroic. The effort to process the data and register is like-- like, conceptually it's easier, but it takes so much longer.
And so those data were recorded, I think, in 2018. Like, the calcium imaging was done in 2018. We just have maybe 50 cells co-registered between the EM and the calcium imaging. I might be underestimating that. It's maybe 200. But there's thousands of cells in that volume.
So there's been a lot of analysis of the calcium imaging by itself, but that's not interesting compared to combining it with that structural data. So that data exists. I don't know if it's-- I don't know that it's been released in any format, but I think just because we're kind of waiting for the EM to be able to co-register it better, but all of that exists.
And the microns data set is-- if you haven't heard of it, it's a data set. Like, if you go Google search MICrONS Consortium, you'll find it. And that's been out and widely used.
AUDIENCE: Let's thank Saskia.
[APPLAUSE]
have an interactive transcript feature enabled, which appears below the video when playing. Viewers can search for keywords in the video or click on any word in the transcript to jump to that point in the video. When searching, a dark bar with white vertical lines appears below the video frame. Each white line is an occurrence of the searched term and can be clicked on to jump to that spot in the video.
