Insights into AI algorithms drawn from hippocampal function
Date Posted:
December 2, 2022
Date Recorded:
November 4, 2022
CBMM Speaker(s):
Matt Wilson ,
Ila Fiete All Captioned Videos Advances in the quest to understand intelligence
Description:
Matt Wilson - MIT BCS, MIT CBMM
Ila Fiete - MIT BCS, MIT Quest, MIT CBMM
PRESENTER: Ila Fiete and Professor Matt Wilson, they're both colleagues in the Department of Brain and Cognitive Sciences. They're going to share some research on hippocampal function in the brain and how it relates to and can potentially guide new AI algorithms. So I think, Matt, you're going.
MATT WILSON: Thanks, Jim. So yeah Ila and I are going to talk about both experimental and computational work that hopefully provides some insights into principles of biological intelligence that can both inform and be informed by artificial intelligence, again, the grand scheme of things. And the system that we're going to really focus on is shown here the hippocampus.
The hippocampus as [INAUDIBLE] pointed out, one of the hallmarks of biological intelligence is the ability to rapidly learn from experience. And the hippocampus shown here at the pointer shown here, this is a rat brain but in the human. You see the hippocampus there as well. It's a evolutionarily conserved structure. That's really critical for the formation of memory of experience.
But it's also involved in forming memories of space and is critically involved in spatial navigation. And the parallels between the two episodic or experiential memory and spatial memory is something that was explored by John O'Keefe early in the 1970s where he made a discovery that led to the Nobel Prize in 2014. And that is by recording from individual neurons in the hippocampus he discovered that these cells had spatial receptive fields.
An animal moving around in space, cells would fire in different locations creating what he described as a cognitive map. And that is as Leslie pointed out if you embed priors into your models that can actually advance learning. And it appeared that rats as well as humans were able to form these spatial priors and then use them for learning and navigation.
And this is a little movie just illustrating the phenomenon Ed O'Keefe had discovered. And you see here this is a track little rat shown in green. What we've been seeing here, there are neurons that were recording from using multiple electrodes. And these cells will fire as the animal moves along the track here. And you hear this nice well do you hear this kind of discharge this kind of pop, pop, pop, pop, pop, pop, pop, pop, pop
[LAUGHTER]
And different cells will fire in different locations. But there's another way in which we can illustrate this property. So we'll see if we actually use an approach instead of showing you the activity we'll simply decode the activity and ask the question given ongoing neural activity, what would the location of the animal be in order to produce that pattern.
Well, we see a little triangle here, if we use this Bayesian decoding algorithm. The neural activity would track the position of the animals and moved-- this is the place code. Hippocampal code tracks the animal's location in space. But something interesting happens when the animal stops, and as the code seems to go away, no longer reflects the animal's current location. But instead, will jump to remote locations on the maze.
What you would see is the animal would stop here and then suddenly the hippocampal code would rapidly-- over the course of about a half a second would follow a trajectory in space, moving forward ahead of the animal, and will be thinking about where it would be going in the next several seconds. And that replayed memory would have a structure illustrated here, and that is you have a long sequence, about five meters in length, broken up into smaller segments. So that memory of experience broken up into small segments re-expressed in the form of these time-compressed trajectories.
And the idea that these trajectories reflected a future location that the animal might go to suggests that perhaps they're involved in planning, animal thinking about where it's going to go. And indeed, if you look at this activity in the context of a task, the animal's now, in an open arena. There are many locations they can go, many places they can think about, but these replay trajectories seem to converge on a target goal location. If it needs to get to a goal, it thinks about where the goal should be.
There's another form, though, of replay, and this form of replay also takes the form of trajectories. But these replay events can move backward in time. And that is as though you are rewinding an event, rather than playing it forward. And these reverse replayed events are interesting because they, in principle, could solve a problem in reinforcement learning known as the temporal credit assignment problem.
This is what you encounter in trial and error learning. You're trying to do something, for instance, the animal wandering around a maze, suddenly it encounters the exit. Now what it needs to know-- what it needs to learn is not simply where the exit's located, but what were the sequence of actions that led up to that. And as you have to propagate value information backward in time backward along the path. And in principle, pairing this reverse replayed event with a reward signal like dopamine could solve that.
And indeed experimentally, if you record in the reward areas the ventral tegmental area that provides dopamine, what you see is that these replay events, both forward and reverse, are paired with reward signaling. So when the animal thinks about going to the goal, the reward signal internally is reactivated. So together you think, OK, replay, it's involved in planning, maybe it's involved reinforcement learning. It's related to reward.
But this reactivation occurs in a different context, and that is during sleep. So we'll take the animal off the maze, no longer performing a task. It's sitting here in a quiet little chamber sleeping. And during sleep, you see the same kinds of events, forward replay, reverse replay.
And we know that these replay events actually do contribute to learning. If you disrupt the events, either in quiet wakefulness or sleep, it impairs learning. If you enhance, them there are a variety of methods for doing that, you can improve learning. So quiet wakeful sleep-- quiet wakefulness, sleep, replay involved in learning.
But there's an interesting property to the replay during sleep and that is it is not paired with reward signaling. In fact, the dopamine reward system is explicitly suppressed. Now that doesn't mean that reward learning doesn't work during sleep because if you synthetically-- if you go in and activate the reward system during sleep, you can get animals to learn things that they never experienced during wakefulness. You can implant reinforced memories during sleep.
So this says reward learning can work during sleep, but we're not going to do it. Well, what kind of learning might go on that doesn't involve reward? Well, it turns out the hippocampus is involved in this form of learning, which was explored by Edward Tolman back in the 1940s. And what Tolman observed was animals simply allowed to explore environments learn something that later, if they were asked to perform a task, led to improved performance. What was it that they were learning? He termed that kind of learning latent learning, without reward through experience.
And what they learned, he suggested in this case, was the formation of a cognitive map, which, you might recall, is what O'Keefe argued he had discovered by recording place cells in the early 1970s. So we asked what would a cognitive map look like if it was formed just through experience?
And so using calcium imaging approaches, putting a little camera on a rat's head allowing it to explore simple mazes like the little T-maze here, and then using an approach those manifold learning to take a high dimensional, the activity of many neurons, and then map them down into an easily visualizable space. In this case, a two-dimensional space.
And if we do that as an animal explores a number of environments, so here you can see an animal exploring a T-maze, a square maze, an H-shaped maze, what you see is this internal-- the mental representation. This is formed with any knowledge of the space. It's just by performing this manifold dimensional reduction of the neural activity.
You see that the mental map comes to conform or share similarity with the physical map. This is something that develops with experience, and that is over multiple sessions you see the similarity increase. So the similarity starts off poor and then you see that the mental map comes to reflect the physical map.
But interestingly, this change in this internal cognitive map is sleep-dependent. And that is that if the animal doesn't get to sleep, there's no change, no change in the map. So something that it's thinking about or dreaming about during sleep is leading to a change in the map that makes it reflect the physical space. What is it about the neural activity that's actually changing?
It turns out there are two types of cells that participate in this map formation. One, what we refer to as strongly spatial shown here, these are the place cells. And over experience, they largely do not change. But there's another set of cells, the weakly spatial cells, and those do change. And what changes is the correlational structure.
You can think of these weakly spatial cells, as cells that reflect information about cues. They can reflect information about the movement through space, or the topology. And so with experience, these cells come to conform, or map onto the spatial manifold.
And what the general model that we think explains this is that animals have this internal cognitive map, a spatial map, that reflects the locations in space, the topography of space. With experience, it incorporates information about nonspatial information, about cues through exploration, incorporates it into the map, forming a topology that reflects ultimately what we refer to as the cognitive map, space plus experience.
And then in general, if we take a powerful unsupervised learning algorithm like MuZero developed by DeepMind, which is capable of learning things through experience, we can see that there are these functional homologies, the need for representational learning, which we see in the development of the cognitive map, the use of prediction in future action and reward estimation, as well as offline search and simulation in the search of novel states or solutions. So with that, I'll turn it over to Ila.
ILA FIETE: Thanks, Matt. All right. Well, welcome those of you from out of town to beautiful Boston today. So I'm going to tell you a little bit-- so I'm a computational neuroscientist, a theorist, a background in physics and math. And what I try to understand is some of these aspects that Leslie, and Josh, and Jim have mentioned, which is how it is that the brain performs very data-efficient on-the-fly learning and inference. And so this is using some of the elements that Matt has introduced.
So the hippocampal system. Matt was not boastful enough, I assume-- I think about the hippocampal system. I like to say it sits in the very top of all the sensory hierarchies in the brain and then it drives, in turn, downstream outputs that ultimately lead to actions in the world. So the first statement is first of all, hippocampus is structuring our understanding of the world. It is the seat of episodic memory. It is also the seat of spatial learning and memory.
And an interesting observation is that as we move through the world, we sample it continuously. We just experience a time series. As we go through the world, we see things, we experience things, and so on. So it may be that you take the circuit through some space experiencing and observing various things. And as we've already seen, the hippocampus has representations of the things that you've seen. It tries to organize them in some way.
The slow learning, as Matt showed you, you can get low dimensional representations within the hippocampus. As an animal gets to learn a single environment over lots of experience, it gains this low-dimensional structure. But the point is that initially, hippocampal representations are extremely high dimensional. So you're taking these objects and observations and you're embedding them somewhere in this very, very high dimensional representational space.
And the question-- OK, and so that's just an observation, very high dimensional space. This is the circuit, this might be the connectivity. The hippocampus tends to represent things as a time series, what led to what led to what.
So it's kind of just representing these sequences. But we have these additional abilities. Memory is not simply memory, but it's the ability to make inferences like, for example, now that I'm over here, what is the direction in which I should go in order to land up over here, back to the first place that I went?
So I have this spatial understanding of the structure. I don't just represent things in the world as a sequence. I understand how they relate to one another and the structure.
So the question is if, in this high dimensional space, the structure is not organized, it doesn't reflect this low-dimensional structuring of Euclidean space, then it's not clear how to build the inference. If I were to take a step in this direction, where would I land up? In the high-dimensional space, it's not clear at all.
So what we are discovering about the brain and about how the brain solves these problems is that the brain contains-- it actually takes the world and it forces it into-- it's like pushing square pegs into-- square pegs into round holes all the time. So that's what is also called inductive biases. And it turns out that the brain has this real bias towards factorizing the world into simple low-dimensional representations. And I want to show you one example of such a low dimensional representation.
So in all the experimental paradigms that I'm going to show you over here, the situation is the following, which is the animal's running around in a two-dimensional box, totally doing whatever it wants to do. It runs around, eats Cheerios, stops, looks around, scratch itself, rears up on the walls. Very, very rich high-dimensional behaviors.
So this is not the case of it exploring some very low dimensional thing in a low dimensional way. But nevertheless, if you look at this one circuit that feeds into the hippocampal system, the thalamic circuit, what you find is that the circuit of thousands of neurons, about 2,000 neurons, the states across all of those behaviors localized to just this 1D ring, or manifold. And in fact, we can parameterize along this manifold and then color the states and the manifold according to this parameterization by fitting this low dimensional spline to it.
And we can then completely in an unsupervised way decode states along this manifold and then generate a curve of state versus time. And it turns out that when you look at state versus time here, that's in blue. But then you look in black, and black is actually the physical direction, compass direction, in which the animal was facing during the times that it's running around.
And so in other words, this set of internal states is representing a compass estimate of which way the animal is facing and it updates even in cue-poor and dark environments. In other words, it's really an internal construct where it's inferring where it is based on its movements.
So this is an internal compass. Here is a video of what this looks like. Actually, the same system during sleep. This is during REM sleep. And it turns out that the states of this 2,000 neuron circuit, again, localized to this one-dimensional ring and nothing else.
So even in sleep, these states are just organized on this simple one-dimensional manifold, and it turns out that this one-dimensional set of states is the same as the blue states down here, which are the waking states. So the brain really wants to factorize representations about the world into this one-dimensional variable. There's a second low dimensional variable representation that the brain has, and this is these very striking cells, the grid cells, which were discovered in 2005 and got the Nobel Prize in 2014.
And these grid cells are cells. This is the response of a single neuron as a function of the trajectory of this animal as it's exploring the box, and doing its thing, and scratching itself, and all of those things. And you can see that the single cell is firing in a tremendously remarkable pattern. It's doing this periodic firing.
It's firing at these spatial locations, which are on the vertices of a regular triangular lattice. It turns out that when we do the same of-- if you do the same kind of manifold analysis of the states of these grid cell circuits, they lie, again thousands of neurons, in fact 10,000 neurons, that are then representing space as a set of two-dimensional states. But because of the periodic representation, these two-dimensional states are locally Euclidean, but then globally a [? torus. ?]
So I just want to then tell you how these biological discoveries, this discovery that the brain is taking all of this rich data, movement, and experiencing the world, seeing sensory inputs, and reducing them to these abstract things like I know where I am in this two-dimensional space because the cell is indicating that where you are as you move around in this two dimensional space, what is the utility of these kinds of codes? What are they used for? So it turns out that these kinds of codes provide a way to structure the world.
So these grid cell representations are invariant across waking and sleep. Just like this compass, head directions cell representations are invariant across waking and sleep. So they provide a scaffold, so to speak, with which we interpret the world. So we take all our observations about the world and we kind of hook them onto these internal states. So as we move around the world, these internal states are updating based on your movement.
They're activating a certain pattern of states as the animal moves around. And then it takes sensory data and experiences and hooks them onto those. So this is like a clothesline kind of analogy that I would make. So you've got a clothesline, which is your internal structure, your scaffolded states of grid cells, of these head direction cells, and you're taking your experiences and just clipping them on to the structured scaffold.
So now once you've done that, now these aren't randomly scattered in this high dimensional space. You're actually inducing some spatial relationship to these observations and it allows you to do this zero-shot inference without-- which means zero shot, which means you can take all of these observations, and having never traversed this route before, you can then predict, if you had traversed that route, what you're going to see at that point.
So this is a way of inducing structure into the high dimensional-- in these high dimensional spaces that are used not just for spatial representation, but also memory. So the other thing that this kind of abstract representation buys you is it's actually a very ingenious, it's a bottom-up revelation of a new type of coding scheme that has actually multiple periodic codes. These are grid cells of different scales at different parts of the brain, and together these grid cells of different scales combine to represent locations with a range that scales exponentially with the number of neurons.
So rather than having linearly many states that you can represent as the number of neurons grows linearly, instead this kind of coding scheme allows for exponential growth in your coding states as you increase number of neurons linearly, which is very important because our spatial-- our experiences in the world are rather large and vast, and so this is a way to organize very large spatial experience. The other interesting thing about having these kinds of very structured representations is the following.
So suppose that I've got these internal states in the brain that are rigid. So this is the brain. Here is the external world. And I've got some external variable in the world.
And the way I represent these variables in the world mentally is by mapping those variables in the external world onto these internal states over here. So I could visit each one of those states and exhaustively then build this one-to-one correspondence, but that's very slow and very inefficient, and that's why deep neural networks take so long to train and learn.
On the other hand, this scheme that I just showed you, these grid cells and head direction cells, these internal representations can take a velocity input like a movement input and update states internally. So now if you have an external variable and you move through that world in this external variable space, and we've at the same time got a way to move on this internal manifold.
And if you can now just anchor one state in the external variable with one state in the internal variable, and if you can build a correspondence between movements or velocities in the external world and velocities in this internal states, then once you've built this one anchor, you can, again, on the fly generate and predict what the code should be for all the remaining states.
So this is a way to do very data efficient inference. And in fact, it turns out that the same neural circuits as grid cell circuit is used to represent conceptual variables that are not spatial. And I'm just finishing. So they're used to represent conceptual variables that are not spatial.
This is a monkey free viewing an image and these same grid cells seem to be active in a grid-like pattern as a function of the eye-- where the gaze is falling on this visual image rather than as a function of animal position as it moves around the world. So in other words, just like I mentioned in the previous slide, the brain is repurposing the same low-dimensional network and representations to now represent something very different, which is gaze position in an image as you're freely viewing.
Similarly here there's a cartoon image and humans are moving a joystick. And as they move the joystick north, south, east, west, then the neck or the legs of this cartoon bird stretch. And it turns out that the fMRI signatures, neural signatures of representation of where you are in this conceptual cartoon bird space seem to also reflect the same kinds of modulations as they do when an animal is traversing the physical world.
So that's basically what I wanted to tell you. And so I will just conclude by saying that it's possible to extend these metric structured representations to also represent non-Euclidean spaces by now constructing chunks that are Euclidean and then piecing them together in a more complex cognitive map. And so the idea is that the hippocampus takes these locally Euclidean representations and then stitches them together to in a topological way. So now you get these topometric maps and representations that represent information in rich ways.
So just to summarize here, there are many low-dimensional representations in the brain. This is the brain constructing abstractions and concepts. These representations are an invariant scaffold that are repurposed again and again for memory and for inference. And these circuits generalize beyond coding, and these fragmentation processes allow for representing locally Euclidean, but globally non-Euclidean objects as well. So thanks.
PRESENTER: OK, thank you. Thank you, Ila and Matt. I want to just make a connection here to notice these structured representations that they're talking about that they can see in the brain. This is a theme that you're seeing in many of the talks and will be echoed again later.
So this is what's actually happening at the neural hardware to build these structures that get gives us these extra inductive powers that we're not able to have without those kind of structures. So always want to call it out when you hear the words world model or structured representation. This is what we're referring to, and this is a beautiful example of that in this navigation situation. So thank you guys both.
have an interactive transcript feature enabled, which appears below the video when playing. Viewers can search for keywords in the video or click on any word in the transcript to jump to that point in the video. When searching, a dark bar with white vertical lines appears below the video frame. Each white line is an occurrence of the searched term and can be clicked on to jump to that spot in the video.
