A Conversation with Josh Tenenbaum
Date Posted:
September 1, 2022
Date Recorded:
August 16, 2022
CBMM Speaker(s):
Joshua Tenenbaum All Captioned Videos Brains, Minds and Machines Summer Course 2022
Description:
Part of the Brains, Minds and Machines Summer Course 2022
Links to videos to watch as prerequisites for this discussion:
PRESENTER: Good morning, everyone. Our speaker this morning is Josh Tenenbaum. Josh is a professor at the Department of Brain and Cognitive Sciences at MIT. I said speaker, but in fact, this session is planned as a conversation. You all watched videos of Josh's talks, and I guess, Josh will be happy to answer any questions you have about those and anything else in the world as he knows everything. So Josh, I don't know if you want to do some intro.
JOSH TENENBAUM: Yeah, I might just do a very brief intro. If that's OK. I thought it would be an opportunity to try something different. And I hope actually valuable. So I suggested to Boris a few different talks of different lengths and different kind of focuses that I have given recently on covering the broadly the same kind of material that I would present in a traditional set of lectures in Woods Hole, but with different focus.
So one of them has more of-- there was a computer vision keynote talk for people interested in vision. There was a short thing focusing more on common sense and social intelligence. Another talk focusing on the question of what kind of computation is cognition? And broad perspectives there, so hopefully, you watched one of those. Maybe you watched more. If you didn't, that's all fine too.
The thing I thought I would just say for a couple of minutes at the beginning is maybe to provide a perspective on the work that, not only the work that I do, but the work that my whole field does that I represent, and how it's complimentary and some ways contrasts and other ways deeply similar, but how it complements, I think a lot of the other work that you hear about in the summer school.
So this is a summer school on brains, minds, and machines. And that means there's people with three different goals. That doesn't mean that there's three different groups of people. There's lots of-- everybody is their own person. And all of us are here because we're all interested in all of these goals. But some people come from different perspectives, and so emphasize more one or the other. Often, we come from one as our core focus, and then we are connecting to the other two.
So what are these goals? Well, there's-- try to understand how the brain works. That's a-- brain is a physical machine that produces intelligent behavior. That's broadly what neuroscience is trying to do. Another is to try to understand how the mind works. The mind is not physically distinct from the brain. We think of the mind in some sense, as like could think of it as the computations that the brain does.
You could think of it as the software where the brain is the hardware, but however you want to think about it, the mind-- or you could think about it the way people have thought about it for a very long time, not just thousands of years of recorded history, but, surely, well before that, the way we thought about minds is the way that we have thought about each other. Humans have only very recently started to understand anything about brains, but for a very long time, and I mean as part of our evolutionary history, we humans understand each other with what sometimes called a theory of mind.
Our brains have models of each other's brains, but it's not traditionally in terms of neurons or synapses, right? Because humans didn't know about those until quite recently. But for many, many, many eons, humans have had models of the brains of other humans that have supported our social interaction, our linguistic communication, what we are doing right now even though we're not in the same place, and I can only kind of see from a weird side angle.
But I can imagine a lot of things, and hopefully you can too that allow us to communicate and hopefully to make deep contact in whatever discussion we wind up having. That's a remarkable ability. It's in this case, supported only by technology, or supported necessarily by recent technology.
But just the ability to make some kind of connection of information, and hopefully something deeper than that with other human beings rests on fundamental abilities of the brain, which if you want-- if you want any non subjective notion of what a mind is, then think of it as that. The model that brains have of other brains that allows them to do this kind of thing, and as important as anything that the brain does.
This is a view of minds and theory of mind, which Rebecca Saxe one of our CBMM colleagues has very nicely articulated, and I'm just kind of borrowing and adapting it here. So that's kind of the mind's perspective. That's where I come from both. I like talking about. And then there's the machine's perspective, which is how do we build artificial machines that perform intelligent behaviors that are at least in this context, that are human, human like, human level?
We'd like to build machines, AI systems that can do the things that our brains and minds do, and we can be interested in that at the hardware level, the software level, and building a machine requires understanding all of those things and many levels of abstraction across those. So what makes this summer school so exciting is that we have people representing all these disciplines, and who are really interested in connecting and drawing inspiration on each other.
But what I think you'll see in the work that I do, some of the other people coming more from the mind or cognitive perspective, is a different-- a difference-- in some sense a different set of approaches or orientation to what we're trying to do. I should say also, which is implicit here, is that everybody in the summer school, we use a range of different methods, different experimental methods, computational methods.
Everybody here is either directly working on or collaboratively working on, or really committed to a computational perspective. So what you're getting in the summer school are computational ideas and tools for thinking about brains, minds, and machines. In addition, perhaps to exposure to some kinds of experiments. Some of you might be running behavioral experiments, or maybe even looking at neural data and so on.
But the computational toolkit that we develop when we approach brains, minds, machines from a minds perspective, looks somewhat different from the ones when you come from a brains perspective, and especially these days, a lot of the work in AI and machine learning that you might be hearing about, a lot of the most recent attention is on approaches that at least superficially, and in some cases quite deeply, maybe, are more brain inspired, like neural networks basically, artificial neural networks.
Although if you look at the field or more broadly and in over decades perspective, or places maybe outside of brains, minds, machines, you'll see a tool kit that also is more in some sense, cognitively based. But just to contrast somewhat, let's call it the neural-- biological or artificial neural network perspective, which certainly has been very valuable in the brains of neuroscience perspective and in a lot of kind of brain inspired machine learning.
What are we doing there when we use these neural networks, which I think many of you are-- you've seen from any of the other speakers before. Many of you might be using in your projects or your own research. So you're treating the brain or a machine as an input-output device. We can observe the inputs coming in. We can get cameras, or see the images, for example. We observe the outputs, and we're trying to build models that model the input-output relationships.
The functional relationship between inputs and outputs, and see that as a way to produce intelligent behavior. If we can capture the input/output functions of the human brain, right? Then we will have a machine that does-- it at least gives a one plausible account what's going on inside. It could be interpreted as a scientific hypothesis of how the brain works, or it could be used as a framework for getting robots or other AI systems to do the same thing as the brain.
And we're in the summer school and in CBMM at MIT and beyond, we're deeply committed to the idea that models of the brain and models-- AI systems for human computing will be deeply related, in some sense, the same thing. So if you treat the system as an input-output function, and especially if you think of it in terms of a physical machine and like what can we learn about the computations from thinking about the circuits, then that very naturally leads you to several theses, basically, which is what you see in neural networks.
It leads you to the idea that the computations should be most generally approached as some kind of function approximation because there's a function-- unknown input-output function, sensory data and behavior out. What is that function? I don't know, but I'm going to try to approximate it with some device. What is that device? Well, deep neural networks are very good functional approximators, right?
They can approximate arbitrary functions, and they are trainable in a large scale efficient way. So if we have input-output data examples of some plausible kind, then we can train these systems to approximate the input-output function, and thus provide hypotheses about scientific models and ways to get machines to do the same thing.
So that's one approach, and I should say again, it's implicit, but these neural networks are both mathematically, they are very good function approximators in some ways. But they also-- because they use some of the same kind of circuit elements at a certain level of abstraction, and I mean by that like linear sum of inputs and nonlinear thresholds, but also, the ideas of convolution and multilayered architectures and so on.
Things that draw on cellular motifs and systems level motifs from neuroscience, then that helps to make them seem and be more plausible models of the brain. So that's one approach. And I'm not against that approach. Some times I use that approach in collaborations with some of my colleagues here, but the cognitive science approach is a complementary tool set, which, again, we're also-- we also look at inputs and outputs of the brain.
But thinking in terms of software level and the level of minds, like I was talking about the way that our brains model each other, and have always done that. It kind of leads us to a different set of theses, right? I mean, on the conceptual side, this idea that brains have models of other brains, and you might also say we have models of our own brain, our sense of self, right? That guides our introspection. That also is an important part of being human.
And a lot of when we talk to each other, we talk-- when we talk about what's going on inside us, right? Again, this is something humans have done for a long time as part of communicating, connecting and teaching, right?
So if you think about how we introspect, or how we communicate with each other, it leads you to focus not only on the input-output relationships, but on more of the inner stuff, right? Like more of what's going on in between. We have concepts about how minds work. We talk about sometimes traditionally beliefs and desires. What do you want? What do you think? Why don't people-- why don't these other people agree with me? Or, well, I understand where you're coming from, or I'm really trying to figure out what do you-- what are you really looking for?
When we talk that way, it's because we're thinking in a certain way. It's because we are thinking about each other's thoughts. So from the cognitive perspective, you're not just interested in the input-output relations, but you have certain-- you call them preconceptions or biases, or just call them concepts about what goes on inside minds or brains. And you might say, well, from a scientific perspective, shouldn't we ignore that?
Psychology tried that for a while. It was called behaviorism, and it didn't work very well. I'm not going to go into that history here, but both prior to the early 20th century, and after about the middle of the 20th century, the main focus in psychology and the study of cognition has been to study both the input-output relations, and then to have a certain paradigm of in some sense, how thought works.
That mind and here's sort of the basic assumptions, and if you-- in the talks that I gave you, you can see all of these in play, right? Our minds-- the fundamental thing we do is we don't just approximate functions, and then like train those functions from data to recognize patterns in the hopes of generalizing beyond the training data, that's like the machine learning perspective. But we do more than that. We build models of the world. You might hear about mental models, or world models. But our minds build models of the world, and that includes one of the main things that's in the world, which is other minds.
So we build models of the physical world, and of the other agents, could be humans, but also, animals, or maybe robots and AI systems, or aliens if we ever meet those. But we build models of the physical world at each end, the other agents that are there, and indeed of our own self in the world. And it's these mental models that are the basis of our intelligence. They're the things that allow us not just to find patterns in data, but to really understand and to explain what we see to be able to imagine things that maybe we haven't ever seen, but our minds models can simulate those in some ways.
And then that allows us to set goals and make plans to achieve those goals. Goals which could go way beyond, vastly beyond anything that was ever rewarded in our experience like if you think from a reinforcement learning perspective. The goals that humans adopt from themselves for themselves are stunning in their variety, and here I'm very inspired by the work of Laura Schulz, another one of our colleagues, who is really emphasized in her recent work, the vast and dizzying array of human goals and what that tells us about human cognition.
But, I mean, again, I don't think that I need to go into this here. Just think about any of the hobbies that you or your friends have, or what you spend your days doing as a scientist or an engineer, and think about how much that goes beyond anything that was directly reinforced in your developmental, or evolutionary history, and it's quite stunning.
So humans have this ability to build models of the world to make goals, to imagine possible ways the world could be that it isn't, to set certain things as goals and then to make a plan of how to get there. And plan may not work, problems may come up along the way. You might have to debug or fix the plan, but those are all parts of human cognition. And then from this point of view, learning is all about how we build models of the world. How we revise them. How we combine them, how we make new models in cases like when I build a-- when humans have built models of quantum mechanics or neuroscience.
These are all things that evolution didn't give us. We had to make these models, and we make them not only for ourselves, there's an individual process of discovery and model building, but much of it, some might say, the most important parts of how our minds build models of the world is a cultural and social process. So we build models culturally across individuals, across generations. And then we learn and share them socially through communication like what we're doing right now.
So the cognitive perspective is to try to understand all of those things and to be, of course, interested in the hardware of the brain but really thinking about math and engineering tools that come more from software. So that includes the theory of computation, it includes algorithms, data structures, programming languages. All these other aspects of computer science, which we would say are more about the software than the hardware level.
We're also interested-- and I think this is one of the most exciting frontiers in compilers, namely formal tools that link up either different levels of software abstraction or the software and hardware levels. Because ultimately, we want to understand how does the mind work in the brain. And that's going to require taking the software levels of abstractions, like algorithms and data structures, and compiling them into circuits in some ways or understanding how that works.
So these are the formal tools that influence where I come from and that we build on. And what this has led to is the set of ideas that you can see in each of those talks. There's a lot of technical stuff there. And all we can do at a summer school like this is just kind of advertise it and very basically introduce it. But if we want to take one piece of jargon to sum things up, it's this term probabilistic programs and probabilistic programming, which is kind of a cognitive parallel to the term neural network.
What are modern neural networks? Well, they start from a motif that was inspired by how neurons work. but then has scaled up into a very, very quite different thing that now you could say is something like scalable end-to-end differential programs. So probabilistic programs started from an idea that we and others worked on going back two decades or more that if you want to understand how minds are able to build these models of the world and use them to make sense of new data and solve new problems, well, the idea of probabilistic inference is central because the data coming in is just a little it's just fragments of what we need.
And if you want to understand how do our models work and how do we use them to interpret sparse data and how do we learn the idea of broadly a Bayesian approach, where you have your models of the world are expressed as probabilistic generative models that can be both priors to make sense of the data, but also as hierarchical probabilistic models that can be a way of how you can learn new models. Because you basically have priors on priors and hypothesis basis of hypothesis basis. And so model building is a form of inference in a hierarchically layered model.
And where the program's part of probabilistic programs comes in are tools from algorithms, data structures, programming languages that make this more scalable and more general. So the models are not just the kind of probabilistic models that statisticians have long built, but they allow us to express representations that are rich enough to capture the world, including the physical world, like the work we've been doing on intuitive physics that some of the talks talk about or physical scene understanding. That's like intuitive physics grounded in visual perception, but also theory of mind.
If we want to represent and think about other people's thoughts, that is not something you can do with a mixed effects linear regression model or a hierarchy the kind of tools that probabilistic modeling and statistics have long used. Rather it requires the ability to do probabilistic inference over algorithms for planning an action on perception. So the models we build for theory of mind are models in which we make inferences about the algorithms going on in other people's heads.
And that's what we mean by probabilistic programs. Probabilistic programming then also adds other engineering tools that let you scale these approaches up. And I'm not really going to talk about that here, but I'm happy to answer questions on it. So this has become increasingly a powerful engineering tool for building robots that can understand the world in more cognitive and flexible ways. And the Vision talk the CVPR keynote talk talks somewhat about that too.
So hopefully, that provides some context. This is a little bit longer than I thought I would speak for, but still most of the hours open for questions. But I thought that would be helpful to just frame broadly the kind of work that we're doing, how those different talks fit together and how it complements some of the other perspectives that you've been getting in the summer school. And now I'd be very happy to discuss any specific questions you have or general thoughts. So it's up to you guys.
AUDIENCE: I have a general framing question to start. I think your intro engendered a few additional questions if no one else jumps in after this.
JOSH TENENBAUM: Can everybody just say their name since I haven't met you in person?
AUDIENCE: Of course.
JOSH TENENBAUM: Say your name and maybe just where you are from.
AUDIENCE: Hi, my name is Leah-- my name is Leah Johnston. I'm a vision science PhD student at Berkeley. And it's great to meet you. What some of your videos were also kind of getting at, this idea of a theory of other people's minds and how I am moving around in the world, especially as a child building a theory of what to expect from other agents in the world gets to some of the things that we talked about and argued about a lot, like earlier last week about consciousness too.
And I was wondering-- this is kind of a zoom out question, but as you think about robots that will be predicting the actions of other agents and how comparing that to how a child moves around in the world and learns about the other agents around that child, how does consciousness inform your view of this theory of other people's minds and social learning and that kind of thing?
JOSH TENENBAUM: Yeah, it's a great question. Just so I get some context. So the discussions about consciousness, in past years at the summer school, we've had Christof Koch talk about consciousness. Was this inspired by-- was he there and talking about this? Is that the idea of this?
AUDIENCE: Exactly, yes. That was the talk that engendered a fair number [INAUDIBLE].
JOSH TENENBAUM: Yeah, I mean, Christof's perspective on consciousness and mind are pretty different that do of reflect these different perspectives. He, I think, comes at what I might say is, if anything, as a physicist. So it's one about thinking about-- and the Tononi perspective, and so on. It's like thinking about statistical mechanics type ideas, emergent phenomena. How do you measure or quantify consciousness the way you might have a measure like entropy or something?
Yeah, I think what that perspective is trying to get at is something about the coherence of consciousness. That consciousness is somehow some state of a brain or some other system that has some global coherence. And that's a very interesting one and important, I think. Whereas the cognitive perspective is, again, it's complementary in that it emphasizes the aspects of consciousness, which are really about this aspect of modeling a world where it's an internal world. And you're modeling both the external world and your inner world and your self in the world.
So I think for many people when they talk about consciousness-- like average people, normal people, not physicists-- what they mean is they're talking about something about the character-- not just the content, but the character of their brain's model of the world and of itself and that. So when we talk about qualia, what is the character of my experience? The redness of red?
Or when I talk about emotion or my own self consciousness, what is the character of my experience of myself? And you know this is clearly connected to the issues that I'm trying to talk about and that cognitive scientists talk. About but it's often hard for us to, I think, address the subjective character of our experience of the world or of our own internal processes because the tools that we have-- the engineering tools don't really have any room for that.
So I used to say for a long time that consciousness is really interesting. But the computational cognitive science toolkit doesn't have much to say about it.
Now I've changed that a little bit because after saying that in some talks, another one of the world's experts on consciousness, a scientist named Stan Dahan, who is a French cognitive neuroscientist, a long-time friend and colleague of Nancy's and Elizabeth Pelkey who's another member of CBMM and myself. He's written some books on consciousness and papers over a number of years, including early work he did on what was called the global workspace theory, which emphasizes also these kind of coherence ideas similar to Tononi and Christof.
But more recently, he's developed a perspective that also tries to integrate that with this modeling idea that I was talking about. And Stan has said that he thinks that the tools and perspectives we've been building could provide a computational model of his broad theory of consciousness. And I said, do you really think so? And he said, yeah, pick up my book and read page 235 or something. So towards the end of his book on consciousness, he talks about this idea of basically the computations that allow the mind to model itself and to model others in recursive ways.
And it's not like we've really developed this in any particular way. I'd love to do that. Stan and I have had a few conversations. I think it's just a pointer to the idea that perhaps the tools that we developed and talked about in some of those talks could provide a more, let's call it software or algorithmic level. One that doesn't address maybe the mysterious notions of qualia, but really tries to provide a very meaty substantive computational account of the key computations of consciousness as people like Dahan or others like Graziano, for example, at Princeton have developed. So his attention schema idea is also importantly related.
So I would say it's not something that I'm an expert in, but I it's very much worth people looking into. And I've had folks in my lab look at it a little bit. We'd love to look at it more. Is how these ideas could provide computational accounts of the Dahan and Graziano perspective. So if you're not familiar with those, I'd suggest reading each of their books and then we can think and talk more about this. But that's a perspective thing.
AUDIENCE: Awesome. Thank you.
JOSH TENENBAUM: Yeah.
AUDIENCE: I have a question here. That's fine. So it's also more of a general question. Arguably--
JOSH TENENBAUM: Say your name again and--
AUDIENCE: Oh, right. I didn't say my name at all. My name is Isaac Ashkenazi. I'm from Hebrew University.
JOSH TENENBAUM: Great. What's your field?
AUDIENCE: A PhD theoretical neuroscience, but I'm focusing mostly on modeling of cognition recently. And it's actually coming from this perspective that, arguably, part of the reason of a big part of the reason that the black box approach was taken is in one way, you can call it convenience in experimentation and modeling. But also you can call it scientific rigor. Famously, behaviorism came to oppose introspection and stuff that was seen as less grounded.
So when we take the approach of mathematical modeling of cognition via probabilistic models, et cetera, so it seems to be that we need to make some very strong assumptions. Both assumptions about the computation and algorithmic level as to Bayesian inference, et cetera. And when we try to define function, we very often appeal to optimality and we make strong assumptions about what kind of optimum we're trying to achieve on an isolated module while ignoring the other part of the system.
So those strong assumptions, as opposed to just looking at the data, making minimal assumptions and trying to formulate with dynamical systems or neural networks that learn by themselves and then they give you the phenomena is one issue that you have to make strong assumptions. And you have to know how much your results confirm what you started off with versus maybe things that you completely neglected. And also on the front of doing experiments, again the problem is introspection. How do you actually test this hypothesis in a rigorous way?
So my question is if you have any, again, general framework words of wisdom to give on this approach in general?
JOSH TENENBAUM: Sure. Well, it's not really words of wisdom, but it's just perspective. So that's a good question. And I think you know a lot of people have made similar arguments. I think the appeal to a lot of people of a black box function approach is that it feels more assumption free. And in some sense, it is.
But as soon as you start to interpret it as a scientific hypothesis, if you're trying to use it not just to approximate the input/output functions of the brain, but actually to make connections between the elements inside your model, like the neurons and the elements inside the brain, you're making assumptions. And you're making a lot of assumptions. And all science basically is based on assumptions.
Please show me any field of science that does not make assumptions and that has made progress. The key about science-- and this is different from religion or dogma or whatever-- is that assumptions-- this is part of a scientific paradigm-- assumptions are powerful tools that help you make progress and none of them are taken for granted. You make some assumptions and you see what you can do with them. They let you make certain models and interpret them in certain ways and suggest certain kinds of data and how to link up your models and data. All science works that way.
As you said, you articulated well some of the kinds of assumptions that we've made in our work. But again, these are not axioms. And some of them are things that have changed over the years as the data suggests that we should change them. You mentioned behaviorism as an approach as you said to what was seen as being a more rigorous objective alternative to introspection.
And that was a story that was told. But if you actually look at where behaviorism got, it didn't really get very far. It produced some very valuable data on learning curves. But as an account of perception, as an account of human thinking, as an account of really anything interesting in human intelligence, can you point to any insight that it gave us? I mean, I'd really like you to tell me one if you have one.
We don't know of any. And that was why cognitive-- that was where cognitive psychology came from. Psychologists realizing, this is fine. If you want like general laws of how animals-- how quickly animals learn when you give them various inputs and outputs that are not the normal things that their brains evolve to do, that's interesting. Brains can make-- they can do arbitrary function approximation. Effectively, that's what behaviorism was studying.
But it didn't actually provide a set of foundational building blocks that allowed us to understand, for example, how we perceive objects or scenes. Or how we understand other people's behavior in ways that were quantitatively empirically testable. Whereas, the cognitive approach does exactly that. And you can see this a little bit in the talks or in a lot of the papers we write. I'd be happy to talk more about specifics.
But you say, how do you test these ideas? You test them the way science has always been most successful. You do detailed quantitative experiments where we systematically control many factors and very others in terms of the stimuli we give people. Like in the context of theory of mind, we give people a range of different either perceptual or verbal stimuli, but especially various perceptual stimuli about agents moving around in some environment.
And we ask people to make judgments about what are the goals or the preferences or what are the beliefs? How much does this one think this was doing this or this was doing that? Or their social relation. Is this one trying to help this one or hinder this one? Are these friends or enemies? Are they good or bad?
We ask questions like that, and then our models make quantitative predictions and we get some pretty striking-- when you do properly controlled experiments, you can get some pretty striking quantitative fits to people's, for example, judgments about those different aspects of mental life, including both things that, in some sense, we think are objectively correct. And others which might be incorrect.
So I'm not going to go into this, but in some work from Rebecca Sax's lab using this toolkit from her student de Houlihan, they showed-- they've been applying these to people's inferences about others' emotions. How people are feeling or will feel in charged social situations. And we're often quite good at reading people's emotions. And sometimes, quite wrong.
And these models capture quantitatively both the ways people are right and the ways people are wrong. So they make the same kind of mistakes that people make in sometimes misreading how other people are going to feel in certain situations. And again, this is where psychology and cognitive studies have really always been most effective is when they make models that capture graded quantitative patterns of success as well as failure.
The intuitive physics models that we've built that talk about briefly in some of the talks and that's informed a lot of the work that Nancy's lab and I've worked, Nancy's lab has been studying the neural basis of human intuitive physics. How we look at a scene in a glance and get a sense of, will this thing fall over or what's going to happen if I push here? Or how heavy is this? Is this heavier or lighter than that?
These are also places where our models capture graded patterns of success with explaining most of the variance in human judgments in controlled, but still quite complex, situations. We give people like these Jenga blocks. 10 or 20 blocks. Is it going to fall? How likely is it to fall? Will it fall this way or that way? Is one of them heavier or lighter than the other? What happens if I push or bump it here?
And if you look at the data, we often show these really nice correlation scatter plots with a lot of things lining up along the y equals x line. But it's important to interpret that and to look at the different space of models. Because what you see when you look at that is that this really nice model fit, that's not the model fit between people's judgments and the ground truth physics. It's the model fit between people's judgments and our models, which make certain approximations and do probabilistic inference.
They have certain kinds of noise. They assume, or rather in a testable falsifiable defeasible way. But it's part of the models of assumptions that our brains don't perfectly accurately represent the physical state of the world. We can see three dimensional objects and we can approximately get their size, shape, and position, and approximately mass and friction. Not perfectly.
And as a result, when we run these probabilistic approximate simulation models, they're not perfect. They make only good guesses. And the correlations we show in behavior are correlations between the guesses in the model and the guesses people make. But if you compare people's guesses to the ground truth physics, they're also wrong in systematic ways. In particular, they often think that certain configurations of objects are unstable when they're actually stable.
So we tend to have these stability illusions in which we look at something and it seems like it should be falling over, but it doesn't. And in fact artists or hobbyists often exploit this. People make sculptures or statues or even sell whole buildings which catch your attention because they look like they should be falling over. But in fact, they're stable. And our models predict those stability illusions.
So again, this is about capturing graded patterns of success and failure. And I think that's very important. You see the same kinds of things in neural network models. So the standards of how you test the models and their assumptions are broadly similar because that's just the way science has made most progress. But ultimately, what you test is just a particular model. You don't test these framework-level paradigm assumptions that the mind does approximate probabilistic inference, or that the mind has certain symbolic structures.
You can't test those any more than you can test you know that the brain is implementing some kind of LSTM or transformer or convolutional network. Rather you test a particular model that instantiates a particular one of those sets of assumptions. And you see what kinds of quantitative accounts it can give a behavior or of what's going on in the brain. We also do neural decoding to test these models.
So in that sense, it's broadly similar. And I think these are ways in which, though there's a certain relationship and connection to the introspective approach that the behaviorists were reacting against, in a sense, these are ways that we advance very much beyond that. But also what the behaviorists ignored is that actually the introspection is-- they are the ones who invented psychophysics.
So the thing that the behaviorists react against was actually the people who invented quantitative psychophysics that we all use in our work. And it's a misreading and honestly, a kind of injustice to the introspectionists to say that all they did was introspect. They did some introspection and then they invented quantitative psychophysics in order to test those models.
So anyway, I think one should be wary of any stories or narratives that one field tells about like, well, why are we doing what we're doing in contrast to what some others approach? And that includes you should be skeptical of what I'm saying too. But again, I think if you want more detailed answers to those questions beyond these words of wisdom, you have to look at the papers and engage with what they're doing.
AUDIENCE: OK. Thank you.
JOSH TENENBAUM: Yeah.
AUDIENCE: Hi. My name is Susanne Haridi, and I'm a PhD student at the Max Planck Institute for Biological Cybernetics in Tubingen. And I was wondering, so a lot of the ways we are modeling things when we use neural networks seem to me like we are modeling human behavior or human learning in the form of learning by experience. But to me, it seems that humans are able to learn very well just from explanations as well.
And I was wondering how our current models of cognition fit that learning from explanation and our understanding of how humans learn things.
JOSH TENENBAUM: Yeah. I mean, that's a great question also. Thanks for all these questions. They're all really good ones. Well, I want to address that question, but I also want to-- but I want to actually ask you, do you really think-- so the word learning is one of these really interesting words that means different things. Like again, there's the thing that psychologists and cognitive scientists have meant by learning, which is like a thing that children or adults do when they're learning new ideas or learning new facts or learning new behaviors. And neuroscientists often mean that too.
But when you look at learning algorithms in machine learning, what learning means is basically just adapting the parameters of the model to data. And those could be the same thing, but they might not be the same thing. And I think-- a lot of people recognize these days that the learning algorithms that we use in machine learning, especially in neural networks, involve much more training data than any organism gets in their lifetime.
And so when we say we build a learning model, it's often not meant to be a model of the learning of an individual organism, but maybe something like evolution. Or maybe just a way of like fitting the parameters of the model and not at all in account of learning of the system. So some people use neural networks as models to model actual learning of an organism in its lifetime. But that's not-- I just want to make it clear.
I think everybody agrees that a lot of the ways we use neural networks as a model is not meant to be an account of biological learning. You guys agree with that? Sometimes it is, sometimes it isn't. Then let's talk about explanation. Do we all agree with that or is-- yeah. No? Yeah?
Well, for example, if you ask Jim DiCarlo-- I don't know if this came up, but he will often say-- he's done some of the leading work in using neural networks which trains on data to model the visual system. And Nancy's lab has used some of these same models for the human visual system. But Jim will usually say, we're not modeling the learning of the system. We're just using that as a convenient tool to build up some brain-like representations that are functionally optimized.
So you could say it's, in some sense, we're modeling at certain level learning because we think biological learning is trying to optimize these same functions. But the actual mechanisms of stochastic gradient descent on these data sets, that's not a model of learning at the mechanistic level. And I'm just quoting Jim on that. Although, other people-- I don't know if he talked about some of the work of his student Michael Lee, they've also built some models in which there's like one layer of neural network which tries to model some rapid human perceptual learning, which goes on top of a representation that might not be learned.
I'm not saying nobody uses neural networks to model learning. It's just that a lot of what we do when we use neural networks is not actually modeling learning. And that's often because humans or other animals just learn from a lot less data than those networks need to train their representation. And we should distinguish evolution from learning.
But now you ask a really important question, which is humans often learn not just from sense data, but from explanations. And by explanations, do you mean-- can you give a particular example of what you're talking about? Because there's many things you could be talking about, but yeah.
AUDIENCE: Oh, I mean, in the physical domain, you could imagine, for example, a weird scenario which you're being described to like, the red balls always fly upwards, which you have never experienced but this is what you're described. And then if you encounter this scenario, you can interact with it immediately without having to learn this. Or also in the sense of understanding other's minds, you could be explained that-- I don't know-- Julia really likes ice cream. And so her motives will be easier understandable to you when you see her working in a certain direction or things like that.
JOSH TENENBAUM: OK, good. So one kind of basic thing I think you're getting at is just learning from linguistic description. I can tell you a fact or something in language, right? And that's right. Yeah, so that is very much the kind of thing that we and others are interested in building models of. And that requires you have to have a model of how language works and to do that. So you have to have a model of how the sounds that you're making and the words that you're saying turn into some kind of a piece of knowledge, like parameters or structure in a model.
And so we and others have worked on that. Both of those are examples of things that we do. And those are good examples because those aren't especially hard. Once you have models of intuitive physics or intuitive theories of mind, then it's not that hard to translate the words you said into something that is in the same math basically as those models. And that often involves basically what are sometimes called relational representations. Things that look like little chunks of programs or graphical models, certain kinds of nodes and edges.
So those are various ways that people-- those are basically formal representations, mathematical representations of the meaning of those sentences that you said. And if we can translate those words into those chunks of math, then we can-- then they're in the same terms as the rest of the model. And you can talk about how your model builds out by adding those pieces in. Does that make sense?
AUDIENCE: I would even go a step further. Because sometimes explanations also change the whole way in which we think. For example, if someone explains the principle of falsification to you, then this might change the way you approach all future questions that you pose.
JOSH TENENBAUM: Yes.
AUDIENCE: This is quite a life changing thing that just happened through, I don't know, 15 sentences or something like.
JOSH TENENBAUM: Yes, right. So this is again one of the things that we and others in cognitive science are really interested in is abstraction. And the ways in which our minds models the knowledge that we have really varies quite broadly from very specific things. I can look at the room you're in, I've been in that room many times. I have a sense of what the chairs feel like, that they're made of wood, have that nice old feel. I have a sense of about how many of them there are.
I know there's more than 100 and less than 1,000. I can count each one of them. So I have very specific knowledge. you told me your names, I'm going to have to hear them a couple of times to remember them. I have very specific knowledge about the situation.
And then I have more general knowledge about rooms and lecture halls and very general things about human gatherings. Or other ideas, as you say like, I might learn about falsification or I might learn about the principles of logic or I might learn about Bayes' rule. Or I might learn about convolution and signal processing and how that relates to convolutional neural networks, gradient descent, which builds on notions of calculus and so on.
So, yeah. And I can explain-- in this sense, I can give a extended linguistic description of an idea in sometimes just a few sentences. Which can, as you say, change how you think for the rest of your life, which is really amazing. So I can't say that I can tell you everything about how that works. But this is one of the deep reasons why we're interested in this level of modeling.
In fact, this is actually-- we're actually working on some version of that problem right now. So I can't really report concrete research results, but the idea-- I can tell you about the idea and hopefully, you can see why it's maybe interesting. Is if you think about-- so this idea of probabilistic programming that I was describing. Fundamentally, what that means is that our knowledge takes the form of programs or code.
And if I give you something like-- if I tell you a specific fact, that might be like adding in one little line of code to your code base-- your mental code base. But if I tell you something more general, like a principle of falsification, well, that might be adding some meta program. One of the things you see in programming is you have programs that manipulate other programs.
So I might be giving you a new kind of program that's a higher order program that can manipulate other programs or it can be used-- in a generative perspective, these meta programs can generate new programs. So if you want to think about, where do new hypotheses come from, my lower level hypotheses are programs of a certain then. I could have probabilistic generative programs which generate spaces and priors on those programs.
And if I give you something at that level, then it can provide many, many other effectively new programs that can trigger in other situations. So we're working right now on how to basically build models of translating natural language, like what we say when we talk to each other, into those kinds of higher order programs. And that's a hard thing to do, but it's very interesting.
And I should say one of the ways we're doing that is we're using what I think are quite exciting forms of neural networks. They are these neural networks, which have been produced-- many people are probably familiar with GPT-3 these neural language models and related models called codecs, which basically produce model natural language to code. So those are very useful ways to model at a certain level of abstraction, this kind of translation.
Again, I wouldn't interpret that too literally about what goes on in the brain because those models require vast amounts of training that your brain doesn't have, but they may capture it at a certain level of abstraction. So anyway, I think that's very interesting. The one last thing I guess I'll say about explanation is that to many people, what explanation also refers to or requires is or taps into is notions of causality and counterfactuals.
When we often when we explain, if I say, why did this happen? I say, well, because of this. And what I'm getting at is I'm trying to give you a hypothesis of what caused the thing you want to explain. And causality is often analyzed in philosophy and in cognitive science via counterfactuals like when I say, x happened because of y, one of the things I mean is that well both x hat and y hat to have happened. But also I have to think and it has to be true if this explanation is correct, that if y hat not happened in the same situation that did happen, then x would not have happened.
So that's a counterfactual dependence. And one of the things that I've been really excited about that's come out of work from our group, a lot of this work was done by Toby Gurstenberg, who was a postdoc with us and has now been an assistant professor at Stanford for a few years now, are really nice models of counterfactual and causal both reasoning, as well as perceptions. So he gives people a broad range of perceptual situations where a ball hits another ball. And you ask, well, did this ball cause that ball to go in the hole?
Or there would be three balls. Did this balls prevent this ball from causing that ball to go in the hole? Or just like, how responsible was this ball or this ball for making this ball go in the hole? Or you can extend this to a social situation, as he's now done with one of his students Sara Wu, where you say, well, did this agent cause this agent to do that thing or help this one do that or get in the way of this one? And so people make these judgments about causal responsibility in either a physical or a social setting.
And he builds-- and we build together these probabilistic counterfactual reasoning models, basically. Where you have a probabilistic program that's able to make-- basically imagine possible worlds, which are not exactly what happened but where you imagine changing things. If you know Judea pearls intervention framework for causality, it builds on that but in richer representations of probabilistic programs.
So we can build models of how people make counterfactual inferences, which are also guesses. So again, it's probabilistic because we don't know exactly what would have happened in all counterfactual worlds. But we can make a guess of how likely this would have happened if the world was different. And then we can test those as quantitative models of causal responsibility.
And for those of you who like quantitative behavioral experiments, I recommend you look at Toby Gerson's report because nobody is better than he is at generating really elegant experimental designs which give beautiful mathematical fits. We used to joke that if the correlation wasn't 0.98 or above, then it wasn't Toby. So anyway, if you come from more of a physics background and you like really nice quantitative model fits, Toby's work showing how you can capture these ideas behind a different part of explanation of causal and counterfactual reasoning in both perceptual and other more cognitive contexts is very, very beautiful work.
AUDIENCE: Hi. So yeah, my name is Nathan. I'm a master's student from Belgium. I also work with Robert Young at BCS. As you said, it seems that human cognition is better explained as the ability to write programs and to build domain specific languages, to build models. And it seems that this ability is very different from what animals could do.
And so my first question is, how can we explain that through an evolutionary point of view? So is there at some point in evolution there was a critical change and then there was some kind of bootstrapping that allowed humans to develop these abilities? And also my other point is that if we want to bridge the gap between the brain and neural networks and the mind with, for example, prototype programs by having, for example, neural networks that can construct new languages and write programs, you think it would help to have a community of neural networks and evolve such a community?
Which would be a very different paradigm than how we are currently training a neural network, which are just considering one neural network on one data set. So yeah, do you have any ideas about it?
JOSH TENENBAUM: Yeah. Again, several great questions there. So let me try to address them quickly and in order. So the first question, you asked is one that many people have asked and it's not just about the work I'm doing. It's whether you build the computational models like we do or any computational models at all.
The question of how we got to human intelligence from evolution, the same path and mechanisms that led to many other forms of intelligence. But without putting humans in some higher level status, it's just an objective fact that humans have had an impact on the world and transformed the world on Earth and their own lives in ways that no other animals have done.
There's definitely something distinctive about human intelligence. And lots of people have tried to say, well, how did that work evolutionarily? You can ask that question without computation. Maybe computation will help.
I'm not an expert on this but one thing I do reflect on, which I think is related to the point you were saying, is that if you look across the different-- many people have different stories or accounts. And lots of people have tried to say, this is the one thing-- the key one thing that-- because it didn't take that long it appears for human intelligence to emerge and our brains seem so similar to the brains of other animals in a lot of ways, it feels like there should be just like one or a small number of magic things or just special things that happened.
And people have proposed many candidates for that. I can think of like at least six or seven, and I don't know if I'll remember them all here. But some people have said the distinctive thing is natural language or some ability to do what we're doing right now. Other people have said, well, no. It's really about symbolic some kind of higher level symbolic thinking and that enables natural language. And some people say, no. Language enables symbolic thinking.
Other people have said it's really about social cognition or theory of mind. It's our ability or our interest in modeling other people's minds. And some people say, no, that depends on language. They say, no. That's what enables language. And it's all these circular loops. Some people have said it's really about tool use. So the distinctive ability to make tools, which like what we're doing right now, we're talking to each other over Zoom.
That's based on so many tools. Software, hardware, electrical engineering, cables. I mean, the tool use that or the ability to make new tools-- technologies that got us to the point so that we can do what we're doing right now, hold the microphone in your, hand make all these things. That's a story that goes back over two million years to the first human tools that, at least are known to science. These things called hand axes, which are basically rocks that have been shaped into general tools that you can find all over the Earth that were made by early hominids.
So for about two million years or more humans have been making tools and doing this in a way that no other animals do. And passing this on culturally because nobody is able to figure out how to make a computer, a microphone and internet or even a hand ax for themselves. It's very hard. But somehow knowledge builds over generations and gets passed on and grows over time. So some people say that's the magic thing.
And some people point more generally to the culture that lets that happen. And I guess my view-- and other people have argued this too-- is there isn't any one of these things that was the magic thing, but all of these are distinctively human. And if you really look at them, especially from a computational lens-- so if you build if you try to build computational models of any of the things that I mentioned, they all wind up using some of the same machinery.
Which is actually, not surprisingly for me, I think the machinery that we're talking about, probabilistic inference on programs and abstractions. So it's possible that this isn't a story of evolution. But what I'm saying is there's a whole bunch of things that people have cited as the basis of distinctive human intelligence. Social understanding, language symbolic, behavior, tool use, culture, long-range planning that's another one. What we see interestingly is that they all reinforce each other.
They use they build on the same computational tools and each of them makes the other better. So this is not an evolutionary story it's a correlational story. Some nice birds have come to join me on the balcony here. But it's interesting that, again, maybe there's something about how a co-evolutionary story about how all these abilities make each other better that could have explained it could explain how so much change could happen so quickly. Because if all of these things reinforce and bootstrap and scale each other, that could explain how actually quite a lot has changed, but so quickly.
So that's my speculations on evolution. And I think it is related to some of the ideas like the ideas you mentioned about community of models. And I know that's what Robert has been-- one of the things Robert's lab has been working on and I very much it's very interesting. I think all of modern neural networks as instantiated in all the code that's been built in TensorFlow and PyTorch and Jax and other frameworks like that, which are really what's as responsible as anything for, why do we have neural networks in AI the way we do? And why are neuroscientists and computational and cognitive neuroscientists able to use these tools and people in many other fields?
It's because of the cultural processes of building these modeling toolkits and ecosystems. So the idea that there's cultural evolution of spaces of models and tools has been hugely important. It's been as important as anything in the evolution of neural networks and AI. People tell these stories, why do neural networks work now when they didn't work in the '80s or the '60s? And they say, because we have more data and more compute.
But I think we all know that, just as much, if not more is the fact that we have these programming languages and the ecosystems of humans working with them all together and jointly developing tools together culturally. And spaces of models and starting to understand what works, what doesn't work. And that very rapidly accumulates and grows these tools. So there's a similar kind of story-- a cultural evolution story that also links up to the evolution of hardware.
But an evolutionary story that explains the growth of neural networks-- and I do think that we need more of that and we need to push that in more ways in neuroscience if we want to just understand better what's going on. My view is that probably that alone is not going to explain how the-- just all we need to link the mind to the brain, but it's one approach. Another approach that we've been working on is more of a top down approach. And when I say we, I mostly mean my colleague, Vikash Mansinghka. So Vikash is an MIT PI who directs the probabilistic computing group.
And there have been some of the leaders in developing probabilistic programming languages as technologies, especially like a language, for example, called Jen, which is language developed by Marco Cusimano Towner in Vikash's group a couple of years ago. And it's really, that I know, the first probabilistic programming language designed to do the kinds of things that we're doing in intuitive physics or scene perception or theory of mind that can really scale in engineering ways. That can work in large scale systems and do real time approximate inference that you could put in a robot.
But the thing Vikash has been interested in really recently is actually compilers that compile probabilistic inference-- Monte Carlo or sampling based probabilistic inference in a language like Jen to neural circuits. And he's got some very interesting work that I only hear about from him so I don't know the details.
But they've been showing ways to compile inference into networks of stochastic spiking neurons, like actually quite realistic biological models. Honestly, more biologically realistic than most of the artificial neural networks because they try to capture realistic spike timing and they look at canonical cortical micro circuit data.
And what they are seeing, at least Vikash tells me, is really intriguing-- both really nice performance and intriguing fits to ask key general motifs of neural computation. So that's also really interesting. And that also could be the basis for how we might understand how the mind works in the brain is if we have these probabilistic programming models at the cognitive level that can then be compiled down into circuit level probabilistic models, much as on the engineering side.
We have compilers that take source code that humans can understand and then they compile it down into machine code that can run on their physical machine. That's another route to trying to establish the mapping on the natural science side. And I think that complementing the more kind of emergent evolutionary approach that, say, Robert's been working on, we want to really see where those top down and bottom up perspectives meet up. I really think we will make progress on that over the next 5 to 10 years with that combination of perspectives.
AUDIENCE: I'm Natalia. I'm from the Yale University. I'm doing my PG in visual neuroscience. So I just had one quick question. So I was wondering if you identify if there is like an upper limit of how functionally similar these probabilistic models that we viewed as scientists can be from what the brain is actually doing. And if so, if there is such an upper limit, if you have any idea of how it would look like? And as an example of something that I think or identified that might be very dissimilar between what we have so far, is that most of these models that we build seem to be very task specific?
So they are very good at solving some specific tasks, but when we think about human intelligence, one of the key aspects of it seems to be the flexibility in which we change or switch between physical and social models, how we integrate both to making, for instance. So, I don't know, if you could say something [INAUDIBLE].
JOSH TENENBAUM: Again, you asked three good questions there. I'll try to answer them as quickly as I can. So you asked if there are any upper limits. And again, I think any scientific paradigm there are limits of what you can do today. And then a lot of what people are trying to do is push those limits. So we don't know what are the real limits, we just know what our limits of the tools we have right now.
And our progress is driven by people asking exactly that question. Thinking, OK, well, what are the limits? Where can we go with it? And that's true in any paradigm, I think. So I don't want to say anything about what are the true limits, because who knows? And I could say a lot more about when you say. Well, you asked about how these models correspond to what the brain is really doing. Where I thought you were going to go with that was more what I was trying to address in the last question, which is how do they relate to the level of neural circuits.
And again, that's where I would say, well, they don't unless you do the kinds of things that, say, Robert or like Mehrdad Jazayeri at MIT is also trying to do where you basically train neural networks to approximate probabilistic inference. Or the stuff that Vikash is doing in which you take a more top down approach and you compile the probabilistic models into neural circuits. So that's a way to get beyond that limit that the pure software level doesn't address.
But then you ask another really interesting question, which you said, well, you look at these models and they look like they're doing one thing. One set of models are doing intuitive physics, another set of models are doing social understanding. Is there some unified way to put those in the same terms or to understand how they interact? And I'd say, that's a great question.
And in a sense, it's understandable because you might say, well, when you look at the cortical circuitry neurons look like neurons. So is there some like cognitive level unification? And I would say, yeah. Well, first of all, one of the reasons why we like probabilistic programs is because these are languages for model building where those models are all expressed in the same language. The same way that if I have a neural network-- I can use neural networks and train them to do vision, and I can use them and I can train them to do language.
And the models wind up being quite different, but it's the same-- I mean, that the weights are all very different. But maybe the principles of stochastic descent or maybe even some of the same architectures are similar-- shared. But the actual model is different. The weights that do one are not the weights that do the other. And similarly, for us, the modeling languages and primitives might be the same, but the models that do intuitive physics and the ones that do intuitive psychology are different.
Now you might say, yeah, well, OK. But it looks different because you didn't just start off with a single architecture and train it one way or another. You built it to do this and you built it to do that. But that's because what we're doing when we're building these models is we're-- at least in those cases, is we're trying to capture what evolution did. Evolution built things, it didn't learn things.
Evolution doesn't work by gradient descent. It doesn't work by modifying parameters and fixed architectures. Your brain was built by evolution. And evolution built machinery into your brain. And one of the really striking discoveries to come out of Nancy's work-- Nancy Kanwisher's work as well as others who've worked with her like Rebecca Saxe and Federico-- is that you can look in the human brain using fMRI and other technologies and see in it specific subsystems which in fact do these things.
There's a visual system. There's a language system. They're quite distinct. Both of those can be parceled out into other subsystems. There's a part of your brain, a theory of mind network, which does basically thinking about other people's thoughts. And there's an intuitive physics network that Nancy and I and folks we've worked with have mapped out and discovered. These intuitive physics network, by the way, overlap-- when I call it an intuitive physics network it doesn't mean that's the only thing it does.
The intuitive physics network also seems to be involved in tool use and action planning, and even action understanding. But it's but it's selectively and distinctively engaged, and you can localize it from some of our intuitive physics tasks. Now you asked about how these integrate. And I think really, you're exactly right. Let's just take the case of intuitive physics our understanding of physical objects and dynamics and other agents.
Well, it is true that in the brain and in our models, these seem to be distinct systems. They also interface and build on each other. And in particular, our understanding of other people's actions, their goals and plans, it has to build on our intuitive physics. When I see somebody reach for something or doing something or moving around through space to get somewhere-- like if I were to see any of you get up from your seat and go out to the door, why did you do what you do?
You move don't move in a straight line. You move in a rather indirect path because that you can't just like you have object permanence-- you can't walk through the chairs. If you want, you can't also fly. So if you're going to go up from the bottom of the lecture hall to the top, you have to go up the stairs. You can't just fly through the air. If you were a bird, you could fly through the air. If we were in outer space and zero-g, it would be different.
So our understanding of the physical affordances of objects in an environment is fundamentally constrains and is the basis for our action plans. And we humans understand that not only for ourselves, but for others. So we've shown in models that we've built of intuitive psychology or agent understanding that they actually sit on top of intuitive physics models. And that's crucial to their success.
And we show that with human adult judgments and even really excitingly actually, in young children and even babies judgments. So a project that actually came out of the summer school indirectly and the CBMM, Sherri Lu-- who was a graduate student at Harvard with spelt and then did a post-doc with Rebecca Saxe and also worked with me and with Tomer Ullman. All CBMM PIs-- is now just finishing up her postdoc with Rebecca and is going to be a junior faculty member at Johns Hopkins.
So Sherri did some really striking studies as part of her PhD where, with Tomer Ullman, built computational models of basically how agents-- how babies-- 10-month-old babies-- infants, nonverbally can see other agents moving around. Little like balls like rolling, hopping, rolling up things. So they don't even look like humans, they have little smiley faces on them.
But they can like jump over things, roll up ramps, jump across gaps. And it showed that 10-month-olds made systematic inferences about the desires and preferences of these agents. How much any one agent wanted a certain goal based on the costs it was willing to undertake in its action. And those costs were physical. They were measured literally as work done, like integral of force applied over a path. So basic physics.
And it showed that at least, again, at a just at a behavioral level, you could capture graded inferences that a 10-month-old infant might make that says, well, this one likes this one more than that one. Or this one likes this one a lot more than that one, a little bit more than that one. Based on the path that the agent was willing to-- or the path that the agent followed and how much cost they were willing to pay.
So if an agent jumps over a medium sized wall to get to something but not over a really high wall, they don't do that same jump, then that suggests that, well, they kind of like it a medium amount. Whereas, if the wall-- if they jump over a low wall but not a medium sized wall, then they don't like it as much. And what they showed in this paper that was published a few years ago was-- this was a 2017 science paper, Lu and Ullman are the first two authors.
They showed that these models generalized across a number of different physical scenarios. Jumping over a wall, rolling up an inclined plane of different slopes, jumping over a gap of different gap-- of different widths. And that the same model could explain graded inferences based on how much physical work an agent was willing to do. They've since generalized this-- and this was also part of a CBMM project to not just how much work are you willing to do, but how much danger or risk are you willing to undertake?
We have a sense-- and in this they showed with 13-month-old babies, 10-month-olds didn't quite have the sense yet, which might have something to do with the fact that they're not doing the risky stuff of walking around the world. But 13-month-olds even have a sense of how risky emotion is. Are you getting near to the edge? If you're trying to jump a deep gap, that's riskier than jumping a small gap, even if it's the same amount of work.
And the amount of risk of danger that you're willing to take in your action also is diagnostic of how much you want something. So it's a really beautiful, elegant studies with very young babies that show that their understanding of other agents actions depend on these factors, which are really physical one's. Force and risk of damage, basically.
We don't know how that works in the brain, but one of the things that we'd like to do that Sherri is working on with Rebecca-- other folks actually at Hopkins like Leila Issac who was a CBMM post-doc grad student postdoc are also working on this-- is trying to understand how these intuitive psychology and intuitive physic systems interact in the brain guided by the models that we've been building that show how intuitive psychology builds on intuitive physics. So we deeply believe in the importance of what you're talking about.
That these systems need to be integrated. That both, they need to have an integrated computational formalism, which we've been trying to develop. But also these functionally distinct brain systems. They're not isolated. They are interacting. And understanding those interactions will be key to really understanding where common sense comes from.
have an interactive transcript feature enabled, which appears below the video when playing. Viewers can search for keywords in the video or click on any word in the transcript to jump to that point in the video. When searching, a dark bar with white vertical lines appears below the video frame. Each white line is an occurrence of the searched term and can be clicked on to jump to that spot in the video.
