Computational Models of Cognition: Part 1 (1:07:34)

Computational Models of Cognition: Part 1 (1:07:34)

Date Posted:  August 16, 2018
Date Recorded:  August 16, 2018
CBMM Speaker(s):  Joshua Tenenbaum
  • All Captioned Videos
  • Brains, Minds and Machines Summer Course 2018
Associated CBMM Pages: 

JOSH TENENBAUM: The general title of these two lectures is "Computational Models of Cognition." But they'll kind of be divided, and hopefully into two natural segments. The title of the first one is "Engineering and Reverse-Engineering human intelligence. The focus of the second one will be more about learning in particular. OK.

And I'm going to spend some time-- a sort of extended introduction-- talking about the big picture questions that we're all interested in, I think, certainly that I'm going to be representing here in the Center for Brains, Minds, and Machines. You know, I'm definitely coming from the minds part with a strong machine component, but really extending down to the brain. So there's going to be some brains, minds, and machines in this talk. But I'm really trying to represent the cognitive science perspective. And I think together with Jeremy's talk, afterwards you'll get a nice, good dose of cognition today.

But part of what I want to do is really introduce the philosophy or the perspective that we come from, as well as a lot of specific material about how we've been starting to understand intelligence in engineering and scientific terms. OK.

So let's start off with some big questions. Here's one. You might think of this introduction as like several different introductions, which are all trying to triangulate the perspective. So I think it's broadly a premise of this entire summer school that intelligence is some form of computation. And that means the mind and the brain is some kind of computer. But what kind is it? What kind of machine is it? How do we think about it in engineering terms? And how do we think about intelligence? What kind of computation is intelligence?

And one way to start this off is to think about three possible answers to this question-- three possible answers, three actual answers that have been proposed. Each of them has a history spanning decades. In a sense, all these fields of cognitive science, neuroscience, artificial intelligence-- they all kind of grew up together starting in the 1950s. And they've had-- it's not like they've always followed the same philosophy throughout. But they have had waves of exuberance for different kinds of views. And if you look at the rise and fall and the comings and goings of these different ideas, you see some important patterns.

In particular, you see, I think, three big ideas that have proved their worth over the years, OK, although at different times, people have been skeptical of them. But I think at this point when we look back on the decades-long history of the field, and look forward, we see, OK, these are three good ideas that we collectively have had.

And I want to bring them out at the beginning. And one theme to start off with at the beginning is to say, I don't think any one of these is the one right view. I think they all have important insights. And a lot of what I want to be thinking about, and for you to be thinking about, is how to weave them together. What might we do going forward that combines their best insights and tools? All right.

So one view is one that I think you've already heard a lot about. And it's probably the one that has been getting the most attention these days, certainly on the AI side in the broader public eye, but also especially in neuroscience because it's the one that is most naturally both inspired by and useful for neuroscience. OK.

And that's the idea of the brain as some kind of a pattern recognition engine-- OK-- for seeing patterns in data, and then doing intelligent things with them. And a particular-- the methodology of, say, what we call neural networks or deep neural networks, especially if we're thinking about vision, which, of course, is a lot of what people in the summer school are working on. We're thinking about, say, deep convolutional neural networks. But of course, I mean this to include recurrent networks and other ideas in the spirit.

But this idea, which, if we want to give a one-slide summary of, again, things you've seen probably many times, but the idea that starting from the basic unit of a neuron in cortex like the Hubel-Wiesel V1 simple cell, for example, that can be formalized into what was originally called a perceptron; and then going beyond just a single neuron-like unit to multi-layer hierarchies, such as the neocognitron or [INAUDIBLE]. These were examples from the 1980s and the 1990s.

And a history that unfolded over the last couple of decades of making networks that were deeper, with layers that were bigger, and ultimately-- or I don't know about ultimately, but at this point now, almost 10 years ago, culminating in a set of deep-- what came to be called deep neural networks, mostly because they were a lot deeper than the ones before, and that such as the AlexNet here, that when applied to large data sets-- of course, that's the definition that keeps changing, but large data sets of, for example, the famous MNIST data set of handwritten characters or handwritten digits 0 through 9 or the ImageNet data set, when applied to these large data sets, really took an important step forward relative to the previous state-of-the-art circa 2010 in computer vision. And that started getting people's attention in a number of other areas of AI. But really, where they've had their most success is in things like speech recognition, basically problems of pattern recognition.

And then, of course, again, as you heard from Jim DiCarlo a few days ago, you can close the loop. You can go back to the brain, and in particular the part of the brain that originally inspired these architectures-- the visual system, and in particular the ventral stream that Hubel and Wiesel pioneered the study of, and many other people have contributed to, of course, and developed; and see, well, actually, these same architectures really seem to provide right now our best account of what individual neurons across multiple levels of representation in that first-- especially that first feedforward pass through some kind of zoom and some aspect of object recognition.

And again, this story has developed. You heard about this from Gabriel with recurrence feedback. And there's a lot of exciting things. It's not like this is over. This story is maybe just the beginning, but it's proved its worth. OK.

Now, here's another idea that's also proved its worth. This is an idea which really was at a height-- one of several heights at the time I was in grad school. And so, it really was influential on me. And then, in the early part of my career, I did a lot of work that was really very clearly part of this.

This is the idea that you could say the brain is something like a prediction engine. Or you might think of as an engine for probabilistic inference. It's also closely related especially to the idea of causal inference. You can think of causal inferences roughly as like predictions, which can be conditioned on the actions that you take or interventions.

And the formal methods here are a different kind of network-- what sometimes would have been called Bayesian networks or graphical models. And especially from a cognitive point of view, we've been interested in what are called directed graphical models. That's what Bayesian networks mean.

A classic example of that is this one here-- this disease symptom network. It's where there's nodes and arrows. And in some sense, you might say it looks just like a neural network-- right-- circles and arrows.

AUDIENCE: Yes.

JOSH TENENBAUM: The main difference between a Bayesian network or a directed graphical model and the most canonical kind of neural network, such as a deep convolutional network for object recognition, standard one, or for modeling feed-forward processing in the ventral stream, the most-- the basic difference is whether the arrows point up or down. Notice the arrows point down here. I guess in the picture I had before, the arrows point-- a lot of the point left to right versus right to left or something.

But what I mean by the direction is do the arrows point from something like the sensory inputs to the decision outputs. That's the typical direction in a neural network. And that's what I mean by pointing up, like from the bottom up. Or do they point down, in the sense of, do they represent something like are top-down expectations about how the world works?

Do they go from effects to causes, as in a neural network, or symptoms to diseases. That would be a traditional neural network for medical diagnosis. Or do they go from causes to effects, from diseases to symptoms, in which case, they represent not a direction of inference but a direction of causation, but something like a mechanism.

So when we draw an arrow from a disease no to a symptom node we're saying it's an assertion about our knowledge of the causal structure of the world-- a piece of it that says this disease causes this symptom. Or it may be that the semantics of that might mean it raises the probability. Or in an instance where this is present, this one is more likely to be present, for an example.

That's useful. It's useful to have that different kind of knowledge because it allows you to reason about what you can do to plan actions. It allows a doctor to know, or anybody really to know that a medicine, for example, that treats the disease is probably going to be more useful than one that just treats the underlying symptom because if you treat the symptom, the disease and the other problems that it causes don't go away.

But if you treat the disease, hopefully that the symptom and all the other symptoms will go away, too. And it will be a longer-lasting treatment. That's a kind of causal reasoning. And it requires this kind of model to understand and plan with. All right.

Now, you can also use these systems to do inference to go in the other direction or to go in every which way direction. And that's why they're called Bayesian networks. It's the idea that using basically computational generalizations of Bayes' rule, which is about turning conditional probabilities that go one way into probabilities to go the other way. You can take knowledge about how diseases give rise to symptoms or the probability of symptoms conditioned on diseases, and turn it around to get the probability of diseases conditioned on observations about symptoms.

And that sort of represents the typical pattern, or at least the kind of archetypal pattern of, say, medical inference, or a lot of other kinds of causal inference. You see the effects. You work backwards to an underlying causal explanation. And then, you can predict what other things would be likely to be observed or to happen. Or what would the effects be if you now acted on the underlying causes? What would you expect to see differently and hope to see differently in the future?

That's the canonical pattern of Bayesian inference on causal models. And as I hope you can appreciate just intuitively, it's very useful to be able to think that way. It's very complementary to the neural network perspective. And there was a lot of work in the 1990s, especially. These ideas really started forming in the '80s. But there was a lot of work in the '90s and the 2000s developing this as a tool kit for cognitive science and AI as well as neuroscience. So you can read books with titles like The Bayesian Brain or The Predictive Brain, which developed correspondences between this toolkit and some things we might understand about canonicalization circuits and cortex. So that's idea number two.

Here's idea number three. It's the idea of intelligence as some kind of symbol processing or the brain as a symbol manipulation engine. And this is the third idea I'm talking about. But it's pretty clearly the oldest. It's the oldest in the sense that it's often most associated with the early days of AI. When people talk about good old fashioned AI or classical AI, they're usually talking about this toolkit. But it's a lot older than that.

The idea of intelligence as some kind of symbol manipulation may be something kind of like logical deduction. Of course, that was made famous by, for example, Boole in his development of Boolean logic. The book he wrote was not called Mathematics of Logic. It was called The Laws of Thought.

And if you read his book from the 19th century, you'll see it starts off as all about cognition. And you could trace these ideas back in various forms to Aristotle. So they're thousands of years old. You can also trace probabilistic inference back to Aristotle. But he didn't have formalism for it. He just had intuitions. And he used those intuitions a lot in his thinking.

But he introduced, in his actual writing, formal characterizations of symbolic manipulation, like the famous syllogism-- the idea that if you say Plato is a man. All men are mortal. Therefore, Plato is mortal. That applies to Plato. But it applies to anybody else who's a man. And that formula can apply to anything. You could say, alpha is a beta. Beta-- all betas are gamma. Therefore, alpha is a gamma. That's more or less what he wrote, with Greek letters and everything.

And the point is that's a formula. That's a symbolic expression. And you could plug-in anything you like into that, including things you don't even know what they mean, like Lewis Carroll had many examples, like if Plato is a snark and all snarks are boogum, then Plato is a boogum. I don't know what a snark or boogum is. But I know that's going to be true if those other statements are true.

So again, this is an incredibly powerful tool for intelligence. It allows you to take concepts and reason with them. And it often, in the stories that people tell about our fields, these days, it tends to sometimes be-- I don't know-- the villain or the fool. This is like, well, this is the first thing we thought. And we used to think this. And maybe for thousands of years, we thought this. And then, we learned our lesson. And we've now seen the light because this didn't work. And in AI, these systems were not robust. They were brittle. You couldn't learn with them. They couldn't handle messy, noisy real-world data-- all that sort of stuff.

That's a story, which to me as a cognitive scientist, I put in the category of fake news-- real fake news-- not fake, fake news. I think that, well, it's true that early generations of AI systems didn't work nearly as well in real-world applications as today's systems. The reason for that is not because they were using symbols. In fact it's maybe because they weren't also using some other things.

But symbols have always been a part of the AI toolkit. And you can see that most clearly if you look in today's neural network systems. Again, as those of you who are doing deep learning in your own research or in your projects, you're probably using systems like TensorFlow or Torch or PyTorch, or any number of other systems. Or in the probabilistic programming tutorial that you saw I think just yesterday, you learned another problem of symbolic language with WebPPL or other things that that's based on like Church.

So whether you're doing probabilistic inference and probabilistic programs or neural networks or deep learning, you're using symbolic languages. And we wouldn't have today's deep learning, and it wouldn't be nearly as useful or as important or as interesting if we didn't have these symbolic languages. Imagine if you had to take a state-of-the-art neural network for computer vision and wire it all by hand, like write code that generated every weight instead of using the abstractions that you get from a language like TensorFlow or PyTorch today. Nobody would do it, or far fewer people would use these tools.

And of course, those languages build on earlier languages like Python, for example. And of course, Python is just one programming language that is at one point in the evolution with languages that go back to languages like C or LISP, or many other-- or FORTRAN-- many other useful symbolic abstractions.

And of course, those build on ideas for mathematics. We wouldn't have really anything we call engineering or science. We wouldn't have physics. We wouldn't have computer science. We wouldn't have math without formal languages, which are also kinds of symbolic systems.

And I think it's pretty clear we wouldn't have any of those without the original human symbolic system, which is what we call natural language-- English or whatever your native language is. The world is full of what we call natural languages, which are the product of biological and cultural evolution.

They are pretty much unique in the known universe. I'm sure there are probably other things like that out there. But as something that has just naturally evolved, human natural language is the first-- is the uniquely first expression of the general idea of symbol processing. And I think it's pretty clear. You'll hear a lot more about this perspective later in the summer school, that we wouldn't have really anything that we call intelligence. We wouldn't be able to call anything intelligence because we wouldn't be able to call anything at all if we didn't have language.

So I hope you can agree that this idea in its broadest terms is important regardless of what anybody wants to blame it for, for being-- in some sense, it's sort of the victim of its own success. It's such a good idea that, of course, it was the first thing that people tried. And they were bound to fail because we didn't really have computers in the 1950s or '60s or '70s by today's standards. So there's a lot of reasons why our first attempts were not going to work. But it was such a good idea that people thought that was all they needed. And what we've learned is you need this. And you need those other ideas, and surely other things, too.

But again, I think if we've learned anything, or one of the most important things we as a field collectively have learned is that you need at least all three of these ideas-- pattern recognition, probabilistic inference, and specialty causal inference, and symbol manipulation-- symbolic languages for expressing abstract knowledge and patterns of reasoning.

And you put these things together, and what you have on the engineering side is really a quite exciting moment right now, again, which we all know. The whole world knows about this now, and many of us are even participating in as developers, not just users. We know AI is back in a big way.

We have all these AI technologies, by which I mean we have these systems that do things in the real world, or at least a lot of it is the internet. But these days increasingly, that is people's real world. But they do things that people used to think only humans could do. And now we have machines do them. And machines can do them at sometimes human level, or sometimes even better than human level.

But I also like to say, and I think this is really-- this shows the moment, but real opportunity for the science and engineering of intelligence, and for the thing we're trying to do in this community that you're part of here now in the interface between them. It comes from recognizing that while we've had these successes and we have all these technologies and tools that are increasingly powerful, we don't have anything like real AI-- no real AI-- nothing that the original founders of the field would have recognized as meeting their goals. Although they would have been very impressed by these technologies.

But what I mean is we don't have anything-- any flexible, general purpose intelligence that can do each and every one of these things for itself just like you do. You may not be the world's best at Jeopardy, or Go. You may not be a professional race car driver. But any human being can learn any of these things, as well as innumerable other domains of knowledge and performance. And you don't have to be specially engineered by a crack team of AI engineers to do it.

So what we'd like to understand-- I mean, the big thing we'd like to understand is what's the gap. What's missing? And how might we actually build our engineering toward something more like real AI, but by drawing on, and really then developing the science of how that arises in the human mind.

Now, the message of this slide might be a little bit opposed to some of the words I used before. That's just because we're still trying to figure out how to use the words. But I would say the biggest thing that's missing is that in today's AI technologies, what's really driving the technologies are successes in pattern recognition.

Now, I don't just mean neural networks because what I've tried to say is what makes today's neural networks especially good for pattern recognition-- they are really pattern recognition engines. But what's making them work is a bunch of things. Of course, there's hardware, like GPUs and just massive cloud computing resources. But also today's neural networks compared to earlier generations of neural networks-- this is more on the idea side.

We have now sophisticated probabilistic approaches to really understanding what they're doing and structuring the training. And we have also particularly sophisticated, symbolic languages like TensorFlow, for example, that allow rapid innovation and cross-fertilization of ideas. But they've mostly been deployed in today's technologies all in the service of pattern recognition, and just making that work really well.

But intelligence is about so many more things than pattern recognition. In particular, the cognitive science perspective, I want to emphasize, is all these activities that I like to call modeling the world. So think about all the ways that our brains build and use models of the world.

And these are things for which you really need to add to that neural network toolkit-- those other ideas in a much richer way, and not just use probability and symbols to make neural networks better. But for example, use neural networks to make probabilities and symbols work better. That's going to be more the theme of the way I and my work use neural networks.

But think about these activities of modeling, like, for example, all the ways in which we don't just see patterns in data, but we actually understand what we see and explain, or our ability to imagine things that we've never seen, or maybe things that nobody's ever seen. And then, to make plans about actions that could make those things real, and to solve the problems that come up along the way using our models.

I've already talked about causal models, and as an instance of this. And then, think about learning as building new models of the world. That could be making small or big tweaks to our existing models-- debugging or fixing them. It could mean combining or recombining pieces of several existing models to make a new model. Or it could mean synthesizing a whole new model, in some sense kind of from scratch, whatever that means. But that's the kind of thing maybe we might think we do in science.

So these are all activities of human intelligence. And they're things that we're really trying to understand. Again, in the field of cognitive science, you can try to define it in different ways. But one way that a lot of people would see it is as really this perspective. And I don't mean to oppose modeling the world of pattern recognition because, first of all, pattern recognition is an important part of intelligence, and as I've already suggested, it will show more of. It's also probably an important part of how we model the world, especially how we do things tractably-- how we do things quickly. But I think it's really important to view-- to have this broader perspective on intelligence, and understand pattern recognition and the other ideas in that context.

Now, I think it's also fair to say-- I'm an optimist here, in most ways. But to be honest, right we are far from having an understanding of how the brain builds models of the world or how we could get machines to do it in a human-like way at anything like the level of maturity that we understand just pattern recognition. That means both on the scientific side, we don't understand those parts of the brain or those parts of the mind as well, as rigorously or as quantitatively. On the engineering side, we're far from having the toolkit that Silicon Valley can engineer at scale and make useful technologies with. But we're starting to make a lot of progress. And I'll talk about some of that.

And I really think this is a great area where you can see the brains, minds, and machines philosophy at work-- this idea that we have actually a scaling route to get to, to transform and qualitatively advance our understanding of intelligence by reverse engineering how it works in the human mind and brain.

And by reverse engineering here, what I mean is doing our science like engineers. That means trying to approach the basic scientific questions, but answer them with an engineer's toolkit. And again, as I think you've already heard in different ways as part of the premise of this summer school, our best reason to think that that might work in the future is that it's worked in the past.

So here's-- I know you've seen slides like this, but this is my take on just the history in papers of deep learning and reinforcement learning-- again, the tools that have been so useful in advancing pattern recognition today, approaches to AI. And I'm just putting up some of the original papers to emphasize when and where they were published.

Again, as you probably know, the original algorithms for deep learning and reinforcement learning were introduced in the 1960s, '70s, and '80s, pretty much. And they were published in journals mostly of psychology or cognitive science or computational neuroscience, like biological cybernetics or general science journals like nature.

But actually, more than anything, in psychology and cognitive science journals, which to me as a scientist, I'm proud of this. But I think it's striking. And it represents what is a very old theme whether it's Boole or Geoff Hinton or Judea Pearl, who was the most famous developer of Bayesian networks.

All of them were inspired by trying to understand human intelligence in computational and mathematical terms. They might have been trained as physicists or mathematicians or computer scientists. Most of them maybe weren't trained as psychologists, although some of them were. But that's what they were trying to do. They were trying to take that-- broadly that formal quantitative engineering toolkit and understand how humans think and learn.

Although to be fair, these algorithms are very small steps towards that. Most of them were more about-- more like how a rat might learn or a fly. They were really-- or think Pavlov's dogs-- very simple. These are the simplest, most basic processes, not of human learning, but of animal learning.

But what we've seen in history is that you take these-- even these simple ideas. If you can engineer them at scale, those simple, small steps become big and can change the world. So imagine what we could do with the next small steps. And this is really the research that I'm most excited about. And I'll try to give you a perspective on this, for both the morning and the afternoon. But it's also, it's really not just me. It's actually a bunch of people in CBMM are really thinking about our next small steps, which is really this big question.

So imagine if we could make simple, or I don't know. They may not be simple, but at least understandable models of not how most basic animal learning processes, but of how children learn. So imagine if we could build a machine, let's say, that grows into intelligence the way a human being does. That starts like a baby, and learns like a child. This is an idea that I'm especially excited about when I put on my engineer's hat. But it also motivates how I approach most of the cognitive questions that I work on-- scientific questions.

But within CBMM-- and you're going to get to hear from many of these people, those you haven't yet. There's a number of faculty investigators who are working on this project in their own ways. And we're all-- and there's a great network of collaborations between us. And our center has really enabled that.

And the summer school has also helped to grow that. It includes not just PIs, but a number of students, postdocs, and research scientists, some of whom are your TAs here, some of whom have been students here. And the dot, dot, dots are very important here. I'm sure I left off many people. I'm sorry if you're some of them. But it's exciting that this particular question is compelling to a lot of people. And I think we need all these different people's tool kits.

In particular, you're going to be very lucky to hear from Spelke, Schulz, Saxe, and Kanwisher, as well as Winston, Katz, and Gershman later on in this summer school. And they're all going to be talking in some form about ideas that contribute to this vision.

Now, this vision is much bigger than not just the CBMM people on this slide, or left off this slide, but well beyond CBMM, and much before us. One famous proponent of idea was Alan Turing. So well before those papers on deep learning and RL in the '60s, '70s, and '80s is this famous paper from 1950.

This was the paper in which Turing introduced the famous Turing Test as a way to measure potentially the other thing he introduced in this paper, which was the idea that you could think about intelligence as computation. And of course, he was just-- that was just becoming a formal, meaningful term through his work and the work of Alonzo Church and others at that time. And it was barely even a useful, practical engineering idea yet. It's mostly just math.

Now, another important thing he introduced in this paper was a suggestion about how you might build a machine to pass the Turing Test. He didn't really know how to do it. He said it's clearly going to be very hard. But maybe one route is to, instead of starting to build an adult brain in a machine, to build something more like a child brain because presumably that would be easier.

As he says here, just to highlight a quote, "Presumably the child-brain is simpler than the adult-brain. Presumably it's something like a notebook as one buys it from the stationers. Rather little mechanism, and lots of blank sheets."

So what he's identifying here is the importance if you're going to build. First of all, he's saying, this is a scaling root to intelligence, essentially. He's saying, this is the best idea I can come up with how to do this. And that was a good idea. And I think it's actually the best idea that anybody's ever come up with for how to scale a system to human-level intelligence because, again, when we look at the known universe, it's the only actual example we have.

Human children are the only system that we know that demonstrably, reliably, reproducibly starts off with, in some sense, much less intelligence than adult humans, and grows into that on its own. But if we're going to do that, we have to-- and think about it from an engineer's perspective. There are basic questions we have to ask, like, what is the actual starting state, and what are the learning mechanisms.

And Turing made some presumptions about this because, of course, we didn't know at the time. And what he presumed is that it starts off with maybe very little, maybe not very much. And it learns in some relatively simple way-- something like copying down things from the blackboard, or whatever it is that you do when you get lessons. That's the metaphors that he uses in the paper.

Now, I think one of the most important reasons why this is a great vision to pursue now-- this developmental route to AI-- is that we've learned a lot. We've learned better answers to these questions. We don't just presume now. We've learned some things. And you're going to see. You'll hear a little bit about that from me. And you'll hear a lot more about that next week when you have Liz Spelke and Laura Schulz, who are two of the world's leading cognitive development researchers, who actually studied babies, and probably some from Rebecca Saxe and others as well.

And I think, importantly, what we've learned is that Turing was wrong about this presumption. He was brilliant in many ways. And he was honest that he was just presuming. But he presumed wrong. And many people have made that wrong presumption. It's a common assumption or presumption that in some ways babies start off not knowing anything. And children's learning is some very simple process that just-- you just do it enough, like trial and error learning or something. And you just do enough of it. And somehow in some unsupervised or cell-supervised way, you come to be intelligent.

But what we've now learned is that babies' brains start off with a lot more than that. And children's learning is a lot more sophisticated than just very simple kinds of writing experiences down or trial and error learning. So I'm going to talk to you about these ideas. What have we learned about the basic starting state and the basic learning mechanisms? And again, that's kind of going to be-- that's one. It's another way to think about the themes of this morning. OK.

The rest of the first part of today, I'm going to focus on more like what do we start with. I'm going to talk about what's called the core cognition. This is Liz Spelke's term. Or my version of that term is the common sense core. It's a little bit different. And how we've been trying to build engineering models of that with the tools of probabilistic programs that you learned about yesterday in the tutorial, and some ideas that we call the game engine in the head.

I'll talk about also ways in which this actually relates to the brain, and both real and artificial neural networks, and how they come in and have actually helped to make this vision work in more interesting and effective ways recently. But it's not fundamentally a story about neural networks, although it meaningfully contacts them. And then, I'll move to talking more about learning and these ideas of child as scientist or a child as coder. Yeah?

AUDIENCE: Because before we move on, I want to--

JOSH TENENBAUM: Sure.

AUDIENCE: --a little bit about this idea of starting from the [INAUDIBLE].

JOSH TENENBAUM: Yes.

AUDIENCE: When you talk to a developmental neurologist they maybe want to put it even earlier--

JOSH TENENBAUM: Yeah.

AUDIENCE: [INAUDIBLE]. So I will ask your opinion on why do you think the children's brain-- or at what age, why is it a good starting point?

JOSH TENENBAUM: Well, for me, there's a number of reasons. It's because I'm using the methods of cognitive science. That means experimental, behavioral studies, and computational models. We can study experimentally intelligence in children and babies. And you're going to hear a little bit about that from me. And you're going to hear about that more next week. OK.

I don't know how to study intelligence in [INAUDIBLE]. But I'm quite sure that, I mean, I agree with the general perspective that understanding that will be important for understanding intelligence. So I see that as common cause. We're sort of on the same side. I just don't know how to use my methods to ask questions about intelligence there. But it's quite striking.

And again, Spelke, for example, or Laura Schulz have really pioneered new methods for how to actually study intelligence in very young children, and even in very young babies. And then, I've worked on, and other people like me have worked on computational models for actually understanding those things in engineering terms. But I think an engineering-style understanding of developmental neurobiology is also very important.

And I would push back even further to evolution. Again, it's very hard to study anything experimentally in evolution. But I think when you take this biological perspective, I would definitely say-- this is broadly speaking of a biological perspective on intelligence-- the developmental one. And it includes what developmental psychologists call development. It includes what neurobiologists call development. And includes evolution, which provides all the building blocks for this to work. Yeah.

So if I look forward to where our field is going over a number of decades, I look forward to having an engineering understanding of all of those things, and using the tools to integrate the understanding of that. And I think you'll see through in Rebecca Saxe's work and in Marge Livingstone's work, for example, it really does raise these very basic questions of, how does intelligence arise before you come out of the womb, I guess.

But jumping in on where we have a handle in our field, I want to talk about a-- this is again a set of ideas that has been developed by a number of different people. I'm mostly synthesizing, not proposing something new here. But it's a view. In some sense, this is a particular view on when I said model building versus pattern recognition, these are-- this is the model we're talking about.

But it also really has its roots in the starting state. If we want to say, how has evolution and development wired up our brain to build models of the world, it's wired it up, the view is, to build models of this sort. That from the very earliest ages, as early as we can look, humans, and not just humans-- many other animals because these programs are shared with other animals at this stage-- are set up to see the world not in terms of patterns and pixels, but in terms of physical objects and agents, or intentional agents.

We sometimes talk about intuitive physics and intuitive psychology. But what we mean are actual objects in the world that don't wink in and out of existence, for example; that have physical properties like mass, and interact with each other via things like forces, so intuitive notions of those basic physics concepts.

And then, by agents, or intuitive psychology, we mean particular kind of object which can exert force on itself and on other things via its body in order to achieve goals. The world has been made of these things-- objects and agents, as we call them-- for much longer than there have been humans in the world. That's been made a physical object since before there were any life forms at all.

But as long as there have been anything like animals, and not just animals, there has been what we call agents. So these are very old, enduring properties of the universe we've evolved in. And it's not crazy to think that our brains have evolved and developed mechanisms to understand things in this way.

Now, while these are shared with many other animals, they also get enriched and extended in important ways in humans, especially over the first few years of life, and especially once language-- natural language comes into the picture. I think a view that cognitive scientists and linguists have articulated very well, and would be very inspiring for me, is that this stuff that I'm going to be talking about is there before language. But it's the basic building blocks for meaning and language. So when we talk about semantics, you're going to-- again, you'll hear more about this towards the end of the summer school.

But when we talk about what do words mean, especially, well, really, what do children's first words mean that they are most in the position to appreciate and learn at the beginning around age like 1 to 1 and 1/2, it's, they ground out in this understanding. So personally, I think we should understand this first. And then we can understand how language works. And then, there's sort of a third phase of this developmental thing, which is once you've got language, how do you use that to learn everything else.

Developmentally, you could divide things up into the stage of like 0 to 1, or 1 and 1/2, which is mostly what I'm focusing on here, and then, from 1 and 1/2 to 3, where the main goal of-- the main achievement of the child cognitively is to use this common sense core to learn language. And then, from age three on up, it's to use language to learn everything else, including new languages, like math or Python or TensorFlow. That's your route to intelligence.

So the questions I'm going to talk about here are, how do we capture this kind of basic, original, core, common sense knowledge. And this is what I mean by common sense. Many people mean different things by common sense, but I mean this-- this basic understanding of the world in terms of physical objects, intentional agents, and their interactions.

How do we capture this in engineering terms? How do we study it quantitatively and rigorously, as rigorously with psychophysics and behavior as we've done anything in, let's say, low-level vision? How do we understand how it works in the brain, again as rigorously, as mechanistically as anything in just pure visual perception? And how does it actually arise through all the developmental processes that we've talked about?

To illustrate more concretely what I mean, I'll show you a few fun videos. These are two videos I've found on YouTube. And the first one is a 1 and 1/2-year-old illustrating some great intuitive physics, as well as some symbolic action plans. So here we've got this kid who has-- he's doing the popular 1 and 1/2-year-old activity of cup stacking. He's got a stack of three. He's trying to make a stack of four. But you can see it's not working that well yet because he chose slightly-- he's one off in the right sequence of cups.

So he considers his options. And he's got a great idea here. He's going to make a stack of two. He's found the right one. And he's going to make a stack of two, and put that on the stack of three to make a stack of five. But of course, there's some bugs in that plan, which he goes on to debug.

But think about the physical intelligence that he needs to do this-- to make that plan, and to execute it, and to fix it when it goes wrong. It's remarkable. We've made a lot of advances in robotics. And increasingly we're going to be seeing humanoid robots in our lives. But we don't have anything that can do that. By the standard of today's robotics, that's ridiculously impressive achievement. And he thinks so, too.

[LAUGHTER]

Yeah. But he's the one who deserves the real hand, not-- maybe you don't need to clap for him.

But also think about it from a symbol processing point of view. He has the concept of a stack. A stack can have any number of cups. You can take a stack and put it on another stack, and get a bigger stack. In some sense, he knows that. And he's using that to structure his plans in terms of goals and sub-goals. And he realizes he's-- he collapsed when he realized he's achieved a sub-goal, but he knows he hasn't achieved the real goal until a little bit later. And he's about to achieve it. I guess I will let him do that because it's gratifying to us as well as to him.

[LAUGHTER]

There he goes. Ooh, and that's cool. Yes, that one deserves a standing ovation.

[LAUGHTER]

Again, I used to say the robot couldn't even pick itself up. Robots are getting able to pick themselves up. So again, a lot of progress in robotics, but nothing close to the cognition needed to conceive of, execute, and then debug that kind of a plan.

Here's another baby, who's in some ways even more impressive. This baby is younger. This is a one-year-old. This is one of the most popular internet videos because it has both a baby and a cat doing something a little unusual. So here, this baby has decided to try to stack cups on top of the back of a cat. I guess maybe he's a philosopher in training. He's asking, how many cups can I stack on the back of this cat? Maybe it's three. Maybe, can he get-- maybe it's more. It's not--

[LAUGHTER]

At some point soon, I think he's going to realize that it's sort of an impractical goal, or he's reached its end. So that's not working so well. So he switches his goal to now, let me see if I can just get the cups onto the other side of the cat.

Now, that part there where he reaches back-- that's the most impressive part of the video to me. Although I should say, well, there's two impressive parts. So first of all, the idea that he should just switch his goal is like, well, I was-- first of all, the idea that where did he come up with this idea. I'm going to put cups on the back of the cat. That seems like a slightly crazy and maybe bad idea, but I mean weird idea.

[LAUGHTER]

I'm sure he didn't imitate that from anybody. But somehow he decided to try that out. So that's a cognitive achievement, in a sense. And Laura Schulz will probably talk about how that kind of idea of coming up with a weird but novel goal might be really important in how kids learn.

And then, his ability to very flexibly change it-- well, OK, I guess I've done about as much of that I can do. I'll change it to this other thing that's maybe easier-- OK-- putting the cups on the other side.

But then, at a lower level, the thing you saw when he reached back there, right around here-- reached back for that little purple cup-- what's cool about that to me is it's a very dramatic form of object permanence in the sense that he hadn't seen or touched that object for at least a minute, probably more. But somehow he knew there because he reached back for it, and only then turned around to see it. So somehow he was representing that in his mind.

Now, of course, that's structured by the fact that he knows. He's probably done a lot of things with cups. He knows how many cups there are. But he still knew this cup was there. So again, that's part of seeing this notion of intuitive physics, and the fact that it's not just about seeing. From this viewpoint, seeing the world is really about understanding the world and using what your eyes tell you as part of a lot of other parts of your brain to really build up a working representation of the objects in the world.

And again, I think Jeremy will talk about some version of broadly related ideas, not to bias you, but this would--

JEREMY: I'll talk about what the [INAUDIBLE].

JOSH TENENBAUM: Oh, OK. Well--

[LAUGHTER]

But the general idea of tracking the objects around you, perceiving them, attending to them, building working representations of the scene around you, is again, one of the most fascinating things that cognitive scientists study, and Jeremy is one of the world's experts on that.

Again, these abilities are not simply human ones. Here are a few famous videos of other non-human animals. Crows have been especially studied with their abilities for tool using. But think of tool using as just sophisticated plans you can make if you have an understanding of intuitive physics, Whether it's things involving objects in liquid or bending sticks to make tools to dig things out of holes.

Or over here, again, these are just-- these are from scientific experiments in the New Caledonian crow. These ones here are from the internet, again. This is an orangutan who is doing, to me, an incredible feat of intuitive physics. He's stacking up blocks to do what. Well, let's find out. We don't want to spend too much time on this. So I'll fast forward a little bit. The videos a few minutes long. So he's clearly thinking about something here. But give him long enough. And he seems to be doing something maybe even more impressive than that 1-and-1/2-year-old was doing-- yeah, building this whole tower.

Now this video-- it turns out there's some controversy on the internet over whether this video is a fake. The question isn't whether there's someone in an orangutan suit or something like that, or whether they used computer graphics. Some people contend that the video is played backwards-- that actually a human built that tower, and then the orangutan just disassembled it.

Now, it's very hard to tell. People have-- if you play it backwards, it's hard to tell. People have argued on the internet about fine-grained analysis of is the hair blowing this way or that way. And I don't know. I think if you imagine watching this backwards, it's almost as impressive. Imagine if somebody did build this tower for the orangutan. Try that.

So imagine somebody built that. And then, he slowly, piece-by-piece disassembled it.

[LAUGHTER]

That's also a pretty impressive achievement. He could have just knocked it over, if that's what he wanted to do. Right? So either way, it's an impressive feat of intuitive physics and planning.

And then, finally here, this is the famous mouse-versus-cracker video, which I think I first saw maybe even in this room at Woods Hole. Christof Koch presented this in one of his brains, minds, and machines lectures on consciousness. He showed this-- the valiant struggle of this mouse to get this cracker-- big, big cracker back to his nest.

He showed this as an example to illustrate surely the mouse is conscious. And he didn't mean it as a rigorous proof. But he suggested that if we look at this mouse struggling with this thing, how could we deny that the mouse is conscious in probably somewhat of a similar way that we are? And I find that intuition compelling.

But my point isn't about whether the mouse's consciousness is like ours or exists in some kind of subjective sense. But rather, ask yourself what it would take to build a robot to do that. And again, it's incredibly impressive. We are making progress in robots to do that kind of thing. But still, we don't have any robot that could just for itself figure out how to do that. Conceive of that goal. Execute that plan. Especially in this case, it's just too hard for him. So eventually he gives up.

AUDIENCE: [INAUDIBLE].

JOSH TENENBAUM: Yeah. No. But being an optimist, I wouldn't end on the depressing note of him giving up. So he does actually go back and finally get it.

[CHEERING]

Yeah, he makes one last try. He deserves a hand.

[APPLAUSE]

So one last cool video, and then we'll get into some actual substantive science. And, well, we're working our way there. But again, I hope you'll bear with me on some of this gentle introduction because we're really trying to present a whole perspective on ideas that will also be partly an introduction to a number of other lectures you're going to see from other speakers.

So the things I've showed you so far are examples mostly of physical common sense and plans around that. But a lot of the study of human intelligence is more the social side, or what we call intuitive psychology or understanding of agents.

And here I'm showing frames and videos from experiments done by developmental psychologists who work with human infants. And infants typically means 0 to 2-- age 2, although we'll often talk-- we'll say older infants or very young children if we're talking about 1 and 1/2 year olds.

And a lot of this work has been, again, to understand the basics of goal-directed action and how people-- even young people understand it. So how is it that you look at a video like this? This is a stimulus from an experiment done with 13-month-olds by Southgate and Csibra, where you look at this video and you don't just see balls rolling on a plane. But you see something that looks more like a story or agents doing something. How would you describe what these balls are doing?

AUDIENCE: [INAUDIBLE] chasing.

JOSH TENENBAUM: Chasing, yeah. It looks like the blue ball is chasing the red ball, and the red ball is also doing something, right?

AUDIENCE: Running away.

JOSH TENENBAUM: Right, running away. Yeah. Those don't have to go together. You can chase a hurricane, but the hurricane's not running away from you. It's just moving erratically. Or you can flee a hurricane. But in this case, you have a joint kind of game-theoretic competition.

Now, here's another question for intuitive psychology. Which of those balls is smarter?

AUDIENCE: The red one. [INAUDIBLE].

JOSH TENENBAUM: Yeah. How many people say the red one? Raise your hand. How many say the blue one? OK. Most people say the red one. OK. Why is that? It's interesting. And how can you tell? This question actually cued a really cool summer school project a few years ago that has now been led by [? Marsha ?] [? Krieven ?] that led to actually not only a summer school project, but several published papers. And she's now a postdoc in our group, extending this work.

So well, one way to think about it is to say, well, it looks like the blue ball-- it's more like the blue ball looks a little dumb because he seems to think he can fit through some holes that he can't. He keeps trying to go through those little holes, and not getting there.

But also, he doesn't seem to learn his lesson. He keeps making the same mistake over and over again. And we have intuitive concepts of intelligence that are about, well, if you're perceiving the world, you should see basic things, like how big a hole is versus you should know. You should know some basic things that you have. He clearly has some perceptual evidence because he's-- we don't see his eyes. But he clearly has some sense data about where the red thing is. So why can't he see this? What's wrong? Are you blind? Are you dumb, not paying attention?

And then, you should learn if you make the same mistake a couple of times. If it's literally the same mistake, you should stop making it. So those are parts of our intuitive concept of intelligence. And we can apply them even to these balls. But go back and say, how do we even make the guess or the inference that there's chasing and fleeing. If you look closely, most of the time there isn't a strong image motion signal, like the kind of thing computer vision or like MT in the visual cortex would compute.

That's that base that has the blue ball moving towards the red ball. Most of the time, it's not literally moving towards the red ball. It's moving only conceptually towards the red ball. It's followed what you might say is like a geodesic or shortest path, subject to the constraints. It's following an efficient path subject to the physical constraints, which sometimes takes it orthogonally, or in all sorts of different directions.

But we see that efficient path planning subject to constraints, and most interestingly, not actually just subject to physical constraints, but subject to our understanding of the blue ball's incorrect understanding of its own physical constraints. We see it making these plans in which it thinks it can fit through some holes which it can't, which is why it keeps making these false starts and then going around.

We see all of that. And somehow even babies see some aspects, then. We don't know how much of what I've just told you they see. We know we basically see that. And I'll show you some experiments later that we've done with human adults showing that, yeah, you can measure that very quantitatively-- all that stuff.

But even babies see enough of it to understand this as chasing and fleeing. And if you remove the obstacle so that you just see the blue and red balls with exactly the same pattern of motion without the obstacles, they don't see it as chasing and fleeing. And you wouldn't see it that way, either. You would see it as like animate dancing. They're clearly a live agent, and they're interacting. But you don't see the chasing and fleeing without the obstacles because it doesn't make sense.

So the understanding of efficient action subject to an agent's beliefs about its environmental physical constraints is at the heart, at the core of our intelligence. And I'm going to talk to you about how we describe that in engineering terms.

The last video is from what is one of the most famous experiments in recent developmental psychology, certainly in development of social cognition. It's by the psychologists Warneken and Tomasello. And it was part of a study done with humans and chimps. I'm just going to show you one of the humans.

This is another 18-month-old. And again, I put this together with the other 18-month-olds I showed you because it really-- to me, together this package, I think motivates why this stage of cognition is so interesting to study.

What you're going to see here is one subject in the experiment, which is representative of what other kids do. But this is just one particularly cute kid. This is the-- the subject is this guy back here. This is Felix Warneken, one of the experimenters. I'll turn up the sound. There's sound. There's not language. But there's expressions-- nonverbal sound expressions.

And the kid has seen something that you've seen, which is a novel action. You've probably never quite seen that. But look what he does.

AUDIENCE: [INAUDIBLE].

JOSH TENENBAUM: He opens the door, looks up--

[INTERPOSING VOICES]

--down. OK. So just watch it again. And think about what has to going on inside the kid's head. Here, I'll turn off the sound. Put yourself behind his eyes. You're seeing this weird action that you've never quite seen exactly that before. But yet, somehow you're able to understand what the person is trying to do and how to help them. And then, sure enough, he goes and does it.

Now, here my favorite part is what's about to happen when he steps back. And watch how he looks up and makes eye contact, and then quickly looks down. So you see he looks up, and then looks down. Right? Did you see that? OK.

So that eye contact-- again, I'm going to interpret this. But I think it's fair intuition to say that when he's looking up and making eye contact, he's sort of trying to see, did I get your intention? Right? Did I figure out your goal? And if so, then he makes what's called a predictive look down at the hands because if I got this right, then you're going to look here.

And the next step is you're going to do something with your hands. Namely, you're going to put the books in that shelf that I saw there, or somehow inside. So he's checking. Did I do the right thing? And do my expectations now come out? At least that's one interpretation of what he's doing.

So again, think from a robotics point of view. We have robots that have the physical form factor of 1-and-1/2-year-olds right now. We're close to having a humanoid robot that could do all of that if we could just give it the brain and mind to do so. And we're not there yet on the technology side. But I think we're starting to understand much of the basic math and computation needed to do that.

So that's the program that we're developing here, is to try to understand how these things work in computational terms. And what I want to show you is the ways in which we've been building engineering-style models of the mental models of see and understanding and intuitive physics, and over here, intuitive psychology or understanding agents' beliefs and desires and plans built on physics that allow kids to do this, and as well as you to do this. So I'm talking about kids, but I'm not just talking about kids.

So these sketches here-- these are sketches of the mental models that a person has. And I'm going to tell you how to build scientific models of those mental models. But when I was talking about pattern recognition versus modeling the world, I mean these kinds of models. All right. So how do we use these to explain, understand, imagine, plan, solve problems? And how do we learn? Build out these models or make them.

And the toolkit we use is-- and part of it is the toolkit you already learned in yesterday's tutorial, so great. This isn't going to be a highly technical talk, obviously, but more of an overview about things that we've done with probabilistic programs to capture these models.

So these are-- if you like, these are sketches of the probabilistic programs. But to make them actually work, you have to use tools like the ones you learned yesterday. And again, I think as you already saw some exposure to in Kevin and Kelsey's presentation there, and if you've-- the web book that you guys were working with-- you guys did interactive things with the [INAUDIBLE] WebBook, yeah. So you could read more of that if you-- it's a good introduction to these ideas.

But hopefully you've already started to see how these tools combine the good ideas that I've talked about-- the big ideas of intelligence that I talked about at the beginning of today. So it's symbolic languages, probabilistic inference, the ability to do causal models in hierarchical inference, to be able to do inference in a sense about inferences to-- you can see. If you read to later chapters of the book, you can look at hierarchical inference, learning to learn, things like that-- not just defining a model with a certain prior, but actually being able to infer the prior with a hypothesis base of hypothesis basis, as we talk about it in the hierarchical base, or priors on priors.

You probably didn't use the neural network integration tools. Is that right? Did you do anything with that? Probably not. But WebPPL have some tools for doing that. And increasingly, the current new generation of probabilistic programming languages, which includes languages like Pyro, which is a new-- so WebPPL was developed by Noah Goodman, who developed an earlier version. And there's an earlier version of that WebBook in a language called Church, named after Alonzo Church WebPPL and Church are very similar. WebPPL just a little bit more mature. And it's based on JavaScript instead of LISP. So some people like it better. Some people like it worse.

Pyro is another exciting language that Noah and colleagues have been developing as an open source project at Uber AI Labs, where Noah now works part time. And it integrates. It's built on top of PyTorch. And it integrates tools from deep learning with the same kinds of ideas that you can see in WebPPL.

Another exciting example of this is being developed at MIT by Vikash Mansinghka, who, with Noah and others, was one of the co-developers of the Church language. And Vikash and a student in his group, Marco and others, have been developing a language called Gen. It stands for generative models. And it's also a way to combine probabilistic programs with TensorFlow type of neural net models. And it allows you to implement, relatively straightforwardly, many of the models that I'm going to be talking about here, particularly on this kind of physical scene understanding side.

Now, a key part of the toolkit-- this is one part. OK. But another key part, which you heard a little bit about yesterday, and WebPPL has some limited but actually quite cool support for, are various-- it's not just the idea of probabilistic inference over program traces, which you learned about yesterday, but particular kinds of programs, which are particularly suited for-- in fact, designed for modeling things like the physics of objects.

And what I'm broadly referring to here is a suite of programs for simulating the world of objects and agents, which if we believe that that's what core knowledge-- the original sort of common sense is about, then if we want to build models of people's models of those things, then we need programs that capture how they work.

And lucky us-- somebody's already built a lot of those programs. The game industry has built at least a first-pass cut of programs that capture, we think, certain aspects-- key aspects of core intuitive physics and even intuitive psychology. And they've done it as part of tools called game engines, which again probably many of you have worked with.

I'm just curious. How many of you have worked with a game engine like Unity or Unreal? OK. How people know what those are? OK, some-- most people at this point. For those of you who don't know, these are tools that the game industry developed over several generations of games that, at this point, allow a designer to design a new game much more easily than they would have otherwise because there's a lot of things that they don't have to write from scratch. They can focus on what-- the story and the characters and the objects and the world rather than writing all of computer graphics.

So the most basic thing in a game engine is a graphics engine, which renders in real time what the world looks like as a player is exploring it and interacting with it. Game engines also have a physics engine. And many, if not all games have physics on some of the objects. And that's important in the game for making it interactive and open ended. So when you want a player to-- not all video games really use this, but a lot of video games. Most video games in some form give you low-level controls, like move around pick things up. But then, you can do anything you want in the world with your avatar.

And so, if I'm in a world that's like this, and I want to imagine something like, well, let me go over here. And I want to be able to pick up this object. And I want it to come with me when I pick it up. And when I let go of it, what do I think will happen? Well, I want it to-- I think it will fall. I think it will hit the floor. I think it will not hover in midair.

It might bounce a little. It might roll. If I do it right, it might roll well. Let me see if I can-- if I just drop it, maybe it will bounce. I don't know. I'm telling you. And hopefully you're imagining the same thing. Oh, it bounced a little bit. Right, OK. Didn't expect too much bounce. But if I kind of throw it a little bit, I can get it to roll a little bit. Yeah, OK. So there it worked. OK.

So game engines will let me do that-- will let me imagine that, and then do something like that. And it might happen. And maybe that's the secret to solving a level or escaping from a room. Probably there's something else I'd want to do with the duct tape than that. But maybe that's what I have to do.

So those tools, we think, provide a first approximation, maybe-- surely wrong in all sorts of ways, but interestingly right in some ways, and a place to start building on for effectively what evolution and development have put into our brains and other animals, too, for building these causal models of the physical world. And also, these game engines have kinds of what's called game AI, which are basically models of these kinds of agent models.

Again, this picture here-- just to be clear-- on the right, is not-- this is not a model of a child's brain or a person's brain. This is a model of a child's model of people's brains, or sometimes called theory of mind or intuitive or folk psychology. It's a model of how we understand agents, that we see them take actions in the world by implementing efficient plans subject to their beliefs and desires. And we see their beliefs as some function of where they are in the world, and a model of perception. And their actions change the state of the world via physics.

So not all video games use these kind of tools. But increasingly game AI, at least at its height, has tools for things like this, like for having a guard who can guard a base. And instead of just standing there and shooting randomly every so often into the air, the guard will see when the player is-- let's say the player's job is to infiltrate the base. And they have guns and can shoot each other. So the guard will stand there and look around. And when the guard sees a player, will maybe start shooting at them or go after them or follow them.

And you, because you're cleverer than the guard, might lure them out, and then once-- and lure them down a path, and then sneak in or something. So if you want to be able to interact with the guard in that interesting way, the guard has to be able to be fool-able in that way, which means you have to understand this. And it has to work like that.

So the basic approach here is to take those kinds of programs, but wrap them inside a framework for probabilistic inference and learning. And that allows us to capture these aspects of basic common sense. Does that makes sense? OK, good.

Description: 

Josh Tenenbaum, MIT
Past work on human intelligence has framed the underlying processes as pattern recognition engines, as manifested in deep convolutional neural networks; prediction engines, as captured in Bayesian networks, causal models, and predictive coding; or symbol manipulation engines grounded in logic, the lambda calculus, and high-level symbolic programming languages. Systems that can reason broadly about the physical and social world must embody models of the world that enable explanation and understanding of what we sense, prediction of future states, problem solving and action planning, and learning of new models with experience. Such systems must at least embody an intuitive physics and psychology that governs the behavior of objects and agents in the world, which may be created through an approach that starts with the intelligence of a baby and learns like a child.

Lake, B. M., Ullman, T. D., Tenenbaum, J. B. & Gershman, S. J. (2017) Building machines that learn and think like people. Behavioral and Brain Sciences 40:e253.

Associated Research Thrust: