Life as GIM in a GAN psychophysics of Cephalopod camouflage & perception: some conjectures
Date Posted:
February 26, 2020
Date Recorded:
May 21, 2018
Speaker(s):
Jonathan Miller, OIST
All Captioned Videos Brains, Minds and Machines Seminar Series
ANDY: I'm Andy, and I'm a postdoc in the Poggio Lab in CBMM. I'd like to welcome you to our talk by Jonathan Miller. Jonathan is leader of units for physics and biology at the Okinawa Institute of Science and Technology in Japan.
Jonathan has a double PhD. He did his first PhD in the other Cambridge in biology. And then a PhD in physics in Caltech and then went on to a career both in condensed matter physics and then in biophysics. And he's here to talk to us about camouflaging octopi, and cuttlefish. And without further ado--
JONATHAN MILLER: Oh, Andrew, thank you for hosting me, and thank you for the introduction. One thing you left out is that my first PhD was an experimentalist. And that proved to be quite a disaster from many perspectives.
So ever since then, I've been a theorist. And, at the moment-- it means other people have to provide the hands and ensure the safety of the animals. And they tend to keep me away.
So let's see here. In the abstract-- published abstract-- they didn't capitalize GIM and GAN. So people were-- well, I know some-- I guess, my sister was googling it and was extremely disturbed because she did not recognize that they were abbreviations.
So I called it-- well, it's kind of a subtitle if you like. Life as GIM and a GAN. And there's equal with question marks in there. And that's something might be worth discussing at another time.
But, for the moment, you all know that GIM means Generative Image Model. And I've included a definition for the sake of completeness here. But everyone here presumably knows it's-- if you want to produce a set of images that got representative of some ensemble.
Actually, both of these-- when I looked both of these definitions up-- in physics, we try to maintain a distinction between principle and algorithm. And when those get mixed up, I get a little confused. So I think these algorithms are applicable in their way here. But I also think that there are many principles that supersede them or circumscribe them-- include them.
So GIM and GAN appeared in two distinct roles in this talk. Again, conventionally, is thought of as consisting of a generator and a discriminator. And I want to make-- I think it's more than an analogy between the generator with an octopus and cuttlefish and discriminator. It is a predator.
I think it's more than an analogy. To understand-- this is an important video. It's well-known if you haven't seen it, it's worth a couple minutes of concentration. And this will come twice.
So if your attention flags, there we go. So we'll see this one more time. That's a common octopus. So this is the actual star of the show. The cephalopods, they are just the co-stars.
But there he is-- same guy. And we'll see that again. And this video is around 15 or 20 years old. The-- kind of speaks for itself. So an octopus is a cephalopod. Here are some sketches of them. And you eat them. And they share some common traits.
This is a cuttlefish. And it's showing some texturing and coloring. And here, this is the last 40 seconds or so. And it's a little tedious maybe for 20 or 30 seconds.
But keep an eye on this guy. It's a different species than the other ones. Because they're always a little slow here. He will astonish everyone shortly, I do believe if he performs as usually does in this video.
And we'll stop this particular video. It goes on and on and on and on. These are the elements on the skin that are the mechanics of the color change. And they're kind of pixels. And they're controlled by-- motors by muscles.
So, Roger, who is the gentleman who-- the diver who took those pictures, he visited us a couple of years ago to look at the local octopus, which is a closely-related species to the one you saw in the film. And here, he went diving outside our Institute. And he followed one of these octopuses for several hours or days.
This octopus, cyanea, as it's called-- the day octopus-- staple food source of parts of Africa. And what Roger was concerned, and it may affect whether he comes back or not that while he's videoing it to run-- following it around-- a fisherman may come by in spirit. Because that's what they do there. They eat them, which seems like a loss. But it's only a couple of years old-- this one.
So now, well, this is a little dense. And, by the way, it's not an outline of the talk. It's an outline of the next 10 years of research.
But, nevertheless, there is, I think, a concrete proposal-- maybe a little bit of insight here. Some of you will probably tell me you've thought of it, which is fine. Please, tell me where.
But you'll see little is-- I'll say again-- little is known about what's going on here. And I'm not interested, per se, in the mechanics. I'm interested in the computation. And the mechanics are remarkable and a challenge in themselves.
But marine biologists-- they have to learn to dive. And they're trained very traditionally. And when I first ran into some of these pictures as a theoretical physicist, I wondered, well, there should be an i/o transfer function. It might be a little complicated-- nonlinear. It might be time-dependent. I don't know.
But there ought to be something. Nobody seems to have thought about in this way. And, I mean, the novelty is simply that they're outside my window, right? But, certainly-- and I think that it can be-- possibly should be thought of in this way. And I'll suggest some sets of experiments that might elucidate it.
So I'll say this again. But so don't bother the details. But the idea here is that while the octopus here is telling us by taking of the shape and color and texture that it does that about where it thinks it is, right? It is telling what it thinks it's nearby, right? And we'll get round to why it's trying to-- it's telling us this.
And, so somehow, you'd think that if we take some image of this whatever this thing is-- fungus, I don't know-- image, skin, and then iterate this process. Well, that's what the cuttlefish is taking in-- reading the image through its eyes, reproducing something on its skin, that this might converge to some kind of fixed point.
And it might be unstable-- fixed point in which case it won't converge. But that's a different story about what can still deal with it. And then linearizing around that fixed point, there's an eigenstate in some big feature space.
And, well, there's an eigenstate. And I think it's probably what we would call a feature. And then once we've got those things, we can actually stick electrodes in, which is something we can't do for people.
So I'll return to the ideas here. So why is this thing on my window? Well, I think it's really difficult to see here without a telescope. But this is a small island in the middle of the ocean. And you can see the coral reefs here. And the coral reef is where certain of these most colorful and richly displaying animals spend their lives.
And our little institute here-- about 60 faculty now and lab 5 has just been funded. So we're going to grow to 100 in the next few years hiring 6 to 10 faculty a year-- funded roughly at a rate of $2 million per year per faculty member. Of course, you don't get that, I mean, some of the faculty get that, right? You know how this works, right?
And we're hiring an AI. And computer science has begun. We've got somebody-- a couple of people this year. And we'll be expanding in future years.
So just to get back to where we were. This octopus swims around out there in the coral reefs. And we also have a-- I get people to take care of these animals in a wet lab. And they keep me away as much as possible.
People, you can everybody's probably seen the eyes by now. Can you raise your hand when you see the body? OK? Good, good-- not everybody. Good, that's great. OK, it's a real challenge actually.
So the body's in here. And this same animal-- when it's not exposed-- this is the same animal, right? And the textures and colors are quite different. And here they are.
This is side by side. So this is just a matter of timing. This is a little video of some of the animal texturing and with some-- yeah, well, you can see it's busy doing something. I don't know what that is.
And that's one of the challenges here, by the way, and why, as a theorist, I have to get involved. And I have to have a wet lab. I'd rather not. I'd rather not have anything to do with any kind of living organism, in general.
But the problem is that the experimentalists, and, particularly, in biology, I think you know how it is. You can't always report what you see. Because what you see might not fit what the experts know. And then you're really stuck.
And one extremely common issue these days in biology is that things have a certain ideal behavior. And the person gets their Nobel Prize or whatever for describing that ideal behavior. But that may not be typical. And, ordinarily, there may be substantial fluctuations. And it will not get into print
Yet, if we want to do what I've described, we really need to know what these things are. They can't get lost. So speaking of Nobel prizes, this is a little-- it's a little artwork. This is monster, which is a cyanea-- same kind that you saw in the other pictures there. And we have a postdoc, and Nobel laureates, and a cyanea. And you're supposed to figure out which is which. We'll leave-- that's a challenge for another time.
So this is a cuttlefish again in our marine station. And we've got the same-- it's probably the same animal against two different backgrounds. I'm just trying to-- this point, give you some flavor of phenomenology as we have-- there are much more detailed descriptions. And, yet, from the point of view of an i/o transfer function, they're not necessarily useful.
So, here again, the same species. And, let's see, we've-- so these chromatophores as the color pigment cells are called are used not only for camouflage but also for communication. And so they're under selection for multiple purposes.
So, for example, this is the stereotypical hunting behavior. You saw that. It kind of-- and this is not behaving. Let's try it again.
So this is that stereotyped hunting behavior. This guy is hunting. Because there is a shrimp in there only it's a live animal at this stage, I believe unless trained. Then I think he or she gives up.
A few misses. I mean, what does it actually try-- what is the intention of the animal here? There is speculation about it, that's all. And they tend to hang out all together.
And the last shot in here-- oh, you see some of the three dimensional texturing here. And what makes it decide to exhibit a certain three-dimensional texture? No one really knows.
And this ends-- our video ends, I think. It certainly does end. But, shucks, see if I can get it to end where I want. It ends, I think, with a couple-- I'm not going to-- maybe I'm not going to get it.
So the animals-- this particular species likes to hang out above stones. And it takes on the texture and color of stones. So this is a different species of cuttlefish and I want you to raise your hand when you find the two cuttlefish in here. OK, OK, OK, OK.
All right, so this is-- that is this kind of a flavor-- if one did a force-- first, one might ask you how long or might ask a predator how long it took to find those. You see one here, and one there. And if they exchange sides, they would adopt the corresponding colors and textures.
And they're not transparent, by the way. The one person I showed it to observed to me that this wasn't fair what I was showing them. Because I didn't say what the relevant scale is. Are you looking for something this small, or this big or that big?
And he's right, it isn't fair. But life isn't fair. And the predator isn't going to know what scale either.
The predator's swimming by. And at some distance, if it's too close, it'll eat the animal, right? But, otherwise, it could be swimming at any kind of distance.
And the animals pretty decently [AUDIO OUT] vary in its shape. It's pretty decently scaled invariance over, I guess, a couple of years of growth and more than an order of magnitude. So it really isn't such a bad-- I'm not really cheating by not telling you that in advance.
But what's salient, though-- I find salient, again, is that there's some background. And the cuttlefish is telling us in the process of trying to hide from the predator-- it's telling us background it thinks it's in, right? I don't see any other way to interpret it. And I don't know what other-- I'll come to that. I mean, humans can kind of do this if you ask the right questions.
So octopus genome was recently sequenced. [AUDIO OUT] care about this stuff. But it's worth remarking that the octopus or the cephalopods are up here on this tree of life.
And we diverged with them from about 700 million years ago, which is a little while. And the octopus has a brain. And people talked about being one of the biggest brains in the invertebrate world.
They don't tell you is this about all optical lobe. OK, so it's not like it's got a big cerebrum or riot or a cerebral cortex. It's got optical cortex. And everything else is practically dwarfed by the optical vertex.
So, dwarfed by the optical cortex that in these CT scan images of developing embryo brains, the optical lobes, which would go out like that and like that are not even shown because they interfere. But so they're really way out. And at least at this stage, they're way out here-- massive objects. And they've been removed for clarity.
You'll also notice that there's a hole in the middle, and that's for the esophagus. So the brain actually is a genus-one object, which-- so people wonder how it's reasonable to ask, has animal found the same solutions to identifying-- to processing images as we have?
This is something-- it's found some kind of solution. Because, apparently, it can-- it is telling us that it can recognize this [AUDIO OUT] backgrounds-- visual backgrounds. And it might have converged to the similar kinds of solutions, and that would be interesting. On the other hand, maybe it found different kinds of solutions.
I mentioned earlier-- I guess this-- this is kind of a joke. I don't know if anybody will laugh. But so biology is complex and difficult and challenging. And one of the difficulties is-- and it really comes to light fairly-- really in a minute to explain why the natural questions that you're all thinking of and would raise your hand and ask, I can't answer.
And, really, it's one important point is that there's a tendency to not report things that don't fit the expectation. So this involves a logical calculus, which there is no contrapositive here. No biologist would approve of the contrapositive of this thing.
But it [AUDIO OUT] I'm not sure it makes a lot of sense, but, nevertheless, it's kind of a reality, right? There are experiments that people have not had a chance to do. And, these at face, remains explored.
So I think it's roughly [AUDIO OUT]. It's really the first thing or one of the first things that I think that that last postulate there was relevant for. So we think all the optical input is through the eyes. The skin contains opsins that are principle light-sensitive.
But there've been claims that the skin can be cut out and remains light sensitive. But it's probably a mistake. It's probably heat detection-- the light's heating it up.
These animals are more or less optic lobes with legs. Well, that's a physicist's idealization. The octopus and cuttlefish are colorblind.
Well, look that's what-- everybody agrees about this. And, yet, it really is a little bit surprising. It means that somehow in the marine world, color must be correlated with texture. And it's built-in so the [AUDIO OUT] octopus recognizes is-- a color is non-orthogonal direction.
But this is based on one octopus. They looked in one species of octopus's retina. And they found one photoreceptor. So I think it remains to be determined.
Other issues we face-- the animals don't always cooperate. They can be tired, bored, or hungry. What we'd like to understand is what is the space of skin pattern and [AUDIO OUT] for any given species of octopus and cuttlefish?
You could imagine if you could follow a cuttlefish around in the wild for a long, long time, and it didn't get eaten, and you could at least understand what that individual-- the space available to the individual.
But that's more [AUDIO OUT] remains more or less impossible. And even to just look at a large area of ocean, and try to see all of them and accumulate statistics that way is probably not practical.
So we need to do it in the lab. But it's far from clear that we know how to get a cuttlefish to explore its space in the lab. And more on any particular background, what is the typical pattern if there is one? What is the distribution of patterns and textures?
So this is just to cover myself. Everything we do here is-- yes, that's right. That's right. So let's see here. What drives the evolution of camouflage? And I think I've [AUDIO OUT] alluded to it already.
Oh, please. OK, this is a mystery. So I'm just going to show-- I'm just going to show a [AUDIO OUT] and a cuttlefish. But I can't start this one. What about the next one?
Yeah, this is a good one. So we've talked about adversarial-- I mentioned a generative adversarial network, [AUDIO OUT] and
ANDY: You should press the play button on the--
JONATHAN MILLER: Press the--
ANDY: --on the left button.
JONATHAN MILLER: On the left.
ANDY: That's right. See how we're doing this? Yeah.
JONATHAN MILLER: OK, so this is a crab and an octopus. And on the one hand, at this size, the octopus can eat the crab. On the other hand, if the octopus were much smaller, the crab could and would eat the octopus.
And I think this has interesting lesson about two competing agents. And, although it's slightly artificial, I'll get to that in a second. But the crab-- yes, crab could injure the octopus.
But I think mainly people seem to think its main concern is [AUDIO OUT] the hell out of there. But the strategy for doing that is perhaps of some interest. That's really the critical element here.
But so I think the point [AUDIO OUT]-- that's a seal. And the issue here is this seal, actually-- the divers have been diving there with their lights. And the seal figured out that it's a good place to get dinner. So it's always hanging around. So that's a little artificial But this is what goes on out there. It's brutal.
So octopus is a tasty meal for all kinds of fish and for mammalian predators. The predator wants to perceive the octopus's signal. The cephalopod octopus wants to be perceived by the predator [AUDIO OUT] as noise.
So the perception of the predator drives the evolution of octopus camouflage. This seems natural, right? I mean it's something that's happened over millions of years of evolution, right? So I can't-- I mean, this is a teleological argument. But I think people [AUDIO OUT] agree plausible.
So as an incidental gift to humans, the octopus report to us it's own perceptions of its surroundings. And it's a rare talent for an organism to do such a thing. Well, I don't know.
So question asked the audience, [AUDIO OUT] what animals can report to us their perceptions in sufficient detail in a way that we can readily comprehend? Well, all fish-- this was a hoax as you may know-- the painting elephant, chameleons-- limited range, slow. Chimps are not that good at [AUDIO OUT] to my awareness.
It may be that it's more common among fish than we believe. This is not. Parrots and zebrafish-- as I talked-- and zebrafish. Well, a parrot can certainly report to us its perception [AUDIO OUT] for a detailed way-- it's auditory perception, right? Because it can repeat what you say. It can mimic.
OK, but this is an expensive experiment with a parrot. [AUDIO OUT] I don't know if there are any other animals that can do this in such great detail.
And it's not that they're a matter of other animals being stupid or this animal being smarter, right? Because honeybees can convey to each other where they've been as you know with their feet-dance-- at least that.
So an octopus, as far as we know, can't tell each other anything about that. So is an octopus intelligent? And what I have to say is this. We [AUDIO OUT] probably agree, that all these guys are charismatic.
But shortly after June 13, it may become evident that the bacteria are the smartest. I'm serious about that, right? Intelligence, and, I think, where this awareness of this is generally basically [AUDIO OUT] growing slowly in the biological community is intelligence about what?
I mean, octopus are terrible learners, for example. Tortoise is a better learner. A stingray is a better learner. This octopus can open the jar but never seems to improve at it. It does this very well. The second time [AUDIO OUT] as it does. It doesn't ever speed up.
So I'd argue that bacteria are intelligent. And I'm happy to have this argument with anybody who wants to. But I would say I'm not a species-ist. And I believe it's like superheroes. Animals are like superheroes. Each have their talent.
And, really, [AUDIO OUT] case that one is smarter, one has to actually say what one means by intelligent. And this is very hard and something I'm interested in discussing.
So let's see. Yes, it will become evident. Nevertheless, it will become evident that the bacteria are the smartest. And to me, if I survived-- won't be evident to the bacteria, right?
So, that, I think is certainly salient. So then to predators, the octopus wants to appear as noise, predators want to see them as signal, and this is a generic evolutionary competition that leads, presumably, to some kind of local optimization.
And what's novel here is the timescale of these changes which are not evolutionary. Well, the changes that the octopus is displaying-- that the repertoire of textures and colors may have arisen on evolutionary time scale, but they occur on the scale of seconds or maybe minutes. The uptime for one of these chromatophores-- these color pigment cells-- is milliseconds.
So what is the background for which the evolution has presumably optimized the octopus? The ensemble of natural marine scenes, right? It has not seen a terrestrial forest image ever.
And it would be a surprise if the octopus could actually camouflage in a forest image. Nobody knows the answer to that. But at least among-- some workers agree that the octopus is not going to give you the Mona Lisa, right? You can show it the Mona Lisa, but it's not going to display the Mona Lisa. Others in the field actually are still [AUDIO OUT] for that Mona Lisa.
So what are the probes of perception? There are several. And the one that we are trying to apply here is [AUDIO OUT], I guess back to Helmholtz and maybe Young, I don't know.
So you know we have red, blue, and green photoreceptors. But they weren't characterized physically until-- I don't know-- until 100 years after Helmholtz figured out that this is how we see color.
There are other probes of perception-- hallucinogens. And we tried that with the octopus, that's another story and is not for this talk [AUDIO OUT] audience, I don't think.
I don't think it's too unnatural. As I said, I'm not interested in what were circuits or mechanisms but the software-- the copy [AUDIO OUT] principle of the calculation.
But what could that be? And so let me-- we've been a little slower than I thought. So what could it be? One obvious thing is the photographic replica.
And I'd say that when I got in the field, everybody in the field that I know or that I heard of second-hand, this is what they thought was going on was that the animal is doing some kind of pixel by pixel recreation of the scene. And it quickly-- it's got a retina. It's got an optical fiber. Whatever goes in has to be tested. It has to go up that fiber-- has to do it pretty fast.
I don't think that these camouflage events happen in 100 milliseconds. So maybe a real serious calculation has to be done. But I think this is just about impossible.
But I'm not going to rule it out. It's what everybody believes. Other possibility-- deformed or noisy photographic replica. [AUDIO OUT] might make some sense. You've got too many bits in this figure to cope with.
So if you compress it somewhat-- so we add these pictures. Yes, there's the picture here of these purple-splotched rocks which occur quite often around the coral reef, or maybe the animal sees a purple splotch and accesses the purple-splotched rock neural circuit that stores a single-quenched image of a purple-splotched rock. And maybe that's what it's doing here.
More likely, I think is it's accessing a circuit that stored [AUDIO OUT] features of some ensemble of purple-splotched rocks. And we think these are genetically encoded.
So let's see here. So a kind of-- I'm getting into tie-up phase right now. What I hope you've [AUDIO OUT] is that the octopus is somehow perceiving its surroundings in the ocean. And it's reporting them in the process of trying to evade detection by predators. It wants to be noise in the marine background.
And I've shown you a couple of examples. There's this one, and maybe this one, and maybe a few others that kind of suggest that these might be in the neighborhood of some kind of fixed point where what I mean specifically is the fixed point of some kind of iteration.
Right, an iteration of showing the animal some image. The animal responds with some pattern. We now take that pattern and show it to the animal. And we iterate this, right?
And what [AUDIO OUT] happened? I don't think-- I think these images I've shown are pretty suggestive. But I think it's quite unclear at the moment.
We [AUDIO OUT]. So I don't know if they-- I hope the case that the cuttlefish is doing some kind of generative image modeling that is part of a generative adversarial network over evolutionary timescales is plausible, right? And I'm interested if somebody thinks it's not so plausible.
But to actually characterize this i/o transfer function, under the hypothesis that there exist some fixed points, and I've only showed you some kind of anecdotal evidence. We don't have anything quantitative yet. But we're going to need some kind of external generator and mixed-image generator and an external discriminator to implement this iteration. Because the cuttlefish does not inspect its own skin, right?
So the predator is actually-- a perception of the predator is what matters. But we are not at the-- well, [AUDIO OUT] could imagine querying the predator by some kind of forced-choice experiment. And maybe that's what we have to do.
But, rather, at the same time, it seems worthwhile to ask whether we can apply a generative image modeling and generative adversarial network and for that purpose. So one well-known generative image model that may be familiar to some of you are those of Soncelli and Portilla.
And there is a little box here, which is a natural quotes image. And then there's-- if you like the simulation of the image created by essentially imposing constraints. And they come up with some number of constraints that-- I believe 740 that humans that are required to reproduce a human's perception. And they do that on the basis of forced-choice experiments.
Not saying I believe that number. But on close inspection at a glance, I must say, I didn't really understand what was going on here at first. Because I did not notice that what we've got-- what you see out here is not actually identical to what's in the box.
So here is another example. And a [AUDIO OUT]. So a strategy that suggests itself would be to kind of cut out-- to look at that cuttlefish camouflaging on one of those backgrounds near that. It seems to be an approximate fixed point. Then cut out that cuttlefish image. And [AUDIO OUT] generate a simulated image alá Portilla and Simoncelli. And then iterate this process.
I don't know if that's the best way. But that's certainly the strategy we're adopting at the moment. So I'll end with these two slides that are repeats of what I've shown you earlier.
We have these wonderful pictures from 15 years ago. And they seem to plead for characterization of the i/o transfer function of this animal viewed as a black box, which I think is appropriate just as Helmholtz. You viewed humans [AUDIO OUT] way.
So this is a strategy for which we end a hypothesis really altogether. And it's one that I don't think we can really direct [AUDIO OUT] skinny in human or any other animal at least from the forevision. And then a reminder about why the octopus is suitable. It needs to hide from predators and wants to be perceived by the world of predators and the ocean as a signal.
They're always looking for these guys. But it wants to be perceived as noise. It wants the predator to just keep on going and not notice that dinner is at hand.
And the incidental gift to us, which I [AUDIO OUT] is ripe to be properly exploited is that it's also reporting to us as a byproduct. And we have some, we could imagine, quantitatively characterizing to try to infer what that computation [AUDIO OUT]. And along the lines of an i/o transfer function.
So thank you to Andy for organizing and to the audience-- everyone here for your patience. And-- been a little-- some personnel turnover. So this is not entirely faithful.
So I got interested in this animal a couple of years ago. Before that, I was working on theoretical problems in genomics. So at OIST, we are well funded. And we don't have to apply for the funding.
And it is not that practical for someone to-- doesn't know anything about anything to actually open a project of this kind which requires really intensive wet lab support. [AUDIO OUT] a bunch of animals growing in the lab with some reliability, I think we're going to need thousands of animals and a throughput of animals that we can control in order to obtain the necessary statistics which are, I think, are quite daunting.
But the last thing is that. So now I'm-- we're turning over to-- so now the wet lab's been established. And I'm turning a number of those staff that I can over to computational lists and theorists to [AUDIO OUT] try to actually do the calculations that I'm suggesting here. So I am definitely interested in anything anyone here has to say. So thank you very much.
[APPLAUSE]
have an interactive transcript feature enabled, which appears below the video when playing. Viewers can search for keywords in the video or click on any word in the transcript to jump to that point in the video. When searching, a dark bar with white vertical lines appears below the video frame. Each white line is an occurrence of the searched term and can be clicked on to jump to that spot in the video.
