Vision Scientist Explains Why These Praying Mantises Are Wearing 3D Glasses

[Narrator] Who doesn't love a good old 3D movie?

♪ Let's all go to the lobby ♪

Do you know what I hate

is when producers make me wear silly props

that make me look like an idiot.

[thudding]

But you know who doesn't look stupid in 3D glasses

are these tiny preying mantises.

For the passed six years,

researchers at Newcastle University

have been strapping tiny 3D glasses

onto these little critters to study how their 3D vision

might work and how they might apply

those findings to robots.

I'm Jenny Read, I'm professor of vision science

at the Institute of Neuroscience, Newcastle University, UK.

To learn more we caught up with Jenny Read,

who's leading this research.

Tell us about this system that you are studying?

So I'm studying preying mantises

and in particular how they do 3D or stereoscopic vision.

So you use the different views that your two eyes see

to work out how far away things are.

And that's an ability that was only discovered in humans

in the 19th century, and used to be thought, you know,

really complicated and only higher mammals

like humans and monkeys and maybe cats can do,

but back in the 1980s it was discovered

that actually insects have this ability as well,

or at least one insect, the preying mantis.

So, when I heard about that,

I got really interested and thought,

we need to find out how that works.

How is these tiny insect brains

achieving stereoscopic vision?

Are they doing it in the same way as we do?

Which would be really interesting,

because I think then that would tell us

that there's only one good way of doing stereo vision,

but the other possibility is that maybe insects

have evolved a completely separate way

of doing stereo vision.

In that case, that would also be really fascinating

to find out how they've done it

and maybe get new ideas

for how you can achieve stereoscopic vision in machines.

So this is one of my favorite methodologies

of all time, can you talk about

how you went about studying this system?

How you went about attaching 3D glasses to mantises?

So that was the first puzzle that we had to solve

when we were beginning this study.

So I mentioned the initial research

in the 1980s on insect 3D vision.

That used prisms,

and it's a really beautiful and clever technique

and because prisms bend the light rays,

that has the effect of basically

stereoscopically moving everything either

closer or further away,

but we couldn't use that for the experiments I wanted to do.

Because in order to answer these questions,

we needed to present much more complicated 3D images to it,

so I needed to find some way

of presenting 3D images just like we do for humans,

so we spent a lot of time trying various techniques,

but in the end, the only one that's actually worked for us

is using colored filters.

So you might be familiar with these red blue 3D glasses

that I don't know, you used to get in cereal packets

when I was a kid and in the 3D movies back in the '50s.

That technologies kind of been superseded for humans,

but for the insects it still works really well.

The only this is we didn't want to use red light

because insects, most of them, including mantids,

don't see red light at all.

It's just not a wavelength that they're sensitive to.

So we shifted everything towards shorter wavelengths

and we used blue and green light.

So the idea is we have a computer monitor,

and we display the left image,

say on the blue channel,

and the right image on the green channel

and then with these colored filters,

we ensure that each eye of the insect

sees only the appropriate channel,

and that's work done by my very talented postdoc,

Vivek Nityananda, and he spent so much time

figuring out how to achieve this

and ironing out all the problems

that we encountered along the way.

And of course, as you say, the big problem

was how do you attach glasses to an insect, right?

They don't have any ears that you could put glasses on with,

and so Vivek came up with a harmless form of glue

made out of beeswax and resin,

so natural ingredients that aren't going to harm the animal,

and with a wax melter

he just melts a little daub of beeswax,

places it on the mantis' forehead,

and then with tiny 3D glasses that he's cut out previously,

he holds them with tweezers and just pops them in place.

The beeswax dries within seconds

and then the glasses are held in place

in front of the mantids eyes.

And you might think they would be bothered by that,

but actually they seem really indifferent to it.

So can you elaborate a bit

on what kinds of images you're showing to these mantises

and then what you are learning from

how they're reacting to such images?

So here I'd like to distinguish between

basically three prongs of our research.

We're pursuing behavioral, electro-physiological,

and computational strands of the research.

So the behavior is trying to figure out, okay,

what can their stereo vision do?

How do we break it?

What kind of images does it work with?

What kind of images does it not work with?

And this is something vision scientists do all the time

and it helps us figure out, basically,

the algorithms that the brain is using

in order to achieve 3D vision

and that's where the computational part comes in.

And the thing is for the behavioral experiments,

we're basically relying on the animals

to tell us what they can see.

So, they're experiencing a 3D illusion in our experiments,

if they reach out and catch,

or try and catch the virtual object.

So the trouble is that limits us

because whatever we display

has to be attractive enough to elicit a strike.

If the animals see something in 3D but isn't interested,

it'll just sit there, and then of course we don't know,

is it seeing anything?

Is it not seeing anything?

So, Vivek's first job was to try and figure out

a really attractive stimulus

that would drive the behavior really well,

and what we've come up with was a black disk

that just spirals around the computer screen,

getting, in ever decreasing circles

and it comes to rest just in front of the mantis,

and that really seems to get them going.

So that was our first stimulus

that we used to basically prove

that the glasses were working.

And having established that,

then we could use more complicated images,

so we've been using images

made up of black and white patterns of dots,

and the reason for this is we want to then manipulate things

like the correlation of the pattern between the two eyes.

Basically, human stereo vision works

by cross-correlating the two eyes' images.

So it's where your brain is kind of

sliding across the images and going

Ah, yep, that is the best match

where the correlation is highest.

And so that was my starting point for mantis stereo vision.

So, with these random dot patterns

we manipulated the correlation

between the left and right eye images.

We know in humans that totally messes with your 3D vision,

but to our great surprise,

this just didn't bother the mantids at all.

They were able to carry on striking

and grabbing the virtual prey

even when there was no correlation at all

between the pattern of light and dark

in the two eyes' images.

So you might say, Well, if there wasn't any correlation,

how could you define where the prey was?

And the answer is it's the relation

where things are changing in the image.

So it's a stereo vision

that's based entirely on motion,

or at least on change, and it actually,

as far as we know, doesn't work at all with static images.

So it's really completely different

in that sense from our own human stereo vision.

What were you doing to kind of look inside the brains

to determine what is going on in the neurobiology here

when they are looking at these images?

My other extremely talented post-doc, Ronny Rosner,

has been looking at that

and that's been a real technical challenge

because even though he's extremely experienced

in insect neurophysiology,

people don't tend to work with preying mantises,

so he had to figure out how to get into their brains,

how to recall the activity from individual neurons,

nerve cells inside their brain,

and he had to do this while

they were watching these 3D displays.

Ronny's found neurons in the mantis brain

which attune to positions in 3D space,

so a neuron that might like objects here,

or a neuron that might like objects over here.

And by like I mean it's firing more spikes of voltage.

And that seems to be telling the animal

where things are in the space around it.

So, putting that all together,

how would a system like that

that's going on in the mantis insect brain

differ from how we humans perceive 3D objects?

So we're still trying to work out the details of that

but the kind of preliminary model

I have in my mind at the moment,

which we're trying to write computer programs

to instantiate and test,

is that actually the take home for me

is it's much more similar to humans

at the neural level than I have expected.

We thought originally that mantis stereopsis

might be super simple,

like maybe you have inputs in the left eye

and inputs in the right eye

and they're combined very late

so that you basically have an and gate

just before you decide whether to strike.

And if you see something in the left eye and the right eye,

bingo, you strike.

But actually Ronny's finding many classes of neurons

that attuned to stereoscopic distance at all levels

of the brain and he's found feedback loops.

So that's suggesting quite a complicated circuit,

so I think it's been surprising

how similar the neural basis of 3D vision seems to be

in the insect compared to the mammal.

But, I think the difference is maybe at the front end.

So whereas these computations in humans

are being done on the pattern of light and dark

in the two eyes,

in the mantis they seem to be done on the pattern of change.

So where things are changing.

So I personally believe that this research

can stand perfectly find on its own,

it's great, these are awesome findings,

but there's a bigger picture here, I think,

with applications towards robotics in particular.

Can you elaborate how you might take

what you've learned with this study

and then apply them to machines

that are moving around in the real world?

See, I'm very interested in this

because I tend to think of insects as kind of tiny,

amazingly efficient robots,

and so of course it's hugely interesting

to think how could we take what we learned from insects

and put it into robotics.

I take two main lessons from what we've learned so far

about mantis stereopsis.

I think working with change is a really interesting idea,

and not something that's been used a lot

in the machine stereo-literature so far.

And the fact that it works so successfully for mantids

suggest to me that it might be worth taking seriously

for some robotic applications.

Again, you have to think carefully about the applications,

but maybe if you have a drone

that is moving and wants to come and pick something up,

just like a mantis catching a prey

that is a particular distance,

maybe it would work for that kind of application.

Cool and thank you for taking the time to chat today.

Absolute pleasure, thanks very much.