Part Five

From Box Files to the British Science Festival: Part Five

Where were we?

Mark 3 had fallen and Mark 4 was about to. Mark 5 was at least robust, if a little heavy, and looking a bit like an old workshop cabinet with the back kicked off (which is exactly what it was). It was around this time that it became apparent that I had been making the wrong thing all this time.
If you remember, the original idea had been to take a live video stream of the hand, seen in the same place as the real hand, and then add a leftwards or rightwards shift to the seen video image to find out how quickly the brain could compensate for the discrepancy. Now that we had the hand in the right place at the right time, we suddenly realised that we could have much more fun than that.

First came hand switching. This involved simply switching the seen hands so that the left was where the right should be (and vice versa) or by mirroring the right hand onto the left (or vice versa). Because both hands are very similar the brain automatically assumes that the one on the left is the left hand and the same for the right, except that they aren’t. You can see a video of this on our illusions page (https://miragelab.co.uk/current-research). Seeing the wrong hand move (or not move when you expect it to) is very surprising and a little disconcerting.

Great fun for us, but was the point? What could we use it for?
Catherine happened to know a brain-damaged patient (called Mohan) who sometimes had the feeling that he had moved his left hand, even though it was completely paralysed and he knew it to be. He had this feeling whenever someone moved something for him at the same time as him thinking about doing it (for example, turning the page of a newspaper when he had reached the end of the page). We tried to recreate this in the lab be reflecting his right hand onto his left and asking him to pick things up with both hands. When his right hand reached out he would also see his ‘left’ (right hand reflected) hand reach out at the same time. We thought this would give him the feeling that he was actually moving his left hand. It didn’t really work, but it was a good place to start.

Now then, we hadn’t shown the system to anyone up to this point. The whole project had been developed in the strictest secrecy – even keeping it from my own colleagues – but our next discovery meant that we had to show people so that we could try it out on them. One of the first PhD students we showed it to said, “Ooh, this is a bit like the rubber hand illusion, but better.” I hadn’t heard of that so I looked it up. I thought that it was very much not like the rubber hand illusion at all, but I did think that it was better.

Soon after that came one of our biggest breakthroughs: finger stretching; and, as usual, it was all a bit accidental. While Catherine and I were setting the system up one day we did something wrong. I can’t remember now what that was, but it had the effect of making the hand look much too long. It was weird to see your own hand like that, but looking too long, we realised, was not the same as feeling too long so I went away with the idea that if I could program the image to stretch at the same time as someone pulled on the hand then maybe it would really feel like your hand was getting bigger. We thought that if the senses (vision. touch and position sense) worked together then that might be enough to create the illusion of stretching. It was and finger stretching was born.

More of finger stretching later, as well as the disastrous story of  Manchester, the invention of disappearing hands and the meeting that changed everything.