Virtual Space - The Movies Of The Future
The use of head-mounted displays with head-tracking and computer control of the image to show movies and video games with an all-round view.
by Bob Yarwood
This essay has been written to publicise the idea of having movies and video games (and eventually television programmes) which are watched with a head-mounted display helmet and computer control of the images presented to it, so that instead of having a picture projected onto a screen, viewers will appear to be inside the scene with the action going on all round them. As far as they can see and hear, they will be in an entirely different place - that is, wherever the camera was which shot the scene. They will be able to turn right round, and look up, and the scene would be all round them and above them. They will not be able to see or hear anything of their real surroundings, although they can still feel the chair they are sitting in.
Before I go into the details of the system I will introduce myself and give some details of my background. I am British, 79, and for most of my career I was a Chartered Engineer working in the oil industry. I specialised in the design, installation and testing of control systems for oil refining and chemical processing, and worked on many contracts in the Middle East and Far East. Other jobs were doing research into computerised medical x-rays, and working on Blue Streak the British space rocket. My academic qualifications are a B.Sc.(Hons) in Physics and an M.Sc. in Control Systems Theory. Both these subjects are general, in the sense that Physics is the basis of all the natural sciences and control systems appear in many branches of engineering - mechanical, electrical, electronic, optical and aerodynamic. This background has given me a basic understanding and "feel" for what can be done now in science and technology, and what will be possible in the near future.
I think this will be a major revolution in the movie industry, similar in many ways to the introduction of sound movies (talkies) in the 1920s. It will also revolutionize video games, because instead of watching characters on a screen, viewers will be able to take part in the game in person, actually going into the dungeon or wherever, choosing where they want to go, and having the animated characters all round them. Some of the technology for this has been developed but not the wole system. To me, it will be the obvious next step in movies and video games - in fact it will be what these have been evolving into since they were invented.
I have provisionally used the term "Virtual Space" for this technology, because I think it best describes the fact that you can be in two spaces at once. Your body can be in real space, but your eyes and ears, and more importantly your mind, can be in another space altogether - a fictional or 'virtual' space. I like to think of the head-mounted display (HMD) as a magic helmet which instantly transports the wearer to another place. Anyone who has seen (and understood) the movies "The Matrix" or "Avatar" will see at once what I mean. I don't think there is any chance that things will ever be like they are in The Matrix, but it does show the basic idea. Some people have suggested the term "virtual reality" for this, but I think that phrase has already been taken to mean many things to do with computer graphics, and "virtual space" would be more specific.
I have not put any pictures in this essay, because it is about a new way of making and viewing pictures. Any pictures I put in would only be the kind we have always had, on a page or screen, which everyone knows already, so there would be no point in doing so. If this makes the essay dull, I can only hope that people will use their imagination to see the new method as it will be one day, hopefully! In any case, if and when the method is perfected, it would not be like viewing pictures at all - it would be like going somewhere in person. Later in the essay I do mention a way in which we can can get an idea of what it would be like.
Leaving the technicalities aside for the moment, I think you can see that this would be an awesome experience. Sitting at home in your living-room, you would put on the helmet and immediately be in another place. We get most of our information about the world through our eyes and ears, so that if your eyes and ears tell you that you are standing on top of Mount Everest, or deep underwater, or floating in space outside the International Space Station, or sitting behind the driver in a Formula 1 motor race, it would be very convincing and you would really feel that you were there (although of course you would know that you really weren't!). As I watched the opening of the London Olympics on television, with all that magnificent display, I was frustrated by the limited view of the television camera and screen. With just one virtual-space (VS) camera set up in the middle of the field, everyone on Earth with access to the right equipment could have been right there with an all-round view exactly as if they were there in person. A year ago, the wedding of Prince William and Catherine Middleton was shown on television. If VS movies and television had been developed in time, we could all have been at the wedding ceremony, with a front-row seat.
As we often watch movies in company with friends or relatives, two or more people could watch a VS movie together. There could be voice communication between the helmets, so that the viewers could talk to each other and discuss the action. Everyone would be watching the same scene, but not necessarily looking in the same direction at the same time. In computer games, two or more people would even be able to play in the same game, with each person seeing the others as animated characters - "avatars" as I believe the term is
The components of the VS system are a helmet with a screen inside and earphones for the sound, some means of detecting the movement of the wearer's head, and a computer which adjusts the scene shown on the screen to correspond with it. This is called "head tracking". The viewer thus gets the impression that he is inside the scene, with the action going on all round him, and gets a very powerful sense of "being there" in person. The view in every direction must be available in the computer so that it can select the right view to correspond with the direction the viewer is facing. For filmed movies this means that the camera must have an all-round view - 360 degrees in the horizontal plane and 180 degrees in the vertical. For animated movies and computer games there would be no need for a camera as the whole scene would be written in the software.
The headset and head tracking for this system has already been developed in the form of the Oculus Rift headset. Details of this can be found in the company website, Wikipedia, etc. As might be expected, it has only been used for computer games up to now. All we need now is the 360-degree camera and we are in business. There has been no mention of this that I have seen but it is inevitable that someone somewhere is working on it.
For filmed movies there will have to be big changes in the methods of filming, at least for dramas. There can only be one camera in each scene, because if there were more than one, they could see each other. Likewise, the director and crew will have to be concealed from sight - probably in another place altogether, watching the scene through their own HMDs. The viewers would be in effect invisible observers standing among or near the characters. Filming documentaries would be easier because there would be no reason why the viewers could not see the camera crew and equipment - it would be as if they had gone along with the crew when the filming was done.
For animated movies, the big difference would be that the viewer could choose where he wanted to go in the scene (within limits), instead of being tied to wherever the camera had been placed. Some kind of joystick would be provided to indicate where he wanted to go.
The first VS movies would probably be documentaries made specially to demonstrate the possibilities of the method, like the early Cinemascope and IMAX movies. At first, even to mount a VS camera on top of a truck and drive through some attractive mountain scenery would produce a movie which would be an exciting novelty. The viewers would feel that they were right there on the truck, and there could be a guide and commentator talking to camera in the normal way. Each viewer would feel that he alone was there and that the commentator was talking to him, but of course the conversation would have to be one-sided!
As for what subjects would be covered in VS movies, they would be exactly the same subjects as we see now in ordinary movies, but they will acquire a new interest as audiences experience the personal involvement, the sense that they are going to these places in person instead of looking at a picture made by someone else. Anywhere on Earth, in fact anywhere in the Solar System where a VS camera could be taken, anyone with the right equipment could be there in person. And they would have none of the difficulties that they would have if they were there in reality. Sitting in your living room, safe and warm, you could be climbing up Everest or a vertical rockface with no danger, wing-walking on an aircraft doing aerobatics, swimming underwater with no breathing difficulties, standing in the middle of a forest fire, or skiing down a slope with an avalanche chasing you, and perhaps being caught by it and being buried in the snow. You could be floating in space outside the space station with no space suit. Using something like the Google Earth software you could fly to any place on Earth, across the actual landscape, with Superman. With animated movies we could walk through ancient buildings, or cities, that have disappeared long ago, or new ones that have yet to be built, provided that we have the drawings or sufficient other information about them. Everyone could be a Doctor Who, travelling through space and time to fantastic worlds.
One example of this could be the Crystal Palace in London, at the Great Exhibition of 1851. The building was destroyed by fire but the original drawings and details of the exhibits are available, so that we could walk through the building in company with crowds of Victorian characters which had been programmed in using the latest virtual reality software. We see reconstructions like this all the time in present-day films and TV, but they are just pictures on a screen and don't give the feeling of actually being there.
A more ambitious project, which I will certainly not live to see, would be to have the viewers go on the voyage of the Titanic. Because of the massive amount of research that has been done on the Titanic, all the details of the ship's construction, furnishings and decorations are known, also everything that happened the night of the sinking. Viewers could have their own cabin, and go anywhere they liked on the ship just as the real passengers did. The other passengers could be animated characters, which are now becoming very lifelike, as was shown in the movie "Beowulf". Viewers could sit in one of the boats and watch the ship sinking, or perhaps even be floating in the water waiting to be rescued but with no danger of dying of hypothermia!
A more realistic use of VS could have been at the Sochi Winter Olympic Games. All over the world, millions of people would have been able to see the Games, not on a little screen but as though they were there in person. There could have been VS cameras set up at various places so that viewers could choose where they wanted to go and which events they wanted to watch, while sitting warm and comfortable at home.
Just recently the film "Gravity" was released. As I watched it I thought what a waste - all that money spent producing a film which we have to watch on a stupid screen, when we could have been up there with them!
And of course, the same would apply to football matches!
As the Space Age dawns, anywhere a space probe, manned or unmanned, can go with a VS camera, we can all go. We could have been travelling around Mars right now on the Mars rovers "Spirit" and "Opportunity". We could have travelled on the Voyager probes right across the Solar System, visiting all the planets and their moons in turn, and now we could be speeding right out of the Solar System into interstellar space. In years to come we can stand on Titan and see Saturn's rings filling the sky. We could even go where real astronauts cannot go, like the surface of Venus or Mercury, or deep into Jupiter's atmosphere - or of course the deepest ocean depths on Earth. It is unfortunately true that the vast majority of the human race will never get into space, but this is a way for us all to do just that. I think NASA would appreciate that way of interesting and involving the public in the exploration of space!
In July 2012 NASA published a composite panoramic view of Mars which it said was "the next best thing to being there". Anyone who has read this far will know that isn't true. The next best thing to "being there" is Virtual Space!
(It has long been a belief of mine that human beings, or even unmanned probes, will never travel beyond the solar system. The distance to even the nearest star is so unimaginably vast that it would not be possible even in theory for any spacecraft, manned or unmanned, to travel to it in less than a thousand years our time, even if the problem of having enough fuel to make the trip could be solved.This assumes that Einstein's Special Theory of Relativity still holds, and it should be remembered that the theory is now 110 years old and has not been improved upon yet. I find it amusing that there are so many books and television documentaries made about the existence of life somewhere in the universe, while carefully avoiding any discussion about the realistic chances of detecting it and communicating with it. I suppose it would not do to say to the public "Yes there is life somewhere in the universe, but no we will never be able to find it and communicate with it, so you can forget about it!" I am sure that the only way in which we can ever travel round the universe will be by VS as I am describing it here.).
Looking further out into space, using what is known of stellar positions and distances, combined with animation, we could be like gods, zipping across the Solar System or the Galaxy faster than light, and even move out of the galactic plane altogether and see the Milky Way as it is - a spiral galaxy. We could then nip over to the next galaxy, Andromeda, (a journey that would take light 2 million years) and be back in time for lunch! It would be a similar idea to Google Earth and Google Mars, but more like "Google Universe". We could all be spectators at the Big Bang and the creation of the Universe. In the television series Stargate we could all go zooming through the wormholes to anywhere in the universe.
In the movie "Avatar" we could have been inside that amazing world, riding on the dragon birds ourselves. We could have been sitting in the great hall in Hogwarts, with the spirits zooming about over our heads, or flying on broomsticks between the towers of the building. In the film "The Hobbit", we could go right along with Bilbo Baggins on his epic quest. And of course we could all be literally "walking with dinosaurs", and even riding on them.
We see these things like this all the time on TV of course, but they are all just pictures on a screen - you don't get any feeling that you are actually there yourself.
There will be other uses for VS techniques. Architects could walk their clients, room by room and floor by floor, through buildings which exist only as drawings in computer memory, so that they can see exactly what they would see if they were in the finished building (including the view from each window). The clients could suggest changes to the design, like moving a window or a doorway, and those changes could appear immediately around them as if by magic. The same would apply to interior design - furniture, decorations and color schemes could appear and be changed instantly, so that the client could see exactly what it would be like to live with different styles. Likewise they could drive round a proposed building complex such such as a civic center or sports stadium.
People who have booked to go on a cruise on one of the modern liners, which are like floating cities, could be sent a disk so that they could "board" the ship a week before it sails and get used to finding their way around. They could go anywhere they liked, as often as they liked, so that when they boarded in reality they would already be familiar with the layout.
When there is a return to the Moon and a permanent base is established, as will certainly happen some time, with VS cameras set up in different places we can all visit the base and look around, watching the station personnel working.
People interested in astronomy could have their own virtual planetarium showing the night sky, sitting in their living room at any time of the day or night, with perfect seeing conditions and no clouds or artificial lights to obscure the view. The constellations could be marked for the benefit of those who are just learning to find their way around the night sky. Instead of the present-day security cameras there could be VS types, set up so that anyone could quickly visit the site, no matter how far away it was, and take a look around. (Such cameras would be easier to maintain because they need have no moving parts). The same kind of cameras could be set up at popular tourist places like Niagara Falls, or inside famous buildings like the Palace of Versailles or the Blue Mosque in Istanbul so that anyone could visit them in an instant and look around.
VS could be used to help people overcome phobias such as fear of heights or flying. They could be, virtually, in a plane or on a high building so that they could get used to being in a situation which looked stressful, while sitting in comfort at home. Incidentally, I think the most scary film for those suffering from claustrophobia must be "The Abyss". I felt uncomfortable just watching it on ordinary TV!
Anyone, including people who are bed-ridden or house-bound, could go a theatre, concert, football match or any other live performance with a seat in the front row, go deep-sea diving, or just go walking round outside. With miniature VS cameras we could appear to be inside somewhere which would normally be too small. We might even be travelling inside the human body, as in the movie "Fantastic Voyage". And we could go inside an ants' nest and see the nest as the ants do. With remote-operated vehicles such as the ROVs used in deep-sea exploration or the military drones used in Afghanistan it would obviously be better if the operator was, virtually speaking, on board the vehicle.
VS might start a completely new industry - the virtual travel industry. At present there are many people who would like to travel to other countries but are prevented from doing so by ill-health or old age. I am getting to that point myself - I have always wanted to go to Rio De Janiero but it looks as though I am never going to make it. A small crew of technicians with a presenter could go to another country just like they do now but with a portable VS camera, and travel round filming. When the film is played back it can be shown to any number of people who will feel that they had been there themselves. In some ways it would be even better than going in person, because in a lot of foreign holidays there is a good deal of fuss and inconvenience in a long, tiring and expensive flight, getting to the hotel, finding how to get around, being pestered by beggars and salesmen, and suffering from the heat, smells and flies. With a virtual holiday you would simply be whisked from one interesting place to the next. Although it would not be exactly the same as going to these places in person, after a year or so your memories of the trip would be the same as if you had gone yourself.
If we were going to Egypt for example, we could wander round the Pyramids and then straight over to the Valley of the Kings, then to the temple at Karnak, and finally to Abu Simbel, with proper guides, all in the space of a few hours, while sitting in comfort at home. Even better, using CGI we could go round these places not as they are now but as they were thousands of years ago, according to the best archaeological knowledge. There would be whole cities with buildings like temples, tombs and roads which have now completely disappeared to the eye but can be detected by archaeology using the new scans from satellites in orbit. And, as with all animated scenes, we could go where we liked instead of having to follow a set route.
The greatest difference that VS movies would make is in dramas. The viewer would be an invisible observer standing in the middle of the action. It would probably take some getting used to, but people would eventually realise that it is more like real life. There are some scenes in a film of "Scrooge" (1951, with Alastair Sim and George Cole) where Scrooge and the Ghost of Christmas Past are looking at scenes from Scrooge's past. They are standing in the middle of the scenes but the other characters can't see or hear them:
There are similar scenes in another version of the film:
A Christmas Carol 1984
In VS movies, the viewer would be in the same position as Scrooge in these two scenes, except that he would not have a companion. (Well actually he could have a companion, but it would have to be an invisible one that he could talk to through the helmet-to-helmet communication that I mentioned earlier).
Horror films could be made so that the viewer is the main character. It would be extremely scary, because the viewer would feel personally involved and that he was in danger, with something creeping up behind him. Imagine crawling through the ventilation ducts in Alien, with that monster perhaps just around the next corner! In films with great panoramas, like "Zulu", imagine standing on that plain, with no-one in sight, but an ominous chanting and rattling of spears coming from just over the hill!
I often imagine and visualise this for myself, having a VS set and being inside these scenes
Now that the head end of the system has been developed, the design of the 360 degree camera will be comparatively straightforward. An a arrangement with multiple cameras in a spherical array would be the obvious way to go, somewhat like the compound eye of an insect. There are interesting Wikipedia articles on the subject, and on holography and phased array optics.
Actually I have found one mention of the idea, in my favourite science fiction novel "Eon" by Greg Bear. The heroine is exploring a city that had been built by incredibly advanced aliens a long time ago but is now deserted. She finds a building which is apparently some kind of library but with no visible records, and learns how to work some of the controls. This is a short passage from the book:
"She called up a guide to the city, and in an instant a scene in the city surrounded her. She appeared to be standing on the portico of an apartment in the lower floors of one of the tower blocks, looking down on the busy streets. The illusion was perfect - she could turn her head and look completely behind her if she wished - indeed, she could even move around, even though she knew she was sitting down. In both her ears - or somewhere in the middle of her head - a voice explained what she was seeing."
It will be a long time before we can beam sound and images straight into someone's brain, but otherwise it is an excellent description of what I have called Virtual Space. Incidentally, what I like about "Eon" is that it is the best example I know of "hard" science fiction. Everything is presented as real science, although centuries advanced from what we have today, and with no element of magic or fantasy. With one woman, a genius-level theoretical physicist, as the central character, it has a breathtaking range in time and space. Although some aspects of it are a bit dated now, I can recommend it to anyone who wants to know what first-class science fiction is all about. Much as I enjoyed the Star Trek series, conceptually "Eon" is in a different league. Details of the story are given in Wikipedia. I would love to see it made into a film, whether VS or just an ordinary film, but I don't suppose it will happen. Well, I digress from my subject, so I will get back to it now!
I want to emphasise that I consider VS movies an entirely natural and obvious development of present technology, and one day we will look back and be amazed that it took so long for the idea to be accepted. When the idea becomes widely known and people realise what they could be enjoying, I am sure that someone somewhere will make the decision to invest in the technology. Whether I will live to see it is another matter!
Whenever I watch a movie on television I feel that I am peering into another world through a silly little window. It is as though someone had invited me to a party, but when I got there I had to stand outside and just look in through the window! I always want to get through the window and be inside that world. If VS movies ever happen, I and millions of other people will be able to do just that.
Last update April 23rd 2015