Sean Anderson

CS 348C (VR Topics)

March 20, 1996

Project Writeup

Abstract

Ideally, augmented reality would merge the real world and a virtual world flawlessly - without lag and occlusion problems. But in its full generality, this task requires an accurate depth map of the real world, since the 3D graphics must be rendered only in places where it lies in front of the real world. The needed depth information could possibly be provided someday by a high speed, sub-millimeater accurate, compact time of flight laser range scanner, but such a device will probably not be available for many years.

For the meantime, I approximated the depth information by using chromakeying, thus yielding two (and potentially three) levels of depth. The idea is to color certain real objects green (or blue or whatever the chromakey) so that when they are viewed through the aparatus described below, parts of a virtual world are displayed in there place. The user wears a head mounted display with a six degree of freedom tracker and small camera attached. The camera's output is composited on top of a view of the virtual world, as it would be seen by a virtual camera positioned where the real camera is. Chromakeying hardware composits the view of the virtual world so that it is only visible in places where the view of the real world contains the chromakey color.

Introduction

Augmented reality has shown great potential in collaborative work, where other users can see themselves and others in addition to a virtual world. Gesturs and expressions aid in communication, particularly in the processes of design and education. The Holodeck on Star Trek clearly shows that there are many benefits to augmented reality in its ideal.

Unfortunately there are many problems that need to be solved before augmented reality works well. One of the greatest is correctly occluding virtual and real objects so that whatever should be closest appears closest. Several other ways of approximating this needed depth information have been developed. Many augmented reality systems use a head mounted display or Boom to put 3D graphics on top of either live video [Bajura92, Wloka95] or whatever may be seen through a Type 2 HMD. This technique assumes that all objects in the real world are always further away from all objects in the virtual world, which is clearly a poor assumption considering how close our hands and body parts are to our eyes. Another method [Wloka95] uses the two images of the real world and infers a depth map dynamically using a stereo image matching algorithm; unfortunately the resulting depth map appears very crude. In CAVEs [Cruz93], yet another form of augmented VR, the virtual world is projected onto walls of a small room, thereby permitting two levels of depth. Unfortunately, the users' bodies sometimes incorrectly occlude near virtual objects, and only a single user sees an accurate perspective view of the environment. Responsive Workbenches [Krueger95], which display a virtual environment on a table, also suffer from these two shortcommings. Moreover, when one user points at part of a virtual object, the place he is pointing does not appear the same for other users. A final way of approximating depth is to statically precalculate the geometry [Koch93]. Obviously this approach is unacceptable since if VR is to be interactive, then depth maps must be obtained dynamically in real-time.

Chromakeying Approach

The system I built required chromakeying hardware (an SGI Galileo Video board), a graphics workstation (an SGI Indigo2), a head mounted display (virtual iglasses!, by Virtual IO), a small camera (a Sony Camcorder), and some software to read the tracker and render the virtual world (SGI Performer). The camera was mounted on top of a bicycle helmet worn by the user, and its output went into an SGI Galileo Video board to be chromakeyed in hardware. Meanwhile, the position and orientation of the user's head was tracked with a Polhemus 6 DOF tracker (also attached to the helmet) and sent to an Indigo2 graphics workstation. The workstation used Performer to render the view of the virtual environment as it would be seen by the real camera, given the tracker's position and orientation. The rendering on the workstation was input to the Galileo for real time video compositing with the camera's video. Finally, the composited (NTSC) signal was sent to the head mounted display (virtual iglasses! by Virtual IO) to be viewed by the user.

A large piece of blue paper (Studio Blue, available at photography supply stores) was used as a window to the virtual world. In one case it laid flat on a table (simulating a Responsive Workbench), and in another the paper stood upright as a cylindrical display. The head mounted display (HMD) had a 30 degree field of view, so the real and virtual cameras were adjusted to have 30 degree field of views as well. The real camera was adjusted by pointing it directly at an object and setting the zoom for the lense to just include an object 15 degrees off-center. Because of limitations in the workstation's rendering power and the chromakeying hardware being unable to accept input from a Reality Engine and camera simultaneously, the virtual worlds were kept quite simple. The virtual worlds consisted of a gigantic teapot under the table and a Klingon warship projected onto the cylinder so that the user could walk around and view it from different angles.