A mixed reality system is a system where both augmented reality and virtual reality are combined to work together. According to B. J. Susanna Nilsson, there are three criteria to be satisfied to classify a system as a Mixed Reality system. There must be a combination of real and virtual elements, that are interactive in real-time and they are in three dimensional. Virtual continuum can be used to describe the relationship between augmented reality and virtual reality.
Figure : The Virtual Continuum
The field of Mixed Reality has existed for almost two decades, where a diversity of applications has been developed. Examples of them are in medicine, military applications, entertainment and infotainment, technical support and industry applications, distance operation and geographic applications[1].
Application of Mixed Reality
There were researches made on this new area of studies in recent years and several research article were publish on this topic, the Mixed Reality. One of the research article published was MagicBook[2] by Mark Billinghurstm, Hirokazu Kato and Ivan Poupyrev. The aim of MagicBook is to enable users to be able to interact with the book either as a physical object, augmented or virtual object. The MagicBook requires one or more handheld display (HHD) and also the physical book to work. In the paper, a Sony Galsstron display, an InterSense InterTrax intertial tracker and a small camera coupled with a switch and pressure pad makes up the HHD[2]. The input from the camera is fed into the computer to calculate the position and orientation relative to the picture and virtual images is generated to be displayed through the HHD.
Figure : The MagicBook Handheld Display
One of the features is that if a user is viewing the book in the immersive VR mode, the other user that is using the AR mode can see a miniature avatar of the immersive virtual reality user in the scene. Also, the MagicBook is able to provide VR user an experience of having multiple users simultaneously in the virtual environment.
Figure : Avatar in the augmented virtual world
Another research article by Raphaël Grasset, Andreas Dünser and Mark Billinghurst from HIT Lab NZ aims to bring Mixed Reality into education. This article focuses on bringing edutainment for children through an already published book, creating a new type of 'mixed reality book'[3]. The authors aim to investigate how to design better symbiosis between new technology and a traditional medium and therefore a less disruptive reading experience[3]. Comparing this to the MagicBook, The Mixed Reality Book has more some features that are not found in the MagicBook. Such features are the sound effects, where there are background sounds, animation associated sounds, or 3D sound. The Mixed Reality Book also has immersive effects to make the surroundings of the pages filled with virtual scenery and cinematic effect to guide user to focus on specific part of the pages.
Figure : Different types of tangible interfaces. From left, gaze interaction, finger interaction and tangible interaction
There are three types of interaction are implemented in this project, gaze interaction, finger interaction and tangible interaction. Gaze interaction is responsible for positioning of an element by direct interaction; finger interaction is for pointing or moving virtual objects within a specific path; tangible interaction is responsible for control, where an element's placement or movement is used to control an interface value.
Other than using Mixed Reality on books application, Mulloni, Seichter and Schmalstieg[4] designed an indoor navigation on mobile devices using Mixed Reality. Initially they started off with using AR World-in-Miniature (WIM) views at info points with the combination of turn-by-turn navigation. This kind of assist was well appreciated by users and they feel that such support missing elsewhere[4]. After the first design, Mulloni and others improved their design, with providing Virtual Reality to show way point for the indoor navigation. This design was decided based on their previous findings, where the role of WIM is significant. The new design will transition between VR and also AR depending on situations. In this project, low-key localization infrastructure is used by only having little info points in the building. This will allow users to easily reroute to the nearest info point when path deviation occurs. The MR view shows the instructions using the Mixed Reality WIM views that consist of landmarks of the buildings and also the info points. When info points are not available, VR view will be used to show the current instruction, highlighting the upcoming path to navigate. A short animation will move the user's avatar on the path mentioned. There will also be text information to help users to understand the instructions.
Figure : World-in-Miniature Virtual Reality view of the indoor navigation application
As user approach an info point, they can target it using the phone's camera, the application will then changes to AR view that will provide an overall view of the path, pin pointing the user's current location and also showing the next path to be taken and previous path taken. In AR view, all the info points in the building are shown as well. The application was tested on an iPhone 4, with the combination of gyroscope, magnetometer and accelerometer data. Kalman filters are used to estimate the orientation of the device and GLSL ES shaders for rendering.
Figure : Augmented Reality of the indoor navigation application
Another interesting application of Mixed Reality is the TimeWarp by Herbst, Braun, McCall and Broll[5]. TimeWarp is an interactive Time Travel with a mobile city-based Mixed Reality game. The background of this research is about small elves secretly helping the citizens of Heinzelmännchen of Cologne during the night. Rumor says that they had disappeared to an unknown location because they were trapped by a nosy tailor's wife. Herbst and his colleagues then extended the legend by mentioning that the elves have fell into time hole instead. This gave the goal of the game for each player, which is to find the Heinzelmännchen within the period by time travelling and bringing the elves back to the present. Each player is equipped with a "magic technical" system which enables them to see the elves. To rescue the elves, player will need to solve challenges presented by the elves. TimeWarp is a distributed system with a game server retaining the game resources and thick client for the player system. Each player is equipped with a mobile AR system that is responsible for augmenting the real environment and a handheld-based information device. The information device is a Dell Axim x51v running on Windows Mobile 2005. Bluetooth connection is used to allow communication between the AR system and information device. The AR system is either a Notebook with onboard graphics running on Windows XP Professional or an UMPC with the same configurations. Player will wear a monocular SVGA head-worn optical see-through display which is tracked by a GPS receiver for position tracking. Interaction can be done using either a standard Bluetooth mouse or a gyroscopic mouse. The TimeWarp AR application is built using AR/VR framework MORGON and uses AR/VR viewer Marvin for 3D visualization[5].
Figure : Dell Axim x51v information device. Overall information page (left) and interaction map (right)
