Introduction

Using a simple parallel projection/display for augmented reality often limit the user experience due to limited resolution of both the camera and the display. Also, high latency of the parallel image reduced the sense of reality.

Using a mirror instead of projector/display can improve the situation, but the use of mirror in augmented reality has never been perfectly achieved. Some are just projecting 2D contents and some of them have projected content's position limited or cannot be interacted by hand.

Augmented Mirror is developed to solve the problems and improve the technologies mentioned above. It is able to present a high quality, responsive, interactive augmented reality.

The technique is useful in developing medium/large scale interactive installation that can be used for:

Technique

With the help of real-time 3D motion tracking of the viewer, 3D content is projected on the mirror in the right perspective achieving a augmented reality. In this case, user can interact with the virtual 3D content in a very intuitive way.

With a one-way mirror, which the reflective side is facing viewers, the back side is attached a thin projection surface. The projection is from the back side of the mirror. Since the reflective side of the one-way mirror is a bit transparent, the light from the projection surface will be able to pass through the mirror and seen by the viewers.

To make the 3D content correctly placed in the viewer's perspective, real-time 3D eye tracking is used. The data is then passed to the computer which is responsible for generating 3D content. The body position data is also used to implement interaction between the viewer and the virtual 3D content. The contents are displayed in 3D with the help of 3D glasses, which fully utilized the position data of the two eyes of the viewer, to generated 3D rendering corresponding to the different perspective of the eyes.

The 3D tracking is be done by placing multiple web-cams on the top of the mirror. The image is sent to the computer. A reconstructed point cloud with the images from the cameras and eyes position are collected.

Mind-body Interaction

With the developed technique in the Augmented Mirror research, an interactive installation is made. The title of the installation is "Mind-body Interaction".

It is a large question in Philosophy that whether there are any interaction between the mind and the body of a human. If there is, in what form the interaction is taking place. There is several different ideas toward this topic: substance dualism, property dualism, idealism, materialism etc. The installation is trying to introduce a artistic, new way of mind-body interaction which let viewers to think of this Philosophy question through experience.

While the set-up is nearly the same as what is described above, the added 3D content in the mirror is some abstract lighting and visual effect generated by the viewer's emotional state extracted from brainwave data. The visualized emotional state and the reflected body of the viewer are corresponding to the mind and the body of the viewer. And the viewer is allowed to manipulate or affect the 3D content in the mirror, a mind-body interaction is achieved in a symbolic way.

For getting the emotional state and brainwave of the viewer, a brainwave sensor which is wore on the viewer's head is used.

Software

The software developed is opensource. It works as a toolkit for building Augmented mirror application. Two parts is included:

  1. An OpenFrameworks add-on

    The add-on is able to handle camera calibration, 3D reconstruction and head/eye tracking.

  2. The demo application, Mind-body Interaction, written in haXe

    The "Mind-body interaction" installation demonstrates how can the data from the add-on can be used.

The public repo is located at github.

Acknowledgments

Augmented Mirror is developered by Andy Li. It is started as the Final Year Project of Bachelor of Science in Creative Media, City University of Hong Kong.

Softwares

Serveral softwares are used in the project. Their licenses are as follow:

People that helped

I have learnt from and helped by several people through out the project. They are: