Lumipen 2: Robust Tracking for Dynamic Projection Mapping

Summary

Projection Mapping - images are projected to match the surface of objects, such as buildings - has recently generated much attention, and can be applied to areas such as performance art and human-computer interfaces. Projection Mapping typically targets static or semi-static objects, and few systems can deal with dynamic objects. For Augmented Reality (AR) to be experienced without discomfort, geometrical consistency between the real world and the virtual information is essential, meaning that images should be projected without misalignment on the target objects. A large delay, caused by the time between measuring the object and projecting images on the object, results in significant misalignment in the case of dynamic objects.

In our laboratory, the Lumipen system has been proposed to solve the time-geometric inconsistency caused by the delay when using dynamic objects. It consists of a projector and a high-speed optical axis controller with high-speed vision and mirrors, called Saccade Mirror (1ms Auto Pan-Tilt technology). Lumipen can provide projected images that are fixed on dynamic objects such as bouncing balls. However, the robustness of the tracking is sensitive to the simultaneous projection on the object, as well as the environmental lighting.

In order to achieve robust Dynamic Projection Mapping, we introduce a retroreflective background to the Lumipen system (Fig.1). As a result, the object will appear darker than the background during projection, which is observed using a high-speed camera. The tracking will therefore be robust to changes of the content of the projection such as movies, and changes of environmental lighting in the object's vicinity associated with its motion. The tracking technique enables Dynamic Projection Mapping with partially well-lit content and time-geometric consistency on e.g. a paper moved by a hand (Fig.2) or a bouncing ball (Fig.3). This opens up for new applications of Projection Mapping, such as visualizing a pitched ball as a fire ball.

Fig.1 System configuration.	Fig.2 Projection of an image of rotating circles to a moving paper.	Fig.3 Projection of an image of the earth to a bouncing ball.

Movie

Related Works

• Janken (rock-paper-scissors) robot with 100% winning rate (human-machine cooperation system)

Reference

1. Tomohiro Sueishi, Hiromasa Oku, and Masatoshi Ishikawa: Lumipen 2: Dynamic Projection Mapping with Mirror-based Robust High-speed Tracking against Illumination Changes, PRESENCE: Teleoperators and Virtual Environments, MIT Press, Vol. 25, Issue 4, pp. 299-321 (2017)
2. Tomohiro Sueishi, Hiromasa Oku, and Masatoshi Ishikawa: Robust High-speed Tracking against Illumination Changes for Dynamic Projection Mapping, IEEE Virtual Reality Conference (VR2015) (Arles, 2015.3.26)/Proceedings pp.97-104
3. Kohei Okumura, Hiromasa Oku and Masatoshi Ishikawa: Acitve Projection AR using High-speed Optical Axis Control and Appearance Estimation Algorithm, 2013 IEEE International Conference on Multimedia and Expo (ICME 2013) (San Jose, 2013.7.18)
4. Kohei Okumura, Hiromasa Oku and Masatoshi Ishikawa: High-Speed Gaze Controller for Millisecond-order Pan/tilt Camera, 2011 IEEE International Conference on Robotics and Automation (ICRA 2011) (Shanghai, 2011.5.12) / Conference Proceedings, pp.6186-6191