Portable Lumipen: mobile dynamic projection mapping system

Summary

In recent years, Spatial Augmented Reality (SAR) systems for dynamically moving targets have been proposed, and this field is called "dynamic SAR" or "dynamic projection mapping." In these systems, the moving target is recognized by a high-speed camera and the projection onto the target is controlled at high speed, for example, at 1,000fps. By achieving such high responsiveness in sensing and feedback that spatio-temporal consistency between the target and projected image is ensured at the level of human visual perception, these dynamic SAR systems have allowed the development of new application fields in entertainment and user interfaces. However, one problem with most of these conventional SAR systems is that they have restricted spatial movement because they are fixed in a room. Then, we take dynamic SAR a step further. In this research, we use a 3D-stacked vision chip and an optical gaze controller, and realize a portable dynamic SAR system, called "Portable Lumipen." A vision chip is an intelligent imager that has parallel processing elements adjacent to photodetectors, which enables high-speed image capturing and processing on a single chip, and it is a suitable device for a high-speed and small visual feedback system. An optical gaze controller is a unit to change the imaging and projecting direction at high speed by reflecting light at 2-axis motored mirrors, and achieves high-speed alignment of the projection image without the need for a specialized projector or re-rendering of the projection image.

We developed the prototype system of Portable lumipen in Fig.1 and Fig.2, and confirmed that image capturing at 1,000fps, rapid processing and a 3ms response time for projection were achieved on a mobile device. We also propose potential applications using this portable dynamic SAR system. For example, in anywhere surface user interface shown in Fig.3, the system recognizes a gesture and provides a display on an arbitrary object. In Fig.4, a system for taking a selfy with projector-based makeup is shown. The Portable Lumipen can compensate for hand shaking, and realizes animated makeup. Besides, a searchlight that automatically scans the surrounding environment and tracks a detected target while projecting information on it is proposed. Portable dynamic SAR will enhance a variety of applications including entertainment, interface and combination with a drone.

Fig.1 Portable Lumipen	Fig.2 System inside

Fig.3 Anywhere surface UI	Fig.4 Selfy with Makeup by light	Fig.5 Tracking search light

Movie

References

1. Leo Miyashita, Tomohiro Yamazaki, Kenji Uehara, Yoshihiro Watanabe and Masatoshi Ishikawa: Portable Lumipen: Dynamic SAR in Your Hand, International Conference on Multimedia and Expo (ICME2018), San Diego, California, USA, 23-27 Jul.