ElaMorph Projection: Deformation of 3D Shape by Dynamic Projection Mapping

Summary

Projection mapping is a type of augmented reality which present additional information to objects in the real world by projector. As a result, audiences feel an illusion of changes in the material or shape of the target. Recently, projection mapping and its related applications are currently receiving significant attention. In particular, the advancement of high-performance computer, high-speed camera and high-speed projector have realized a technology called dynamic projection mapping (DPM). DPM involves three sequential processes: sensing, rendering, and projection. These processes are repeated rapidly at such a speed that humans cannot perceive a delay in the projection. Thus, DPM enables the display of images attached to moving objects and further expands the range of content that can be presented. As an application of DPM, we propose a method to give an illusion as if 3D rigid object were deformed based on virtual physical properties. For example, as shown in Fig. 1, our method can make the target object appear to have elasticity, even though it is rigid.

1. High-speed Deformation Algorithm

Several methods have been presented which presents illusion as if the target were deformed by projection mapping. In these methods, a projection image is rendered using pre-made animations. Our research differs from these methods because object deformation in our approach is accomplished via physical animation based on the sequence of spatial movements of the projection targets. Additionally, as the quality of projection diminishes when a part of the projected image extends beyond the projection target, we propose a method for deforming the 3D mesh without extending beyond the projection target. Moreover, artists can easily specify the region to be deformed the by using images named ``elasticity map". Figure 2 shows the results of the projection using several elasticity maps.

2. Shading based on human perception characteristics

When humans see a moving object, the brain processes color, shape, and motion information in separate pathways and integrates them in the higher order field to perceive the real world. Applying this property, there is a method to present the illusion of deformation of a 2D plane by projecting only the motion information using a projector. In this study, we extended the shading algorithm for 2D plane to 3D solid using a model of Lambertian reflection. This enables of illusion of deformation for 3D solid. In addition to this extension, we add a term to further emphasize the shading caused by 3D unevenness to improve the quality of the illusion experience.

3. Estimation of environmental lighting

In our system, the projection image is rendered by incorporating information of the intensity and direction of the environmental lighting. We use a visible light camera to estimate the information of environmental lighting in real time. This enables a robust illusion in various situation where the illumination changes with time, such as party halls and concert halls.

Fig. 1 The concept of our method.

Fig. 2 Projection results for various elasticity maps.

Movie

If you want to use the original video, please send an e-mail for copyright permission to contact .

References

1. Kentaro Fukamizu, Leo Miyashita, Masatoshi Ishikawa: ElaMorph Projection: Deformation of 3D Shape by Dynamic Projection Mapping, International Symposium on Mixed and Augmented Reality (ISMAR2020), Recife, Brazil (Virtual conference), 9-13 Nov. (2020)