MIDAS Projection: Markerless and Modelless Dynamic Projection Mapping for Material Representation

Summary

Projection mapping has become a powerful technique for entertainment and visual presentation. Projection light can overwrite the visual appearance of an object to create an illusion of changes in the material and shape of the target. In recent years, a dynamic projection-mapping technique that can target not only static objects but also dynamically moving or deforming objects has been proposed, and increasingly flexible and immersive effects are being achieved. However, conventional projection-mapping methods depend on markers on a target or a model of the target shape, which limits the types of targets and the visual quality. In order to deal with various targets, including static, dynamic, rigid, soft, and fluid objects, we developed a high-speed surface normal measurement system using near infrared (NIR) wavelengths and a novel high-speed texturing algorithm in screen space. Our system achieved projector-based material overwriting with a uniform material and a tileable-textured material with millisecond-order latency, and it realized dynamic and flexible material representation for unknown objects.

For projector-based material overwriting in which the system projects artificial shading to change the appearance of the target material, markers or models are not always necessary, because shading with a virtual material, lighting, and viewpoint can be adequately rendered even if we can only measure the surface normals of the target. Then we developed the high-speed surface normal measurement system that performs without visible-light interference with the projection image. (Fig. 1) By using the high-speed shading algorithm using surface normals below, our system achieved 500fps high-speed sensing and projection with unnoticeable latency.

Appearance heavily depends on surface normals of the target object, and so projection according to the surface nomals enables the material to disguise itself. (Fig. 2) In this research, we presented high-speed shading algorithm using surface normals for representation of a uniform material and a tileable-textured mateiral. (Fig. 3, 4) Moreover, we propose an area-based regularization method to avoid the undesirable distortion of a texture due to the noise of target motion and deformation.

We also demonstrated advanced applications using the MIDAS-projection system, including albedo-dependent shading and a graphical Midas touch effect. In albedo-dependent shading, the materials are overwritten according to the albedos (e.g. Silver material onto a shirt, and red material onto a tie as shown in Fig. 5.) In graphical Midas touch effect, all touched objects gradually turn into gold (or different materials) as the ability of King Midas in Greek mythology. (Fig. 6) Such tangible and programmable shading in real life will provide interactive, flexible, and immersive material representation.

Fig. 1 System	Fig. 2 (a) Clay（before）	Fig. 2 (b) Clay（after）

Fig. 3 Projection onto a hand（Markerless）	Fig. 4 Projection onto liquid（Modelless）

Fig. 5 Albedo-dependent shading	Fig. 6 Graphical Midas touch effect

Movie

References

1. Leo Miyashita, Yoshihiro Watanabe, Masatoshi Ishikawa: MIDAS Projection: Markerless and Modelless Dynamic Projection Mapping for Material Representation, ACM Transactions on Graphics, Vol.37, No.6, Article 196 (2018).