High Speed Gesture UI: Ultra Low latency with Proprioception

Summary

In this research, we propose a high speed gesture user interface as an example of ultra low latency interface enhancing the usability.

In user interface of smart TV, computer, video game, etc., the high speed property of a response has a significant influence on the operation feeling, because proprioception which is the sense of linking one's own gesture to the operation on the display, improves by low latency of the datapath from user input to output to the user.

The figure below shows the outline of our system. In gesture interface, a delay occurred due to A) Capture delay (caused by scanning imager array = the reciprocal of frame rate), B) Input delay (caused by data transfer to an input buffer of a computer or an image processing unit), C) Processing delay (caused by image processing including gesture recognition), D) Output delay (caused by data transfer to an output buffer), E) Display delay (= the reciprocal of frame rate). In the case that each device does not achive synchronization, buffering time which prevents synchronization deviation called frame dropping is needed. In the system based on ordinary video rate, total latency takes about 150 - 200 ms even if gesture recognition is finished within 1 frame. We implemented each step A) - D) based on 1,000 fps, and used a display with 8 ms delay. As a result, this system realized high speed gesture interface which has only about 32 ms total latency (the time from the camera observing LED light to a photodetector observing the change of the display).

Movies

References

1. Himari Tochioka, Haruka Ikeda, Tomohiko Hayakawa, and Masatoshi Ishikawa: Effects of Latency in Visual Feedback on Human Performance of Path-Steering Tasks, The 25th ACM Symposium on Virtual Reality Software and Technology (VRST2019) (Parramatta, NSW, Australia, 2019.11.12-15), Poster [doi:10.1145/3359996.3364726].
2. Himari Tochioka, Tomohiko Hayakawa, Takuya Kadowaki, Haruka Ikeda, and Masatoshi Ishikawa: Effects of Video Latency on Human Performance Against Physical Sensation in Immersive Environments With Low Delay of Visual Feedback, The 24th Annual Conference of the Virtual Reality Society of Japan (VRSJ2019), 4C-5.
3. Kadowaki, T., Maruyama, M., Hayakawa, T., Matsuzawa, N., Iwasaki, K., & Ishikawa, M.: Effects of Low Video Latency under the Immersive Environment with a Gap between Visual Information and Physical Sensation. Transactions of the Virtual Reality Society of Japan, Vol.24, No.1, pp.23-30, 2019.
4. Kadowaki, T., Maruyama, M., Hayakawa, T., Matsuzawa, N., Iwasaki, K., & Ishikawa, M. (2018, November). Effects of Low Video Latency between Visual Information and Physical Sensation in immersive environments. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (p. 84). ACM.
5. Kadowaki, T., Maruyama, M., Hayakawa, T., Matsuzawa, N., Iwasaki, K., & Ishikawa, M.: Effects of Low Video Latency under performance the Immersive Environment with a Gap between Visual Information and Physical Sensation. In Proceedings of 23th the Virtual Reality Society of Japan (VRSJ2018) (Sendai, 2018.9.19), 12B-1.