New Technology Uses Mouth Gestures to Interact in Virtual Reality
From: R&D Magazine - 10/06/2017

Researchers at Binghamton University, State University of New York have
developed a new technology that allows users to interact in a virtual reality
environment using only mouth gestures.

The proliferation of affordable virtual reality head-mounted displays
provides users with realistic immersive visual experiences. However,
head-mounted displays occlude the upper half of a user's face and prevent
facial action recognition from the entire face. To combat this issue,
Binghamton University Professor of Computer Science Lijun Yin and his team
created a new framework that interprets mouth gestures as a medium for
interaction within virtual reality in real-time.

Yin's team tested the application on a group of graduate students. Once a
user put on a head-mounted display, they were presented with a simplistic
game; the objective of the game was to guide the player's avatar around a
forest and eat as many cakes as possible. Players had to select their
movement direction using head rotation, move using mouth gestures and could
only eat the cake by smiling. The system was able to describe and classify
the user's mouth movements, and it achieved high correct recognition rates.
The system has also been demonstrated and validated through a real-time
virtual reality application.

Read the entire article at:
https://www.rdmag.com/news/2017/10/new-technology-uses-mouth-gestures-interact-virtual-reality
https://www.binghamton.edu/news/story/799/new-technology-uses-mouth-gestures-to-interact-in-virtual-reality
https://www.ecnmag.com/news/2017/10/new-technology-uses-mouth-gestures-interact-virtual-reality
https://www.eurekalert.org/pub_releases/2017-10/bu-ntu100517.php

Links:
Lijun Yin
http://www.cs.binghamton.edu/~lijun

Partially occluded facial action recognition and interaction in virtual
  reality applications
http://ieeexplore.ieee.org/document/8019545
https://www.computer.org/csdl/proceedings/icme/2017/6067/00/08019545.pdf

---

Binghamton University researchers have developed a technology that enables
users to interact in a virtual reality (VR) environment using only mouth
gestures. The researchers say they created a new framework that interprets
mouth gestures as a medium for interaction within VR environments in real
time. "We hope to make this applicable to more than one person, maybe two,"
says Binghamton professor Lijun Yin. "Think Skype interviews and
communication." During testing, the system was able to describe and classify
the user's mouth movements, and it achieved high correct recognition rates.
In addition, the researchers' system has been demonstrated and validated via
a real-time VR application. Although the technology is still in the prototype
phase, the researchers say it could be applied to a range of fields. "Medical
professionals or even military personnel can go through training exercises
that may not be possible to experience in real life," Yin says.