A Multimodal Interaction Framework for 3D Games 

This framework has been designed for a single player game that is created in unity 3D using C#. It uses Myo and Microsoft Kinect 2.0 for capturing the gestures and voice commands. The environment is designed as a maze like scene and has 5 different areas which can be explored by the player. The player can navigate in the environment in different directions through walking, flying and jumping gestures. There is a speech recognition module integrated as a part of the game to take voice commands. The game has two stages:
















Stage 1: The user walks in the environment and looks for candies and presents. The player should show the correct number of candies or presents with their fingers to receive points and proceed the the next stage of the game. This phase uses a kinect based skeletal tracking for recognizing walking in different directions, flying or jumping. The hand gesture recognition also uses Kinect and a module has been implemented which can recognize the number of fingers. 

Hand gesture recognition module performs hand segmentation using the joint location of hands and depth based segmentation based on the distance of hand from Kinect. Then, it fits a convex hull to the hand to find the largest connected component. Number of convexity points detected in each hand is the number of fingers. 




Stage 2: After proceeding to the second stage, the user enters an environment where they have to match an object with a reference object in terms of position, color, size and orientation. Finger spread  and fist gestures were used to grab the object and change the color of the object using Myo armband. 


In a user study the convenience of the use of kinect based gestures vs. Myo based gestures was compared. Majority of our users preferred Kinect over Myo as it made the interaction feel more natural.