What is Hand Tracking input?

Hand tracking input is a technology that allows a computer or device to track the movements and gestures of a person’s hands in real-time. This is usually done through the use of cameras or other sensors, which can detect the position, orientation, and movement of the hands.

In Mixed Reality applications, hand-tracking input can be used to allow users to interact directly with virtual objects in a more intuitive and natural way. For example, a user might be able to use hand gestures to pick up virtual objects, manipulate them, and perform other actions.

How Mixed Reality devices achieve Hand Tracking?

There are several ways that MR devices can achieve hand tracking. One common method is through the use of external cameras or sensors that are mounted on the headset or in the surrounding environment. These cameras or sensors can detect the position and orientation of the user’s hands, and this information is then used to track the movement of the hands in real-time.

Hand Tracking on Meta Quest using cameras

Another method is through the use of embedded sensors within the MR headset itself. Some MR headsets have built-in infrared cameras or other sensors that can detect the movement of the user’s hands. This allows for a more seamless and immersive hand-tracking experience, as the sensors are able to track the hands directly, rather than relying on external cameras or sensors. For example, Microsoft’s HoloLens has ToF (Time of Flight) sensors.

What kind of Mixed Reality devices currently support hand tracking?

Several MR devices currently support hand tracking, including the Microsoft HoloLens 2, the Magic Leap One, and the Oculus Quest 2. These devices use a variety of techniques to achieve hand tracking, including the use of external cameras or sensors, embedded sensors within the headset, and hand-held controllers equipped with sensors.

What can you do with Hand Tracking input?

With hand tracking input, you can experience more natural interactions with digital objects. For example, you can directly press a holographic button or grab an object. The video below shows various types of hand tracking input interactions.

Microsoft HoloLens 2’s Hand Tracking input examples