Meta Demonstrates Quest 2 Body Tracking Without Trackers

Meta researchers demonstrated Quest 2 body tracking without additional trackers.

Current VR systems come with a headset and handheld controllers, so only track the position of your head and hands. The location of your elbows, torso, and legs can be estimated using a class of algorithms called inverse kinematics (IK), but this is only sometimes accurate for elbows and rarely accurate for legs. There are just too many potential solutions for any given set of head and hand positions.

Given the limitations of IK, there are some VR apps today only show your hands, and many only give you an upper body. PC headsets that use SteamVR tracking support wearable additional trackers such as HTC’s Vive Tracker, but the three needed for body tracking cost north of $350, so this isn’t supported in most games.

But in a new paper titled QuestSim, Meta researchers demonstrated a system powered by a neural network that can estimate a plausible body pose using only tracking data from the Quest 2 and its controllers. No additional trackers or external sensors are needed.

The resulting avatar motion matches the user’s real motion quite closely. The researchers even claim that the resulting accuracy and jitter is superior to run-of-the-mill IMU trackers – devices with only an accelerometer and gyroscope, such as the Pico 4’s advertised Pico Fitness Band (Pico claims it works on its own machine learning algorithm, though).

However, there is a catch here. As seen in the video, this system is designed to prioritize producing a plausible the whole body pose over and make sure the avatar hands match the exact position of the user’s hands. The system’s latency is also 160ms – more than 11 frames at 72Hz. Given these issues, QuestSim would only be suitable for viewing other people’s avatar bodies, not for viewing your own body when looking down.

Meta’s current legless avatars

But still, it would be far preferable to see the full body movement of other people’s avatars rather than the oft-criticized legless torsos of Meta’s current avatars. So is this system, or something similar, coming to Quest 2?

Meta CTO Andrew Bosworth certainly seemed to hint at this last week. When asked about leg tracking on an Instagram “ask me anything” Bosworth replied:

“Yes, we’ve been made fun of for the legless avatars and I think that’s very fair and I think it’s quite funny.

Having legs on your own avatar that don’t match your real legs is very disturbing to people. But of course we can set foot on other people, as you can see, and it doesn’t bother you at all.

So we work on legs that look natural to someone who is a spectator – because they don’t know how your real legs are actually positioned – but probably when you look at your own legs, you still won’t see anything. That is our current strategy.”

The short-term solution may not have the same quality as this algorithm. Machine learning research papers tend to run on powerful PC GPUs at relatively low frame rates. The paper does not mention the performance of the described system, but an RTX 3080 is listed as used for training.

The annual Meta Connect AR/VR event is in just over 2 weeks, so any body tracking announcement will likely take place during that.


Leave a Reply

Your email address will not be published.