How is a limited finger tracking controller like Index Controller supposed to work?

Reading the specifications and looking at the extensions provided by my SteamVR runtime I do not understand how an Index Controller is supposed to be accessed to obtain finger interaction. I see the XR_EXT_hand_tracking extension and the improvement thereof XR_EXT_hand_joints_motion_range but no matter what I try xrLocateHandJointsEXT always returns false for isActive member.

In OpenVR you could call GetSkeletalActionData() to access a separate data structure containing finger curl and splay (flFingerCurl and flFingerSplay). It does not seem to be possible to do something similar in OpenXR?

I’ve seen in the XR_EXT_hand_joints_motion_range extensions a small text stating this:

XR_HAND_JOINTS_MOTION_RANGE_UNOBSTRUCTED_EXT: This option refers to the range of motion of a human hand, without any obstructions. Input systems that obstruct the movement of the user’s hand (e.g.: a held controller preventing the user from making a fist) or have only limited ability to track finger positions must use the information available to them to emulate an unobstructed range of motion.

“or have only limited ability to track finger positions” … this would fit the Index Controller case. But SteamVR does not provide the data. This is actually the same under OpenVR. Finger tracking data is not provided but curl/splay data is provided.

Are there any plans to support this or am I missing something here?

XR_EXT_hand_tracking works fine for me with the Index controllers, using the Unreal Engine 4.27 OpenXR plugin with the hand tracking extension. The runtime feeds it with the full hand poses. I’ve even modified it to use the XR_EXT_hand_joints_motion_range extension, and that works perfectly fine as well.

I have full finger tracking data available using the OpenVR plugin as well, so there might be some issue with your configuration, or a bug in SteamVR if that is not available.

I’ve got SteamVR 1.21.8 on Linux. What SteamVR version you have had success with these extensions?

Just confirmed that it’s still working with 1.21.8 under Windows.

That’s interesting. I’m working on Linux here since on Windows OpenVR with SteamVR crashes on startup (reported the bug but there seems to be no interest in it). Maybe OpenXR on windows is more stable.

I tested now on Windows. The result is the same:

  • OpenXR reports hand tracking to be supported
  • xrLocateHandJointsEXT returns 0 (success)
  • XrHandJointLocationsEXT.isActive is false

The code you can see here. Do you see a difference to what you are doing?

That all looks correctly set up to me. If using OpenVR is resulting in crashes, it really sounds like there may be an issue with SteamVR. Have you checked if hand tracking works in other applications that uses it?

On Linux hand tracking works in OpenVR using curl/splay fields (bone joints are not supported). On OpenXR I get isActive=false and SteamVR hangs in a pthread_cond deadlock on exit (this has been reported mulitple times already).

On Windows OpenVR crashes on start as reported here. OpenXR starts and stops correctly but also shows isActive=false.

Other applications can get hand tracking but I’m not sure if they are OpenVR or OpenXR based. If OpenVR based there should be no problem from what I’ve seen so far. I don’t have a reliable test-case at hand yet.

I think most OpenVR applications use the full joint data to drive hand models rather than using the curls and splays, so it not working in Linux seems odd. For a quick test, the realistic hands in SteamVR Home seem to be directly driven the full poses without anything extra blended on to them. I haven’t seen any released OpenXR apps that support hand tracking in SteamVR though.

For your crashing issue, I’d suspect something in the previous OpenVR calls thrashing something, rather than the one where it crashes.

I thought so too but there is only the few necessary calls before so I see not much range for error. Also on Linux there is no such crash so I assume my prior calls are not the problem. But yeah, without any support from SteamVR developers this is a huge guessing machine.

SteamVR Home is OpenVR I guess. Since curl/splay works I guess SteamVR Home uses that information if hand joints are not available. But that’s me guessing without knowing.

Yeah, it’s OpenVR. I looked at how the hand models are set up, and the hands seem to just have the bone poses mapped on them directly. They don’t have any animations that could be driven by curl/splays. They are likely using the full poses, unless they have hardcoded in the same complex conversion that their own runtime already is already supposed to be doing.

That’s interesting. I never got any hand joints information from neither OpenVR nor OpenXR. Never so far. I’m not aware of a way to enable it besides enabling the extension on OpenXR side (in OpenVR I think it’s automatic if present).

That definitely seems like a bug. It might be worth setting up a minimal test case application to verify if the the bug is limited to your engine, or if it’s a bug in SteamVR or something in your system/hardware setup.

Just tried my my OpenXR example app (linux only currently) Monado / Demos / openxr-simple-playground · GitLab with SteamVR 1.21.8 and getting joint poses still works fine. The code should be relatively straight forward to read.

Peek 2022-02-14 12-50

I can’t get it compiled. It requests me to specify the OpenXR_Dir but I don’t know what it wants. I tried pointing it at various directories including the openxr headers, libraries, the sdk directory itself, parent directories thereof. Nothing works. What is required for this parameter?

EDIT: Ah… now I get it. I need to point it at the “src/loader” directory inside the SDK. Might help to write this into the Readme.

So we are a step forward. The example above compiles now and it does provide joint information. It does hang on exit as mine does so that’s a steam bug but at last it does get hand tracking. The big question is now why I do not get this information. Why do I get isActive=false and data is all 0?

Figured it out. I somehow managed to use XR_HAND_LEFT_EXT on both hands and had been testing with the right hand controller. Now I do get the hand joints data.

There’s one question though. The specification of XR_EXT_hand_tracking states you can check if the runtime itself supports hand tracking but I do not think you can check if a controller supports hand tracking? I would prefer to not add a hand tracker if an attached controller does not support it. I could check the isActive state but that’s most probably only scoring a true value after a few frames until the system accumulated enough data to do reasonable hand tracking.

This seems like a pretty big shortcoming in the current hand tracking spec. SteamVR seems to support emulating hand tracking if the device itself doesn’t provide any data, so it can’t use returning false on supportsHandTracking or isActive to indicate lack of hardware support.

OpenVR provides this information with an enumeration. Something similar would be useful for OpenXR.
https://github.com/ValveSoftware/openvr/wiki/SteamVR-Skeletal-Input#EVRSkeletalTrackingLevel

Is there a reason why the hand tracking provided by SteamVR via OpenXR is not sufficient and you would want to not use it in some cases?