HandJointLocations all wrong

Hey all,

I just incorporated hand joints tracking into my PC VR app and I’m not getting what I should be getting.

I did it as in example 12.31.5. Example code for locating hand joints
I see joints on screen, hand follows my hand, but with errors:

#1
if I extend all my fingers while looking at the bottom of my palm, all fingers extend on screen

#2
if I extend my fingers while looking at top of my palm, fingers 2-5 are contracted, not extended
and my thumb displays exactly the opposite of how I hold my hand: i.e. if I’m extending my thumb, thumb joints are contracted, if I contract my thumb, displayed thumb extends

#3
as said above, fingers display as extended only if I look at the bottom of my palm (inside of my hand), otherwise they (fingers 2-5) are contracted even though they are visible to the cameras. If I make a fist, displayed joints contract even more, if I extend index finger, it barely extends - it just barely opens up a bit from a fist.

So it all seems to me like the error is not on my side, but in the recognition of hand/fingers/joints, and runtime is getting wrong array of positions.

What baffels me even more is, that in Virtual Desktop Streamer (and in basic see through) I can see my hand and the overlay hand fit perfectly over each other. So Virtual Desktop Streamer is getting proper joint locations, but my app is not getting them (it’s getting the contracted/fist version).

My app is Win64, Im compiling it with SDK version 1.1.43 (same error was on version 1.0.10.2)
headset is Meta Quest 2.

Anyone has any idea what should I do to get proper joints coordinates that represent my actual hand status?

thanks a lot for the help
M

Hi,

You are not giving much details on what platform/runtime you are using, but your mention of Virtual Desktop Streamer made me double-check our handling of the hand joints extension both with VDXR and SteamVR (I assumed this is what you are using?). I see no issue on our side, tested with 2 different applications: the OpenXR Tutorial (Chapter 5) and the Microsoft OpenXR samples (SampleSceneWin32).

See a screenshot from the Tutorial below. I tested various position for the palm/wrist per your explanation, and never saw any issue.

You can maybe take a look at the two samples I mentioned above and see what is different in your code.

If you are still convinced that the issue is with the OpenXR runtime, I propose that you test your code with another OpenXR runtime. SteamLink or Quest Link expose the hand tracking extension (for the latter, you will have to go through the hoops of registering a developer account and enabling developer mode in your Quest PC app).

Hey mbucchia1,

thanks for the input and help. But unfortunately it did not help me.

#1
I revised my code and compared it to your code from Chapter 5, I added a couple of lines but the end result was the same.

#2
I took your Chapter 5 code and compiled it in Visual Studio. I ran it, and I have the same problems as descripbed before. If I look on top of my hands, my finger 2-5 don’t open and thumb is revers (when my hand has thumb left or right, extended, it is displayed as contracted and vice versa).

Here I’m giving you a link to your code that I imported into VS and compiled (debug exe is included).

I really have absolutely no idea what is wrong here and what’s causing this misbehavior in my case.

Thank you a lot.

Apparently I cannot share a link to my google drive with zipped project

https drive google com / file / d /

with

17rtp84Fa3Jff-U2-Y6UqTDkjYgUbf8wZ

https youtube com / shorts / jP7whYD44jk

I just tested your exact scenario and absolutely no problems here.

Meta Quest OS v74
Virtual Desktop 1.33.4
SteamVR Beta 2.9.6

Set OpenXR runtime to SteamVR.
Started SteamVR Home
Started OpenXRTutorialChapter5.exe

All joints working fine: video here

My best guess is you haven’t actually turned on the hand joints forwarding in Virtual Desktop, and as a result of that, the hand joints are being (poorly) emulated from grip/trigger gestures.

You want to open Virtual Desktop (on Quest), then Settings → Streaming → Advanced Options → Forward tracking data to PC.

The reason this isn’t on by default is because it consumes additional network bandwidth and very few games (none?) actually leverage it today. So it is an “opt-in” setting.

1 Like

Thank you, this fixed the problem.
I didn’t know that there is a sepparate settings for hand tracking and that data is being withheld back by default.

Thank you immensly and have a good day.

1 Like