Access to raw/undistored image from headset camera


I’m working on a research project where we use Varjo XR headsets and we process the video pass-through from the headset cameras in various ways before presenting it to the user in real-time. For now, we are using the Video Post Processing API from Varjo SDK. However, besides being not portable, it has its own limitations. This is why we wanted to implement a custom engine that uses OpenXR.

I sort of assumed that OpenXR would allow for getting the raw/undistorted image from the camera directly “out-of-the-box”, but now it seems to me that this is not the case. Note that we don’t want to only use the video-passthrough as background like in the case of using blend or alpha mode. We want to get the image from the camera, transform it in multiple compute passes and present it to the device.

I have seen the thread “How to get images or videos from the headset camera” from 2019. Is there any update on this?

I have also seen the XR_FB_passthrough extension from Facebook, which seems to allow for what I want:

An application to apply styles, such as color mapping and edge rendering, to passthrough.

However, I could not find any general KHR, EXT or Varjo extension for this. Have I missed anything or am I out of luck for now until Varjo implements such an extension? Do you have any idea for a workaround if it is so? My project really depends on this feature.

This is somewhat challenging for privacy reasons, which is why it’s not provided by default. I would suggest looking to see if the FB extension meets your needs, and working with Varjo to implement that or another extension.

I see. If anyone else is looking at how to do it, for now, I plan to use Varjo SDK for accessing the video stream, then process the frames (possibly using OpenCL) and use their OpenXR runtime to present the frames back to the headset. I’ll report back on the results. I think delay might be an issue.

Thanks for the response!