I’m working on a research project where we use Varjo XR headsets and we process the video pass-through from the headset cameras in various ways before presenting it to the user in real-time. For now, we are using the Video Post Processing API from Varjo SDK. However, besides being not portable, it has its own limitations. This is why we wanted to implement a custom engine that uses OpenXR.
I sort of assumed that OpenXR would allow for getting the raw/undistorted image from the camera directly “out-of-the-box”, but now it seems to me that this is not the case. Note that we don’t want to only use the video-passthrough as background like in the case of using blend or alpha mode. We want to get the image from the camera, transform it in multiple compute passes and present it to the device.
I have seen the thread “How to get images or videos from the headset camera” from 2019. Is there any update on this?
I have also seen the XR_FB_passthrough extension from Facebook, which seems to allow for what I want:
An application to apply styles, such as color mapping and edge rendering, to passthrough.
However, I could not find any general KHR, EXT or Varjo extension for this. Have I missed anything or am I out of luck for now until Varjo implements such an extension? Do you have any idea for a workaround if it is so? My project really depends on this feature.