Computer Vision and XR

AdamSobieski · January 2, 2021, 4:59pm

Hello. I recently opened some proposal issues for the WebXR specification which could also be applicable to OpenXR.

Capturing Imagery, Video, 3D Objects and 3D Scenes

opened 04:29PM - 02 Jan 21 UTC

### Introduction I would like to propose for WebXR means for users to capture a…nd to upload captured imagery, video, 3D objects, and 3D scenes. ### Imagery With XR devices, users could photograph content from their real or XR environments. Users could also select 2D/3D regions of interest with their hands or input devices. It is possible for users to take pictures which contain only foregrounded content by utilizing multiple sensors, hand or device tracking, and, potentially, eye tracking. ### Video With XR devices, users could record content from their real or XR environments. Users could also select 2D/3D regions of interest using their hands or input devices. It is possible to record videos which contain only foregrounded content by utilizing multiple sensors, hand or device tracking, and, potentially, eye tracking. ### 3D Objects This scenario involves uses of computer vision algorithms to capture 3D objects from users’ real environments using XR device sensors and, potentially, third-party services. This challenge is known as “3D reconstruction” or “3D object reconstruction”. ### 3D Scenes This scenario involves uses of computer vision algorithms to capture 3D scenes from users’ real environments using sensors and, potentially, third-party services. This challenge is known as “3D reconstruction” or “3D scene reconstruction”. ### Interactivity Multimodal dialogue systems could interact with users to ensure that imagery, video, 3D objects, or 3D scenes are thoroughly, properly, and accurately captured or scanned.

github.com/immersive-web/proposals

Optical Character Recognition

opened 04:31PM - 02 Jan 21 UTC

AdamSobieski

### Introduction I would like to propose that XR-device-based optical character… recognition be considered as an important XR scenario. ### Optical Character Recognition With XR devices and their sensors, users could scan text and mathematics content, from papers, chalkboards, dry-erase boards, and other surfaces. ### Multimodality While using XR devices and their sensors to scan text and mathematics content, users could read the content aloud to enhance the accuracy of scans. This could also be interoperable with eye tracking. ### Interactivity Multimodal dialogue systems could interact with users to ensure that contents are thoroughly, properly, and accurately scanned.

I would like to invite interested participants to discuss these ideas in this forum and/or that forum.

Best regards,
Adam Sobieski

system · July 4, 2021, 4:59pm

This topic was automatically closed 183 days after the last reply. New replies are no longer allowed.