Keep view upright balance with app's reorientation

OpenXR
1

I have a game where the right-stick can look up and down vertically. I guess this may be why VR games disable that, although I find it is a better experience. I notice when manually controlling the view this way, it needs something extra to simulate how our brain keeps what we’re looking at perpendicular to the ground.

EDITED: I might add, it’s already correct if facing forward, but not if glancing to the side.

Since this is very sensitive, I’m wondering if there’s a good way to feed the “space” to OpenXR and get back something that’s correct, or barring that, reliable math that won’t make users sick. (On a screen I might fudge it, but in VR it seems like it needs be both correct and physiologically accurate, so maybe we should have resources to help us out. The OpenXR specification could provide those resources and guidance.)

EDITED: I think this (Motion Canceling for 6DOF Motion Simulator - #5 by mbucchia1) may technically be the same problem. But I don’t want to jump into it.

2

Update: This is a solution, but it’s not the solution to the problem I thought. I explain below (thanks to @Rectus) that the problem is more like the direction of the roll axes needs to be consistent and so it’s best to let XrLocateViews calculate it 1 time instead of try to add on top of it. (It doesn’t directly “simulate inner ear” but it’s probably the easiest way to avoid having bad “inner ear” results, if not the only way.)

I posted this late last night, and then realized, maybe this is what XrSpace is for obviously, and so I hit the Delete button. It took me a night of sleep to work my head around OpenXr’s highly funky design, and I thought I should post a solution since this is a problem that likely comes up in games that move the orientation out from under the main character, etc.

Also, I figure I might get some feedback in case this isn’t enough.

To work with XrSpace is very counterintuitive. I also have a recentering system to work with. To get space data you have to pass an XrTime, but I can’t exactly see how any of the spaces move except VIEW.

So, to query the recentered space, I have a constant XrSpace representing “identity”. I could probably store this transform, but it’s not convenient. So first I use xrLocateSpace against identity to get the centered space’s coordinates.

Then I reorient its quaternion part with a given value, and use xrCreateReferenceSpace to create a third reoriented space, and use that instead.

Now for some reason I don’t understand, my code has to invert the given quaternion, even though as far as I can see it’s working in OpenXR’s coordinate system (my app has to do a lot of convention conversion around this.) And also, yaw (rotation around vertical axis) for some reason causes a funny effect that is uncomfortable. It seems like rotating the space horizontally just isn’t accounted for, or it’s interacting badly with recentering, I don’t know… but yaw isn’t needed to restore balance anyway. So this wrapper reorientation API removes the yaw and returns it to the caller for their own calculations.

It took me a lot of trial-and-error to wrap my head around what was needed. But after making this change it no longer strikes me as feeling incorrect when I look up with the game controller and to the side with the set (the world doesn’t seem to shift on its axis) so I think it’s working.

3

I’m not sure I fully understand the issue, sorry if this is the wrong thing. I think a good idea on how to structure things is to separate the VR tracking space calculations from the game movement logic as much as possible. Only expose the poses of the tracking space and the poses of each tracked object relative to the tracking space (this is what most engines seem to do).

Have the player object set up with its pose as the origin of the tracking space and the HMD as a local transform of that, so that the inputs are essentially controlling the tracking space instead of the HMD pose directly.

1 Like
4

Update: My brain is screwed on backward at this point about the “inner ear” stuff. There’s something to it, but it’s not what I was thinking when I wrote this. Jump to the bold parts below to get ahead of my misunderstandings. (There’s still some value in this post, and there’s still a real problem to be solved.)

Essentially, the local_space needs this extra pitch/roll information to calculate the “inner ear” requirement. I could try to compute that myself, but it seems risky, and I don’t have authoritative math. Maybe OpenXR needs a special input to make this more explicit.

Now, the truth is, it’s probably as simple as taking the in-game point-of-view and canceling roll, and applying to it the roll of the set… but the roll is already baked into the set’s views, so it seems it’s better to leave that calculation to the runtime.

Does this make sense?

EDITED: I will add for clarity, in my app’s case, after this, the game transform is just yaw and global translation. Plus the view’s translation needs to be transformed into the game’s yaw’s space before it’s added to the global translation.

5

If I understand it right, it sounds like you have the stick controls tied to the view transform. I think pretty much every VR game decouples them.

Think of your tracking space as sitting in an imaginary spaceship cockpit. When you press up on the stick, it should pitch up the world outside it based on the spaceships direction, not the way the player is looking. Inside the cockpit, the pitch and roll should never change. If you have a manual recentering function, it should only add an additional transform with yaw and translation.

6

The head set is added onto the game’s regular non-VR controls. It’s a VR option in a regular game. I wrote a wrapper around OpenXR to reduce it to a simple system, so it’s better to manage centering on the wrapper’s side than force apps to deal with it. It could be done without XrSpace but I assume that XrSpace exists for some purpose. It would be easier to reset the real calibration center, but OpenXR stubbornly won’t do that, even though for practical purposes it needs to be accurate to within a millimeter of the game character, and you’re never going to sit down in just the right spot to be that accurate.

RE “based on the spaceships direction” yes, that’s the same scenario as looking up with a game stick. (Edited: Assuming your brain has some sense of where the ground is in a spaceship, in case it’s zero-gravity, in which case, I don’t know how the brain keeps what it’s seeing upright.) I don’t like that I can’t do both because my brain wants to do whatever it wants to do, and if it short-circuits your brain’s natural responses, that’s going to be a bad experience… plus it’s too work intensive to physically look up and down, and to be forced to do so. RE “pitch up world outside” that is still done with the “model-view” matrix because you’re not going to iterate over all your “assets” and pitch them (their world coordinates) individually. So it’s all happening in the view, and you need to make this “inner ear” adjustment whether you’re in a spaceship/vehicle, or being tossed around (your character) or are looking up and down with a game controller, so it still seems best to leave that to the runtime’s calculation. Most app developers aren’t qualified physicians and so that part of VR should be put inside the open library like OpenGL/OpenXR so industry specialists can mull over it.

EDITED: I realized I just now published that “wrapper” for the first time, the header is at https://sourceforge.net/p/widgets-95/code/15/tree//trunk/include/widgets95.h and the relevant code starts at class xr to get an idea. (The implementation is in a file called xcv_openxr.cpp and xcv_d3d11.cpp.)

8

I mean that you should still be able to use the stick to look up and down, but not inside the tracking space “space ship”. If I’m visualizing it correctly, you would need an extra transform between the tracking space and the game space that applies the inverse rotation of where the regular player camera is looking. The rotation would also have to be done around the natural rotation origin of the players head to reduce motion sickness.

Not sure if anyone has devised any “inner ear” adjustments, of if it is even possible to do (unless you mean adjusting the rotation origin). The consensus for VR design seems to be to never do forced movement directly on the pitch or roll, apart from when the player has a fixed frame of reference like a cockpit, so I doubt runtime vendors would want to implement any adjustments even if it’s possible.

1 Like
9

Yeah, I’m not 100% sure, now that I think about it. But I think I know what I mean. See this from the spec:

XR_REFERENCE_SPACE_TYPE_LOCAL. The LOCAL reference space establishes a world-locked origin, gravity-aligned to exclude pitch and roll, with +Y up, +X to the right, and -Z forward. This space locks in both its initial position and orientation, which the runtime may define to be either the initial position at application launch or some other calibrated zero position.

Basically in a monitor based game, it’s wrong to roll the screen, with respect to the ground, because our brains keep what we see upright.

I may be totally stupid and I may be thinking in terms of non VR with VR. Yes probably, in VR the view is rolled but the brain does the unroll.

For some reason you can’t just do this with the views returned by xrLocateViews, or it comes out wrong. That’s all there is to it. The unroll needs to be one transformation.

When I created this thread I may have realized this, but I lost my train of thought because I’m used to regular monitor games more than VR, and this is counterintuitive.

EDITED: For the record I don’t believe in any of the stupid “conventional wisdom” about VR because it’s just made it so there aren’t substantial VR experiences and no one wants to make a real game with VR outside of maybe vehicle sims.

10

The spec just means that the zero pose of the reference space is guaranteed to not have any roll or pitch. My idea is that all the spaces from OpenXR should be regarded as a static room, which you then compose your own transforms to, to align it with the game world.

But yeah, in the end the only way to know if the conventional wisdom holds is to test it out.

1 Like
11

I will say that there is actually over 3 decades of academic research and experience with VR, so while mass-market VR is relatively new, the basic principles, etc. have been developed over a long period of time. OpenXR is designed to create a “pit of success” - make it easier to do the right thing than the wrong thing for VR, performance, transparently supporting other hardware, etc. For example: you have to render to a runtime-provided texture, instead of passing in your own, to avoid a useless texture copy. You have to locate things at a timestamp, and the only timestamps available are from xrWaitFrame (use it for rendering-related locate space/locate views) and from input states (use for physics: know where and what velocity the hand was moving when the “grab” button was released, etc). You can’t get “tell me the time now” without doing a dance, because “tell me the pose now” is essentially always the wrong thing to do. You have to locate things relative to a base space, so that there’s no confusion about the meaning of a transform due to an implicit base that might differ in different “modes” (seated, room scale, free-roaming AR, etc).

This does mean it may be hard to adapt existing non-VR software to it, depending on how it’s structured. What @Rectus said about keeping separate transforms for navigation and not applying forced transforms/animations to the user are both correct, widely used best practices. Navigation in the world is generally done by an app-maintained transform between some base space in OpenXR (local or stage or unbounded) and your game’s concept of the “origin”. I think the scene-graph diagram might have gotten taken out of the spec as non-normative at some point, but having a description of “typical” navigation design might be a good topic for a developer-facing guide.

EDIT: Forgot to add a well-known paper illustrating the basic idea for the locomotion as we suggested: Robinett, W. and Holloway, R.L. 1992. Implementation of flying, scaling and grabbing in virtual worlds. Proceedings of the 1992 symposium on Interactive 3D graphics - SI3D ’92, 189–192.

1 Like
12

In this case I think it does matter that the runtime knows where the game is looking, and I think it would be a mistake most likely to implement it otherwise. Because the alternative would require knowing and reverse engineering what the runtime is doing, which doesn’t seem like a very sound approach/assumption.

I don’t know if Microsoft’s Cliff House can turn around with the game controllers (I never use them) but it can’t turn around any other way if so. I don’t know if that’s because someone thinks “best practice” is to physically spin in your chair because it’s thought VR can’t do otherwise, or if it’s just a way to ensure the software isn’t ever used and is just an encumbrance. I think to do anything in VR will require building up some immunity to its effects, just like with normal games. I think it will be easier when eye tracking comes on line. I don’t think movement is the problem (in my experience) so much as eyes/depth being fixed and low FOV. I think the brain can compensate for movements, and most can learn to, or else we won’t have VR until someone builds a treadmill that can morph into a staircase, since everyone will be stuck playing vehicle sims, and that’s not a killer app. (Plus I’m not big on aerobic VR personally. I just like having a monitor with a larger FOV than a letterbox/knights-helm that responds to my glances. This won’t ever change.)

13

This sounds like it would over-complicate the situation IMO. With the current implementation, the runtime only has to care about where the HMD is in the tracking space, and leave how the game space works up to the application.

I don’t have any WMR headset, so I don’t know how the cliff house does it. Most VR applications support turning around though, either snap turning or continuous yaw like a flat game.

Forced movement in VR games is a huge issue for many people. There has been a lot of research into it, and the issue stems from the visuals not matching up with the acceleration data from the inner ear. High FOV actually has the opposite effect since the user sees more of the world with parts of the eyes that are more sensitive to movement. It’s possible to make the movement more comfortable by lowering the FOV, or displaying a static floor grid.

This is the reason practically all games purpose built for VR are either set up as roomscale games where the primary movement is direct movement in the playspace, or as cockpit games with a fixed frame of reference. Games that do forced movement usually have options for teleporting instead of direct movement to prevent motion sickness.

14

Yep to all.

Movement in cliff house is physical, plus teleporting. You can also rotate with the thumbstick left/right, which is not continuous, it’s like a rotational teleport (jump a set amount of degrees one way or the other) with FOV restricted during the process to reduce motion sickness. Similarly you can “jump back” with thumbstick down, which is a fixed distance jump/teleport, again with artificially limited FOV. “Continuous” movement with a thumbstick can be pretty quickly disorienting, especially (and ironically/counterintuitively) if you try to “smooth” the movement by ramping up/down velocity, etc. (This is true not only in HMDs but also in CAVE-like projection systems, FWIW, though I think it’s less bad. I’ve still gotten myself a little uneasy from it though.)

(Source: I just got done setting up WinMR on a new machine, seated :smiley: )

15

I don’t think anyone really likes those games, but I’m reminded that Sony and others have a solution that vibrates on the inner ear to confuse it that they think will help a lot with this… hopefully without side-effects.

16

I’m reminded, I purchased a VR conversion of Doom 3 for PSVR for a few dollars a while back. What I couldn’t believe is how fast it moved, which made it hard to believe anyone would want to play it. I really couldn’t believe Id would license their game and release it like this without even an option to slow it down to normal human speeds. It still moves just like stupid WASD PC games that have always moved too fast. In VR at least I hope it will become more trendy to make games that move at natural speeds, since that seems essential, and I wonder if games will start taking cues from VR more and more like how in the 2000s they all wanted to be like MMO games. Anyway, I bet no one has played that Doom 3 game, and there aren’t substantial games like Doom 3 for VR, which is what’s wanted I believe, and why VR is looked at as a sideshow at best and space for developers trying to catch the next wave at worse.

EDITED: Also I should add, the Doom 3 game removed the ability to look up/down with the joystick, so I found it quite uncomfortable, and this is why I think these kinds of games should not remove that ability, even though I also think that the thumb-stick design we have isn’t a good ergonomic fit for vertical look controls.

17

Apart from being disorienting, being able to look up and down imposes a lot of limitations the game, such has forcing it to be played seated. Most commercial ports want to leverage the roomscale and motion controller features available.

There are some VR mods that do support looking up and down though. I remember playing an old Quake 1 mod with full mouse and keyboard controls, and it playing surprisingly well, only nauseating me in a few places. If you haven’t already, I’d suggest looking up some older PC VR mods and see what kinds of controls works for you.

18

I will just say “room scale” is more of a limitation than moving freely through a regular game world, that players want. Room games are a gimmick that you have to have a dungeon built into your house to do. Even in a room, you get to its boundary, and “teleporting” won’t help you get back to the center of your room. So really you can’t do anything but be inside a room that doesn’t move, smaller than the Cliff House. Game companies are out of touch with reality in the VR space. Movement isn’t the issue people think it is. One of the better games I played in VR (there’s almost nothing on PS4 despite thousands of games on paper) was a mod of an old PS2 mecha game called Zone of Enders. I didn’t get sick in it even though it moves very fast. (Edited: But probably still much slower than Doom 3!) Technically it’s a vehicle game, but come on, its vehicle is a body… it’s all in the way you look at it, I mean literally, it’s psychological. If your brain looks at the VR game as driving a mecha, then you won’t have the bad feelings, and you can learn to do that with practice and a change of perspective, and game companies need to be brave enough to go forward and not hold purse strings too tight. Some people may not be able to play VR for physical reasons, but some people can’t play monitor games for physical reasons either.

EDITED: RE “play seated” you can stand up, and even move around some by walking… but yeah, seated is generally more practical. Neck strain is a big problem that I don’t think makes active VR safe or feasible, as much as ads want to show it this way.

19

In another topic a user recommended The Dark Mod VR. I have it installed but haven’t looked at it. I don’t like VR controls because they’re cumbersome, but it looks like Sony’s new PSVR (not out yet) design seems the most practical so far, since it’s a PlayStation controller. I think/hope it has magnets that let you connect the controllers so you can free your hand up to do things. Having their bulk strapped to your arms is a big problem and there aren’t enough useful features on PC controllers to be practical for normal games. My game project uses extremely minimal controls (4 shoulder buttons) but can’t be readily mapped to VR controllers, but I just don’t care for the bulk hanging off my arms while I’m blind. It’s too much of a barrier to entry to strap all that on and be completely encumbered/frustrated as a result.

20

By room scale, I just mean games that are designed around allowing the player to move around the playspace freely. There are many different ways to still do forced locomotion, even in room scale games.

Being able to change the pitch seems completely incompatible with standing up though. I’d imagine most people would fall over.

There has been a lot of research into what works and what doesn’t. For example Valve has done a lot of user testing, and they even ported several of their FPS games to VR. They ended up completely abandoning the ports after developing their room scale tracking systems though, presumably because they couldn’t get traditional mouse and keyboard shooters up to their quality and comfort standards.

Found this old 2013 video where they talk about porting Team Fortress 2 to the original 3DoF Rift devkit. It is very out of date with regards to current hardware and practices, but they talk about a lot of the issues with input and motion sickness.
https://www.youtube.com/watch?v=Gpr0FE2ATaY

21

I actually tried this yesterday. Well I just paused while looking up (induced pitch) and then stood up to walk around. It is like being in a haunted house kind of experience, but I really think maybe that would be mitigated if I’d been standing up when I looked up, instead of changing my frame of mind. Of course standing up your body has to think about keeping your legs level so I’m sure it would take more practice probably. Another problem with standing up and walking around is if “forward” in the game means move in the direction of your waist let’s say, so that neck/look movement is independent (which you probably want normally to be able to glance around while navigating) then standing up won’t be as natural as playroom style movement, because you still must turn with a controller and think in those terms, unless there’s a toggle button to switch to a “free” mode.

I think these are things that must be explored with open eyes/arms…

Yesterday I tried the Dark Mod VR project that was recommended to me in another post. I couldn’t get into it because the controls were super stiff and it wanted to move a mouse cursor with the controller that was super flighty, until up to half the screen width, then it would begin turning.

The parable here, is I have a feeling commercial game companies have never properly focused on controlling movement fluidly. My own project (KING'S FIELD 25th PROJECT by swordofmoonlight) (PC VR demo forthcoming) has always been light years ahead of first-person games in this area. I can’t even enjoy first person games (most of them are made for keyboard input even) because they’re so bad in this dept. Because game companies have neglected this area, not caring about natural movement via controllers, I think it’s caught up to them with VR where it’s more important. I don’t think their stumbling block is VR technology IOW, I just don’t think their controller game is remotely good enough to make it a pleasurable experience.