Gaps when sampling a cubemap

MLLO · November 22, 2024, 3:56am

So I have successfully generated a cubemap containing the rendering of a scene. I am interested in the user to be able to generate a flat rendering of the scene ‘captured’ on the cubemap. The user is allowed to specify the following information:

Where they are looking towards, i.e orientation (yaw) and pitch angles
A horizontal angular range (e.g. 120 degrees)
A vertical angular range (e.g. 60 degrees)

I am using the following fragment shader to generate to sample my cubemap in order to great this flat rendering,

#version 440 core
uniform vec2 display_size;
uniform vec2 offset;
uniform float hrange;
uniform float vrange;
uniform samplerCube cube;
out float color;

void main()
{
    // scale coordinates (0,1)
    vec2 scaleCoord = gl_FragCoord.xy / display_size;
    // convert offset to radians
    vec2 off = vec2(radians(offset.x), radians(offset.y));
    // center the view
    vec2 angles = ((scaleCoord * 2.0) - vec2(1.0)) * vec2(radians(hrange/2), radians(vrange/2)) + off;
    // sampling vector
    vec3 samplingVec = vec3(-cos(angles.y) * cos(angles.x), sin(angles.y), cos(angles.y) * sin(angles.x));
    // output
    color = texture(cube, samplingVec).r;
}

I am creating my sampling using spherical coordinates. My results appear to be okay, except when I choose a largervertical angular range or when the pitch angle is small (or negative). Then I start getting gaps in the bottom (as if I was running out of cubemap to sample), see below. Is this a problem with my sampling? Is it because of the near plane? Is there a way I could avoid this result??

Any comments or ideas to resolve this are welcome!

Thanks,

M.

GClements · November 22, 2024, 12:55pm

Why?

Normally, you’d use the equivalent of:

float u = x / width * 2 - 1;
float v = y / height * 2 - 1;
float s = u * tan(fov_x/2);
float t = v * tan(fov_y/2);
vec3 tex_coord = vec3(s,t,1);

Except in practice, it may be easier to start with NDC coordinates of <±1,±1,k> assigned to the corners of the viewport and <0,0,0> to the viewpoint then transform them by the inverse of the projection*view matrix to get view coordinates. This allows you to use the cube map in conjunction with other objects (i.e. as a skybox).

Or just render a cube with texture coordinates of <±1,±1,±1> at the corners. This is probably easier to understand if you have problems with the inverse-projection maths.

MLLO · November 22, 2024, 5:54pm

Thanks @GClements

Why?

I wanted to be able to keep the angular sampling regular when sampling across FOV. It seemed to me the most obvious way to do it.

Normally, you’d use the equivalent of:

float u = x / width * 2 - 1;
float v = y / height * 2 - 1;
float s = u * tan(fov_x/2);
float t = v * tan(fov_y/2);
vec3 tex_coord = vec3(s,t,1);

I take it that I would use tex_coord as my sampling vector, correct?

Except in practice, it may be easier to start with NDC coordinates of <±1,±1,k> assigned to the corners of the viewport and <0,0,0> to the viewpoint then transform them by the inverse of the projection*view matrix to get view coordinates. This allows you to use the cube map in conjunction with other objects (i.e. as a skybox).

I am afraid that you lost me here.

Or just render a cube with texture coordinates of <±1,±1,±1> at the corners. This is probably easier to understand if you have problems with the inverse-projection maths.

How would this last option allow me to control for FOV and viewer’s current viewing direction?

I do not know if this is relevant or not… once the users establish what FOV they want, I want to be able to sample the cubemap to the maximum viewport resolution available (at least horizontally, ie. if the user set the horizontal FOV to 120 degrees then the maximum viewport resolution, in my case, 16384 is used. Hence each pixel would represent 120/16384 deg).

GClements · November 23, 2024, 12:28am

But a normal view doesn’t have a constant angle between pixels. The angle gets smaller as you get farther from the centre. The angle varies in proportion to arctan(x-xc) where xc is the x coordinate of the centre of projection.

Correct.

Construct the view pyramid in NDC then transform it to world space.

The same way that you do for any scene: by the construction of the view and projection matrices.

MLLO · November 24, 2024, 9:12pm

Hi @GClements,

I am still trying to wrap my head around some of your comments. Most likely is my inability to think about this in OpenGL terms and/or my limited familiarity with OpenGL. While my end result is to build something that uses OpenGL I am also trying to fully understand what I am doing and why.

I still do not understand why my earlier sampling scheme, using Spherical Coordinates, would be ‘incorrect’ or would give me the results I was getting when sampling the cubemap.

You suggested the following,

float u = x / width * 2 - 1;
float v = y / height * 2 - 1;
float s = u * tan(fov_x/2);
float t = v * tan(fov_y/2);
vec3 tex_coord = vec3(s,t,1);

This would generate sampling scheme that does not take into consideration the viewer’s camera yaw and pitch, correct? Would these angles (in radians) just be added to u and t???

My intention is for the user to be extract a panoramic view when moving around a terrain.

But a normal view doesn’t have a constant angle between pixels. The angle gets smaller as you get farther from the centre. The angle varies in proportion to arctan(x-xc) where xc is the x coordinate of the centre of projection.

Again why would a Spherical sampling at regular angular intervals would be incorrect? What exactly do you mean by the above sentence? I must confess I am still having difficulty processing the following two paragraphs,

Except in practice, it may be easier to start with NDC coordinates of <±1,±1,k> assigned to the corners of the viewport and <0,0,0> to the viewpoint then transform them by the inverse of the projection*view matrix to get view coordinates. This allows you to use the cube map in conjunction with other objects (i.e. as a skybox).

How would you do this? What I think you are saying is to use the u and v variables (as you defined above) to sample the cubemap? But then…???

Or just render a cube with texture coordinates of <±1,±1,±1> at the corners. This is probably easier to understand if you have problems with the inverse-projection maths.

Sorry, I am not quite grasping what you are trying to convey yet. Meanwhile I’ll keep trying to make sense of it all.

Thanks again for your time.

M

GClements · November 24, 2024, 11:29pm

The x and y variables would be scaleCoord.x and scaleCoord.y in your code.

No.

Because the screen isn’t a sphere, it’s flat. The angle between pixels gets smaller as you get farther from the centre.

I think you’re just going to have to work through the maths from first principles until you understand it.

MLLO · November 25, 2024, 12:59am

Thank you @GClements

I will endeavor to work through this. I still cannot understand why a spherical sampling scheme would not work given that the cubemap contains information around a viewer. I would have expected that the cubemap sampler would have been okay with sampling vectors at regular angular intervals.

Thanks again.
M.

MLLO · November 25, 2024, 4:16am

@GClements I think I see at least where there is some confusion.

I understand why you state that the angles subtended by each pixel becomes smaller as you move farther away from the viewpoint.

The way I have been thinking about it is a bit different. I have been thinking about that the image being generated is the result of sampling rays on the cubemap.If that makes sense at all. Perhaps this is the wrong way of thinking.

MLLO · November 26, 2024, 2:52am

I am thinking that I may not have provided all the background information needed to answer my question. So I will try to provide this here very briefly.

I manage to map the terrain surrounding a viewer’s position onto a cubemap. The cubemap was generated without any regard to the viewer’s orientation (yaw) and pitch only the position on the terrain. I want to be able to produce any ‘flatten’ view of the terrain from the viewer’s position. The viewer’s viewpoint must be at the center of the view but I should be able to choose any values for the FOV (both vertical and horizontal). So for instance one could choose a horizontal fov of 360 deg and a vertical fov of 60 deg or perhaps a horizontal fov of 120 deg and a vertical fov of 40 deg.
I do have access to the viewer’s viewpoint information (pitch, yaw so on).

Without changing how the cubemap was generated, what would the fragment shader look like to achieve the above goal?

OR

should the procedure to generate the cubemap be altered so that viewpoint information is actually taken into consideration?

GClements · November 26, 2024, 4:33am

That’s how you normally generate a cube map.

Right.

A cube map contains the entire scene as viewed from a specific point. Generating the cube map only needs to know the view position; the orientation (yaw, pitch, roll) and field of view don’t need to be known until you use the cube map to render a view.

Like the one I posted.

Really, for a skybox you don’t even need to use a cube map. You could render into six distinct 2D textures then render a cube using one texture for each face. If you have trouble with cube maps, you might want to adopt this approach. Or if you’re already generating the cube map correctly, just render a cube with 3D texture coordinates at the vertices. The fragment shader doesn’t have to perform any calculations, just pass the interpolated 3D texture coordinates to the texture() function.

Either way (six 2D textures or a cube map), you just need to transform the vertices of the cube the same way that you would transform any object in the scene, except that you don’t apply any translations, only rotations (and perspective).

Rendering a full-screen quad with the texture coordinates obtained by inverse-projecting the edges of the view frustum just reduces the polygon count slightly. The downside is that you have to understand the maths, whereas rendering a cube means you can just re-use whatever code you normally use without having to understand it.

The reason cube maps exist is for raytracing and raytracing-like techniques such as environment mapping (when you’re rendering the scene reflected in one or more objects). In that case, you can’t split the rendering process into six distinct faces, but need to be able to select the face on a per-fragment basis. That’s what cube maps allow.

MLLO · November 27, 2024, 1:44am

@GClements Thank you again for being patient with my questioning and difficulties.

As I mentioned previously I successfully generated a cubemap with all the information around a viewer. I have double checked this by actually saving each face of the cubemap to an image file as I was creating it.

Currently, I am able to roam anywhere around the terrain using a very basic interface. Then by pressing the s key, I am able to generate/‘capture’ and save a terrain rendering onto my computer… I have it set up so that this rendering has an hfov of 120 deg and a vfov of 60 deg (this can be altered).

To illustrate what I get. Here is what I was looking at when I ‘captured’ the terrain,

Next (see after the code) is the result I get after saving the image. I created this rendering using the following shader based on your input in your second message, i.e.

#version 440 core

uniform vec2 display_size;
uniform vec2 offset; // yaw, pitch of the viewer?
uniform float hfov;
uniform float vfov;
uniform samplerCube cube;

out float color;

void main()
{
    float u = (gl_FragCoord.x / display_size.x * 2 - 1);
    float v = (gl_FragCoord.y / display_size.y * 2 - 1);
    float s = u * tan(hfov / 2);
    float t = v * tan(vfov / 2);
    vec3 tex_coord = vec3(s, t, 1);

    color = texture(cube, tex_coord).r;

}

I totally get what u, v are. And I understand that s, t are used as a scaling factor in order to restrict the sampling to the desired hfov and vfov respectively. What I am not entirely certain is why are we using the tangent of hfov/2, and vfov/2 to do the scaling and not just the angle (in my original posting I multiplied both u, v by the angles: hfov, vfov). Where does this expression come from? this is probably trivial but I am having difficulties seeing where this is derived from?

This is the result of using the above fragment shader.

Because of the previously mentioned fov parameters the resulting image I get is much wider than what I saw when roaming (as expected). I does appear that this rendering extends 120 deg horizontally (= hfov) and 60 degs vertically (=vfov) but it is not centered on the right viewpoint. In fact the feature I was looking at does no appear in the rendering. This is the part that I cannot see how to work out based on your comments.

I hope this make a bit more sense. With regards to some other of your comments.

This is what I did in the vertex shader I used when I was building my cubemap? is this what you were referring to?

Yes, definitely this is another part I struggle with.

If there is any reference regarding this type of operation that I can consult, I would happy to do so.

Thank you again for your time.

GClements · November 27, 2024, 3:29am

Those are the corners of the view frustum.

If the apex is the origin, the corners are at <±tan(fov_x/2),±tan(fov_y/2),1>. So multiplying u and v (which are in the range [-1,1]) by the tangent of the appropriate half angle gives you <s,t,1> which is a point on the base (note that cube map lookups depend only upon the vector’s direction and aren’t affected by its magnitude). Thus the rendered quad is a view with the appropriate field of view. Each fragment samples the cube map at the correct point (the sample points are equally-spaced on the base, not equally-spaced angles).

MLLO · December 5, 2024, 2:32am

I fully acknowledge that this is not the traditional way of sampling a cubemap but I would still like to understand why given my current setting I am obtaining the results I end up with (see images below). I have checked whether the cubemap I am sampling is correct, by plotting each of its faces, and this does not appear to be the problem.

Let me recap very briefly ,… Through a very simple interface a user can walk around a 3D terrain. By pressing a particular key, a cubemap containing the depth of scene around the current viewer’s location is generated and then sampled in order to generate a 2d version of the scene centered on the viewer’s viewpoint. The user can specify the FOV to be used for the sampling (e.g. 360 or 120 degreed horizontally and say 50 or 70 deg vertically). The idea is that half FOV will be distributed evenly above/below and on both sides the viewer’s viewpoint .The result of the sampling is rendered using a flat quad covering the entire size of an offscreen window. The window or display size has specific dimensions calculated in the following way. Let me provide an example. If the user specified FOV = (120, 50) and angle interval of 1 deg, then the display size will be (120 x 1) x (50 x 1), if the angle interval had been 0.5 degrees then the size would have been (120 x 1/0.5) x (50 x 1/0.5) or 240 x 100.

I used the following fragment shader to sample the cubemap,

#version 440 core

uniform vec2 display_size;
uniform vec2 offset; // current viewer's camera yaw and pitch
uniform vec2 fov;
uniform samplerCube cube;

out float color;

void main()
{
    // Calculate angles used for sampling
    float u = (gl_FragCoord.x / display_size.x) * 2 - 1;
    float v = (gl_FragCoord.y / display_size.y) * 2 - 1;
    float theta = u * radians(fov.x/2) + radians(offset.x);    
    float phi   =  v * radians(fov.y/2) + radians(offset.y);

    // make sure that phi is between [-89.9, 89.9]
    phi = clamp(phi, radians(-89.9), radians(89.9));

    // calculate sampling coordinates
    vec3 dir;
    dir.x = -cos(phi) * cos(theta);
    dir.y = sin(phi);
    dir.z = cos(phi) * sin(theta);

    color = texture(cube, dir).r;
}

The results are very much dependent on the pitch of the viewer’s camera. If I down pitch too much,

It appears that I run out of values at the bottom. If on the other hand I pitch the camera upwards too much,

The opposite occurs. While the sampling across is exactly what I would expect (and want) this is not true of the vertical. Given the setting I am using, I am wondering if anyone could provide me with some explanation as to what it is happening and whether there is some solution?

Thank you again for all your help.

M.

GClements · December 5, 2024, 6:53pm

Did you swap the top and bottom faces when rendering the cube map?

The results suggest that the bottom face is solid white while the top face is (roughly) solid purple. Which is the opposite of what would be expected.

MLLO · December 5, 2024, 8:04pm

@GClements
Of course!!!
When creating the cubemap I had incorrectly rotated the camera and mislabeled the faces in my cubemap. This had generated a false sense of security when checking the cubemap (it all appeared to be correct). It did not affect the horizontal aspect of the rendering because even though my labels were incorrect the procedure was using the correct information. This was not true when dealing with the vertical aspect of the rendering.

So bottom line… totally my own doing. I apologize for the many messages spent on this. Here are the new results,