GLES20 Z-Order for stacked decals on flat surfaces with texture atlas.

I am drawing multiple batches of quads from a multi page texture atlas. Each pages texture is activated once and then all batches are called to draw their subset of quads that need tiles from the current page.
As a result drawing back-to-front or front-to-back is not possible. So I use depth testing.

Now I need to draw multiple quads with different textures in the exact same place. Imagine a card from a card game with a tile for the background, some for decorations, one for an image and many for text on the card.
(I actually use Signed Distance Field tiles, but that does not effect the problem much.)

I get terrible z-fighting problems with all those layers on the same plane.
Polygonoffset is not really a help because I cannot call it between rendering the quads in one drawcall.
I tried to use my own z-Layer per quad and shift its glPosition.z coordinate towards the camera, in the vertex shader.
That works so-so. I am having trouble finding the right amount of shifting, to not overlay other objects that should be in front.
I would need something like the minimal shift that can be applied to gl_Position. But in some cases that might still push the layers to close to the camera and override other objects?

Is there a better way to do this? I need the cards to be full 3d objects. They are moved around in space and flipped etc. I also need a perspective transform. Switching to ortho is not an option, because I could not animate cards in 3d then.
Maybe filling the depth buffer by rendering all geometry unshaded and each card only with a single quad. Then rendering each card again using GL_EQUAL on the depth test and updating the stencil buffer where the card can be seen. And then rendering the stacked layers (whose order I know) without depth testing but against the mask in the stencil buffer. But that sounds worse than giving up the batched rendering and drawing everything in back-to-front or the like.

To cure depth-fighting, you need to offset the depth, which depends upon the normalised Z, which is equal to the ratio of clip-space Z to clip-space W: Zndc = Zclip/Wclip. So to offset Zndc by a given amount, you need to offset Zclip (i.e. gl_Position.z) by that amount multiplied by Wclip (i.e. gl_Position.w).

The amount by which it needs to be offset depends upon the depth. Specifically, it needs to be proportional to
Zndc2*(zFar-zNear)/(zFar*zNear).

The offset needs to be larger for larger (more negative) Z because the depth resolution gets worse as you get farther from the near plane.

If depth buffer resolution is the only issue, multiplying 1/2depthBits by the above should be sufficient.

Thank you!
So i would be doing something like

gl_Position.z += uDepthLayer * uMinDepthStep *gl_Position.w;

and push each decal by one layer closer to the camera.
The uMinDepthStep would be the smallest change in the z-buffer and as you wrote
1/(2GL_DEPTH_BITS)

Edit:

Or actually 1/(2GL_DEPTH_BITS-1)? At 24Bits that would not make much of a differnce and the larger value might be better to guarantee a change in depth?

But since the z-buffer is not linear, I have to scale the z-shift depending on the vertices positions.
Otherwise i would waste precision close to near plane and might push objects too far forward.
And i might not be able to separate objects that are too far away and require a larger shift.

I am having trouble to understand the scaling formula.
You wrote to multiply by
Zndc2*(zFar-zNear)/(zFar*zNear)
NDC are left handed, so larger values mean further away and would result in a larger shift. That much is clear.
But i am having trouble to piece the rest together.
I mostly look at
https://www.opengl.org/wiki/Depth_Buffer_Precision
http://www.sjbaker.org/steve/omniv/love_your_z_buffer.html
https://developer.nvidia.com/content/depth-precision-visualized
and i cannot get to that scaling formula. It should just be the retransform to clip space, right?
Could you please explain that a bit more?

(And if i calculate Zndc as part of transforming the z-shift into clip space, should i not be able to transform zFar,zNear to clip space once, and then calculate the scaling directly from gl_Position.z?)

Sorry, ignore that. That equation is for eye space, not NDC.

In NDC, the non-linearity has already been accounted for; the relationship between Zndc and depth is linear.

So:

Yes, that should suffice. Except that uMinDepthStep should be twice that value, because Zndc ranges from -1 to +1 while the depth ranges from 0 to 1.

If there are other rounding errors involved, the value may need to be larger. The best thing to do is to test it, finding the value for which depth-fighting disappears.

Phew. You should see the scribblings on my desk. I was beginning to lose faith. :wink:

It does not seem to be that simple. I did a few test. Even with high floating point precision 1/2^24 does nothing and times 2 neither. Times 10 is enough to push the texture to the front. However the effect heavily depends on the near/far values and also on the position of quad inside the frustum. If i make the frustum longer or move the quad outside of the high res area, the textures start to fight again. There clearly seems to be a scaling factor that needs to be handled.

With a 24-bit depth buffer, the precision of the intermediate results will be as much of an issue as the depth buffer resolution; single-precision floating point only has a 24-bit significand.

The absolute error in a floating-point value increases with magnitude, so you may need to scale the offset by gl_Position.w again (effectively re-introducing the Z2 term from the eye-space equation) to account for this.

Or it may be better to determine the relationship empirically; experiment to find the required offset for various Z values and fit a curve to them.

For a fixed offset, folding it into the projection matrix may improve accuracy compared to applying it afterwards.