Difference blend mode, or something similar

I’m trying to use either a solid color (white) or a texture as a mask for inverting pixels. What I’ve found is this post on Game Development Stack Exchange, and, while the blend mode works great for drawing polygons (the circle shown on the screenshot), it does not appear to work correctly for drawing textured rectangles such as text:

image

The funky rectangles on the above screenshot should be glyphs, but my glyph textures use unpremultiplied alpha (I use texture swizzling to convert a single-channel GL_RED texture to (1, 1, 1, alpha)), so modifying their color doesn’t quite work. I’d need to stuff a multiplication by the source alpha there somehow, but there is no way of doing that. If I make my font renderer use premultiplied alpha, then my alpha blending mode needs to change, and that seems to break image and transparent rectangle rendering, which uses unpremultiplied alpha.

My question here is, how do I make the blend mode work with textures? Is it even possible with fixed-function blend modes, or do I have to resort to shaders? If I have to use shaders and framebuffers, how can I achieve the correct result?

The Stack Exchange post suggests

glBlendFunc(GL_ONE_MINUS_DST_COLOR, GL_ZERO);

In case it isn’t obvious, the source alpha isn’t used at all here.

With the fixed-function pipeline, you can use glEnable(GL_ALPHA_TEST) and glAlphaFunc to discard fragments whose alpha value falls below some threshold. With shaders, you would have the fragment shader execute a discard statement depending upon alpha.

Another option (which works fine if the mask is black/white, less so for intermediate shades) is glLogicOp(GL_XOR).

With OpenGL 4.0 or later, you can use glBlendEquation with GL_FUNC_SUBTRACT or GL_FUNC_REVERSE_SUBTRACT to perform “negative blending”. In that case, you’d use a more conventional blending mode with glBlendFunc.

Bear in mind that there isn’t a solution which will work for every possible case. E.g. if the background colour is 50% grey, inverting it has no effect.

Unfortunately, when rendering glyphs, intermediate shades of gray are involved, so using GL_ALPHA_TEST or
glLogicOp(GL_XOR) is not viable for my use case, hence why I want to use blending modes.

I tried to use GL_FUNC_SUBTRACT and GL_FUNC_REVERSE_SUBTRACT, like so:

glBlendEquationSeparate(GL_FUNC_REVERSE_SUBTRACT, GL_FUNC_ADD);
glBlendFunc(GL_SRC_ALPHA, GL_ONE);

This gives me the blend equation dst - src_alpha * src, which does subtractive blending, but isn’t what I want (it doesn’t invert colors, it simply subtracts them from one another to produce a darker color). The difference blend mode is abs(dst - src) so that negative values aren’t clamped, but rather flipped into the positive, which obviously can’t be replicated with the fixed function pipeline in this exact form, so I’m looking for something close to that which can achieve my desired effect.

Bear in mind that there isn’t a solution which will work for every possible case. E.g. if the background colour is 50% grey, inverting it has no effect.

I am aware of the limitations of inverting the colors this way; my application does not have any 50% grays (at least not yet) so it shouldn’t be a problem.

Thinking through this once more, I might be able to alter my shader to premultiply the resulting color by its alpha according to a uniform value. Something like:

in vec2 vertex_uv;

uniform sampler2D sampler;
uniform float premultiply_alpha;

out vec4 color;

void main(void)
{
  vec4 texel = texture(sampler, vertex_uv);
  // Branchless version of:
  // float factor = premultiply_alpha ? texel.a : 1.0;
  float factor = premultiply_alpha * texel.a + (1.0 - premultiply_alpha);
  texel.rgb *= factor;
  color = texel;
}

Using that, I’d be able to simply flip a switch before rendering text or images with my inverting blending mode, and then flip it back off when I’m done rendering.

Edit: it worked! I had to alter my blending mode a bit though.

The final solution involves premultiplying the source color as I already said above, as well as adding the destination color times the source alpha to the final color. The blend function is as follows:

glBlendEquation(GL_FUNC_ADD)
glBlendFuncSeparate(GL_ONE_MINUS_DST_COLOR, GL_ONE_MINUS_SRC_ALPHA, GL_ZERO, GL_ONE)

And the final equation is:

// This is performed in the shader:
premultiplied_src_color = (src_r * src_a, src_g * src_a, src_b * src_a, src_a)
// Then, the fixed-function blend mode is:
out_color = ((1, 1, 1, 1) - dst_color) * premultiplied_src_color + (1 - src_a) * dst_color
out_alpha = dst_alpha

image

Or even OpenGL 1.2 or later.

Difference blending is directly supported via NV_blend_equation_advanced or KHR_blend_equation_advanced.

You could also use any of the programmable blending extensions.

According to the 0 A. D. OpenGL capabilities database (which is quite old by this point - it’s from 2015. If you know a better, more up to date source, please let me know), these extensions don’t seem to be very well supported, with {NV,KHR}_blend_equation_advanced both being supported by only 14% of graphics drivers.

Here. The blend_equation_advanced extensions seem reasonably well supported on hardware of a decently recent vintage (ie: graphics chips that are still getting drivers). Even Intel.

That’s nice to hear, though I have already solved my issue without resorting to extensions. Thanks for the heads up, though!