Emulating GL_CLAMP_TO_BORDER in GLSL on a portion of a texture

[acceleration3]

I am currently making a text renderer in OpenGL and I stumbled into a bit of a problem. I am trying to add a border of X pixels around a quad kind of like GL_CLAMP_TO_BORDER would look like but I’m trying to do it within the fragment shader and instead of a color I just want to make it transparent.

I have this vertex shader that adds a border of a certain size in pixels around my quads:

in vec2 vs_position;
in vec2 vs_uv;
in vec4 vs_color;

out vec2 fs_uv;
out vec4 fs_color;

uniform mat4 projMatrix;
uniform float padding;

vec2 addPadding(vec2 vertex)
{
	vertex.x += (gl_VertexID % 2 == 0 ? -padding : padding);
	vertex.y += (gl_VertexID % 4 < 2 ? -padding : padding);
	return vertex;
}

void main()
{
    gl_Position = projMatrix * vec4(addPadding(vs_position), 0.0, 1.0);
	fs_color = vs_color;
	fs_uv = vs_uv;
}

And this fragment shader:

in vec2 fs_uv;
in vec4 fs_color;

out vec4 fragColor;

uniform sampler2D texSampler;
uniform float padding;

float mapWithPadding(sampler2D sampler, vec2 uv)
{
	// Crop the texture here
	float luminosity = texture(sampler, uv).r;
	return luminosity;
}

void main()
{
    fragColor = vec4(fs_color.rgb, mapWithPadding(texSampler, fs_uv));
}

The main problem is that since this is working with a texture atlas with all the text characters in it, the UV coordinates and the size I get from textureSize() are relative to the entire atlas, and thus I can’t use them to map the border region of the quad. Is this even possible without changing the vertex data structure I currently have?

[GClements]

It isn’t entirely clear what you’re trying to achieve, but I think that you’re trying to enlarge the quad without affecting the result (other than within the border). In which case, it’s possible provided that the texture scale is fixed. As well as adjusting the vertex positions, adjust the texture positions by a proportional amount. This should result in adding more pixels to the edge of the quad without affecting the contents of the unmodified quad.

To determine whether you’re in the border, add quad-space coordinates in the vertex shader:

out vec2 param;
...
	param.x = (gl_VertexID % 2 == 0 ? 0 : 1);
	param.y = (gl_VertexID % 4 < 2 ? 0 : 1);

In the fragment shader, the dimensions of the current quad in pixels are 1/dFdx(param.x) and 1/dFdy(param.y). Multiplying param by these values gives you the distance from the left and bottom edges in pixels, while multiplying 1-param gives you the distance from the right and top edges.

[acceleration3]

That is exactly what I was looking for! Thank you very much for the explanation.