Fragment shader - terrain brush, circle


I googled for terrain brush and I found some unsolved fragment shader codes, they didn’t work. I’m beginner with GLSL.

I think, this code will work, but I don’t know how to do “if”, because if there isn’t my whole terrain is green, cause per pixel it add green color.


#version 400

layout (location = 0) in vec2 vertexPosition;
layout (location = 1) in vec2 vertexTexCoord;

out terrainVertex
    vec2 position;
    vec2 texCoord;
} Out;

void main()
    Out.position = vertexPosition;
    Out.texCoord = vertexTexCoord;

    gl_Position = vec4(vertexPosition, 1.0, 1.0);


in terrainVertex
    vec2 position;
    vec2 texCoord;
} In;

vec4 brushColor = vec4(0, 1, 0, 1);

uniform int   brush;
uniform vec2  cursorPos;
uniform float brushRadius;

void main()

    // Terrain brush
    if(brush == 1)
        float dx = In.texCoord.x - 0.5;
        float dy = In.texCoord.y - 0.5;

        float dist = sqrt(dx * dx + dy * dy);

        if(dist < ?????)
            outColor += mix(brushColor, brushColor, smoothstep(brushRadius, brushRadius, dist));

    fragColor = outColor;

Let me get this straight. You pick a position on your “terrain”, which at present seems to be only variable in x and z with a constant z, and want to apply color linearly blended with current color within a radius around the intersectio point. Correct? Am I right in assuming you’re attempting to write a real-time heightmap editor?

100% correct, let me just little more explain in followed screen (edit in mspaint)

(Its too large for inserting here)

I have world cursor x,z, brush radius and I’m trying to create in fragment shader colored texture in radius

I have world cursor x,z, brush radius and I’m trying to create in fragment shader colored texture in radius

And the circle is to be projected onto the terrain to visualize the area that’s going to be painted with the current radius?

Yes that’s again correct

For this purpose, you probably want to use projective texturing, i.e. a projector hovering above your terrain looking straight down, focusing on your center point with a symmetrical frustum and and an orthographic projection. If that’s too confusing, let us know. :wink:

Isn’t better to use this? Projective texturing mean next texture layer? And I don’t use Ortho projection, I’m using Perspective.

And you have true, it’s little confusing :smiley:

Check out this tutorial by Alfonse.

In short: When projectively texturing, you map world-space coordinates (or generally coordinates in any space as long as they are properly invertible to world-space) to the texture-space (i.e. projected x/y in [0,1]) of a so called projector. Think overhead projector like in a class room that projects some slide on the wall. You (or the camera in your scene) sees parts of the wall, you then transform those parts of the wall into the texture-space of the projector. Say your slide is your texture. After projection to texture-space, you know the texture coordinate of each fragment in the projected area of the wall. Using these texture coordinatesy, you then simply map texels from the slide to little portions of the wall by assigning the wall fragments those texels. In your case, the slide would simply contain a circle and the rest of the texels could be black with 0 alpha-values. The good thing is, all you need is the corresponding projection matrix of the projector and the texture to lookup. You can do the terrain texturing and the projection in one step. Actually this is almost the same as shadow mapping, only that the values of depth texture values are usually used as a factor weighting light contribution and not fragment color.

Somehow I can’t explain it better without drawing something but it’s really a lot easier than it sounds. :smiley: I think Alfonse’s tutorial speaks for itself though. Come back if you got some specific problems.

EDIT: When you got the basic approach, we can talk about performance.

So I need to post Projection matrix to fragment shader for calculation texture coords?

Still some trouble with understanding this proj texturing

EDIT: Currently I just understand why I need proj texturing, but how to get it?

Hi again,

I figured out I already have texture coordinates in fragment shader, I copied from network some shaders with tesselation, and they’re was included.

Currently with

        // middle of camera
        float dx = position.x;
        float dy = position.y;

        float bDist = sqrt(dx * dx + dy * dy);

        if(bDist < brushRadius)
             float str = max(0, mix(-1.5, 0.5, bDist / brushRadius));
             outColor += brushColor * str;

I have cursor in middle of screen cause position.x, y is some in .tes calculated or .tcs, there is also texCoords which I must use, but I don’t see brush when I use formula “texCoords.x - cursorPos.x”

EDIT: I need cursor position from World space right? mean 2D coords to 3D (terrain pos)


Currently I solved it!

texCoords.x was maximum to 1.0 in range 0 to 1.

So I multiply it by heightmap width and subtracted cursorPosition

Finally code:

        float dx = texCoords.x * horizontalScale - cursorPos.x;
        float dy = texCoords.y * horizontalScale - cursorPos.y;

        float bDist = sqrt(dx * dx + dy * dy);
        float bRadius = brushRadius;

        if(bDist < bRadius)
            float str = max(0, mix(-1.5, 0.5, bDist / brushRadius));
            outColor += brushColor * str;

But still I need your help!
My actual problem is my brushRadius is very big, In my function to changeTerrain(raise) is radius 10.0f and with uniform in fragment shader there is also 10.0f! But size is very big, why? (You can see radius on followed picture)

Look on picture, there is also raised terrain, I think u figure it out!

Link to picture

First of all: derp! I feel like an idiot for suggesting projective texturing for this. It’s applicable, no doubt, but way to much overhead by comparison. The solution you have is much easier.

But size is very big, why?

Probably because you pass in 10.f? :smiley: If you want a reduced radius, just pass a smaller value to brushRadius.

EDIT: Thx for recognizing that I “have again true”. :wink:

I’m such a idiot!!! You have again true!

Look on this.

To Shader:

        shader->setUniformValue("brush", shaping_brush);
        shader->setUniformValue("cursorPos", QVector2D(terrain_pos.x(), terrain_pos.z()));
        shader->setUniformValue("brushRadius", shaping_radius); // <-------------------------------

To raise terrain:

//                                                                                        MISTAKE
changeTerrain(terrain_pos.x(), terrain_pos.z(), 7.5f * dt * shaping_speed, shaping_radius / 5.33333f, shaping_brush);

This topic was automatically closed 183 days after the last reply. New replies are no longer allowed.