Nvidia VRS (Variable Rate Shading) extension implementation issue

[zMendes]

Hi everyone,

I’m trying to implement a project using Nvidia’s VRS OpenGL extension. I found a Nvidia repo that demonstrates its usage and managed to run it sucessfully on my machine.

Howver, when trying to implement VRS in my own project I can’t seem to get it working properly. I’m rendering 4 cubes and depending on a texture the shading for each fragment should change. The colors in the cube should be different, but since all fragments appear red, it suggests that they are all being created with the default GL_SHADING_RATE_1_INVOCATION_PER_PIXEL_NV, meaning VRS isn’t applying as expected.

My GPU is an RTX 4060, and since the Nvidia demo runs fine on my machine, I don’t think this is a hardware compatibility issue.

For some reason I cannot upload links, so I’ll just put my code here:

main.cpp

#include <glad/glad.h>
#include <GLFW/glfw3.h>
#include <glm/glm.hpp>
#include <glm/gtc/matrix_transform.hpp>
#include <glm/gtc/type_ptr.hpp>

#include "shader.h"
#include "camera.h"
#include <iostream>
#include <algorithm>
#include <vector>

#define GL_SHADING_RATE_IMAGE_NV 0x9563
#define GL_SHADING_RATE_IMAGE_PER_PRIMITIVE_NV 0x95B1
#define GL_SHADING_RATE_IMAGE_TEXEL_WIDTH_NV 0x955C
#define GL_SHADING_RATE_IMAGE_TEXEL_HEIGHT_NV 0x955D
#define GL_SHADING_RATE_IMAGE_PALETTE_SIZE_NV 0x955E
#define GL_SHADING_RATE_NO_INVOCATIONS_NV 0x9564
#define GL_SHADING_RATE_1_INVOCATION_PER_PIXEL_NV 0x9565
#define GL_SHADING_RATE_IMAGE_PALETTE_SIZE_NV 0x955E
#define GL_SHADING_RATE_1_INVOCATION_PER_2X2_PIXELS_NV 0x9568
#define GL_SHADING_RATE_1_INVOCATION_PER_4X4_PIXELS_NV 0x956B

typedef void(APIENTRYP PFNGLBINDSHADINGRATEIMAGENVPROC)(GLuint texture);
PFNGLBINDSHADINGRATEIMAGENVPROC glBindShadingRateImageNV = nullptr;

typedef void(APIENTRYP PFNGLSHADINGRATEIMAGEPALETTENVPROC)(GLuint viewport, GLuint first, GLsizei count, const GLenum *rates);
PFNGLSHADINGRATEIMAGEPALETTENVPROC glShadingRateImagePaletteNV = nullptr;

void framebuffer_size_callback(GLFWwindow *window, int width, int height);
void processInput(GLFWwindow *window);
void mouse_callback(GLFWwindow *window, double xpos, double ypos);
void scroll_callback(GLFWwindow *window, double xoffset, double yoffset);
void createFoveationTexture(float centerX, float centerY);
void uploadFoveationDataToTexture(GLuint texture);
void setupShadingRatePalette();
void APIENTRY glDebugOutput(GLenum source, GLenum type, unsigned int id, GLenum severity, GLsizei length, const char *message, const void *userParam);
void renderCube();
void createTexture(GLuint &glid);

// settings
const unsigned int SCR_WIDTH = 800;
const unsigned int SCR_HEIGHT = 600;

// VRS stuff
GLuint fov_texture;
std::vector<uint8_t> m_shadingRateImageData;
uint32_t m_shadingRateImageWidth = 0;
uint32_t m_shadingRateImageHeight = 0;
GLint m_shadingRateImageTexelWidth;
GLint m_shadingRateImageTexelHeight;

// CAMERA
Camera camera(glm::vec3(0.0f, 2.0f, 8.0f), glm::vec3(0.0f, 1.0f, 0.0f), -90.0f, 0.0f);
bool firstMouse = true;
float lastX = SCR_WIDTH / 2.0f;
float lastY = SCR_HEIGHT / 2.0f;

float deltaTime = 0.0f; // Time between current frame and last frame
float lastFrame = 0.0f; // Time of last frame
int main()
{
    // glfw: initialize and configure
    glfwInit();
    glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 4);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 6);
    glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);

    // glfw window creation
    GLFWwindow *window = glfwCreateWindow(SCR_WIDTH, SCR_HEIGHT, "VRS", NULL, NULL);
    if (window == NULL)
    {
        std::cout << "Failed to create GLFW window" << std::endl;
        glfwTerminate();
        return -1;
    }
    glfwMakeContextCurrent(window);
    glfwSetFramebufferSizeCallback(window, framebuffer_size_callback);

    glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
    glfwSetCursorPosCallback(window, mouse_callback);
    glfwSetScrollCallback(window, scroll_callback);

    if (!gladLoadGLLoader((GLADloadproc)glfwGetProcAddress))
    {
        std::cout << "Failed to initialize GLAD" << std::endl;
        return -1; // Correct return here, exiting if GLAD fails
    }

    glBindShadingRateImageNV = (PFNGLBINDSHADINGRATEIMAGENVPROC)glfwGetProcAddress("glBindShadingRateImageNV");

    if (!glBindShadingRateImageNV)
    {
        std::cerr << "Failed to load glBindShadingRateImageNV!" << std::endl;
    }
    else
    {
        std::cout << "Successfully loaded glBindShadingRateImageNV!" << std::endl;
    }

    glShadingRateImagePaletteNV = (PFNGLSHADINGRATEIMAGEPALETTENVPROC)glfwGetProcAddress("glShadingRateImagePaletteNV");

    if (!glShadingRateImagePaletteNV)
    {
        std::cerr << "Failed to load glShadingRateImagePaletteNV!" << std::endl;
    }
    else
    {
        std::cout << "Successfully loaded glShadingRateImagePaletteNV!" << std::endl;
    }

    if (!glfwExtensionSupported("GL_NV_shading_rate_image"))
    {
        std::cerr << "GL_NV_shading_rate_image not supported!" << std::endl;
    }

    // OPENGL STATE
    glEnable(GL_DEPTH_TEST);
    glEnable(GL_SHADING_RATE_IMAGE_NV);
    glEnable(GL_SHADING_RATE_IMAGE_PER_PRIMITIVE_NV);

    glGetIntegerv(GL_SHADING_RATE_IMAGE_TEXEL_HEIGHT_NV, &m_shadingRateImageTexelHeight);
    glGetIntegerv(GL_SHADING_RATE_IMAGE_TEXEL_WIDTH_NV, &m_shadingRateImageTexelWidth);

    m_shadingRateImageWidth = (SCR_WIDTH + m_shadingRateImageTexelWidth - 1) / m_shadingRateImageTexelWidth;
    m_shadingRateImageHeight = (SCR_HEIGHT + m_shadingRateImageTexelHeight - 1) / m_shadingRateImageTexelHeight;
    m_shadingRateImageData.resize(m_shadingRateImageWidth * m_shadingRateImageHeight);

    createTexture(fov_texture);
    setupShadingRatePalette();
    createFoveationTexture(0.5, 0.5);

    Shader shader("vrs.vs", "vrs.fs");
    shader.use();

    GLenum err;
    while ((err = glGetError()) != GL_NO_ERROR)
    {
        std::cerr << "OpenGL error before main loop: " << err << std::endl;
    }

    while (!glfwWindowShouldClose(window))
    {
        float currentFrame = glfwGetTime();
        deltaTime = currentFrame - lastFrame;
        lastFrame = currentFrame;

        // input
        processInput(window);

        // render
        glClearColor(0.2f, 0.3f, 0.3f, 1.0f);
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); // also clear the depth buffer now!

        // render the triangle
        shader.use();

        // view matrix
        glm::mat4 view = camera.GetViewMatrix();
        shader.setMat4("view", view);
        shader.setVec3("viewPos", camera.Position);
        // projection matrix
        glm::mat4 projection = glm::perspective(glm::radians(camera.Zoom), (float)SCR_WIDTH / (float)SCR_HEIGHT, 0.1f, 1000.0f);
        shader.setMat4("projection", projection);

        //cube 1
        glm::mat4 model = glm::mat4(1.0f);
        model = glm::translate(model, glm::vec3(0.0f, 0.f, 0.0f));
        model = glm::scale(model, glm::vec3(5.f, 5.f, 5.f));
        shader.setMat4("model", model);
        renderCube();

        //cube 2
        model = glm::mat4(1.0f);
        model = glm::translate(model, glm::vec3(0.0f, 12.f, 0.0f));
        model = glm::scale(model, glm::vec3(5.f, 5.f, 5.f));
        shader.setMat4("model", model);
        glEnable(GL_SHADING_RATE_IMAGE_NV);
        renderCube();

        //cube 3
        model = glm::mat4(1.0f);
        model = glm::translate(model, glm::vec3(12.f, 12.f, 0.0f));
        model = glm::scale(model, glm::vec3(5.f, 5.f, 5.f));
        shader.setMat4("model", model);
        glEnable(GL_SHADING_RATE_IMAGE_NV);
        renderCube();

        //cube 4
        model = glm::mat4(1.0f);
        model = glm::translate(model, glm::vec3(12.f, 0.f, 0.0f));
        model = glm::scale(model, glm::vec3(5.f, 5.f, 5.f));
        shader.setMat4("model", model);
        glEnable(GL_SHADING_RATE_IMAGE_NV);
        renderCube();

        glfwSwapBuffers(window);
        glfwPollEvents();
    }

    glfwTerminate();
    return 0;
}

void processInput(GLFWwindow *window)
{
    if (glfwGetKey(window, GLFW_KEY_ESCAPE) == GLFW_PRESS)
        glfwSetWindowShouldClose(window, true);
    if (glfwGetKey(window, GLFW_KEY_W) == GLFW_PRESS)
        camera.ProcessKeyboard(FORWARD, deltaTime);
    if (glfwGetKey(window, GLFW_KEY_S) == GLFW_PRESS)
        camera.ProcessKeyboard(BACKWARD, deltaTime);
    if (glfwGetKey(window, GLFW_KEY_A) == GLFW_PRESS)
        camera.ProcessKeyboard(LEFT, deltaTime);
    if (glfwGetKey(window, GLFW_KEY_D) == GLFW_PRESS)
        camera.ProcessKeyboard(RIGHT, deltaTime);
}

// glfw: whenever the window size changed (by OS or user resize) this callback function executes
void framebuffer_size_callback(GLFWwindow *window, int width, int height)
{
    glViewport(0, 0, width, height);
}

void mouse_callback(GLFWwindow *window, double xpos, double ypos)
{
    if (firstMouse)
    {
        lastX = xpos;
        lastY = ypos;
        firstMouse = false;
    }
    float xoffset = xpos - lastX;
    float yoffset = lastY - ypos; // reversed: y ranges bottom to top
    lastX = xpos;
    lastY = ypos;
    const float sensitivity = 0.3f;
    xoffset *= sensitivity;
    yoffset *= sensitivity;

    camera.ProcessMouseMovement(xoffset, yoffset);
}

void scroll_callback(GLFWwindow *window, double xoffest, double yoffset)
{
    camera.ProcessMouseScroll(static_cast<float>(yoffset));
}

void createFoveationTexture(float centerX, float centerY)
{
    const int width = m_shadingRateImageWidth;
    const int height = m_shadingRateImageHeight;

    for (int y = 0; y < height; ++y)
    {
        for (int x = 0; x < width; ++x)
        {
            float fx = x / (float)width;
            float fy = y / (float)height;

            float d = std::sqrt((fx - centerX) * (fx - centerX) + (fy - centerY) * (fy - centerY));
            if (d < 0.15f)
            {
                m_shadingRateImageData[x + y * width] = 1;
            }
            else if (d < 0.3f)
            {
                m_shadingRateImageData[x + y * width] = 2;
            }
            else if (d < 0.45f)
            {
                m_shadingRateImageData[x + y * width] = 3;
            }
            else
            {
                m_shadingRateImageData[x + y * width] = 0;
            }
        }
    }
}

void uploadFoveationDataToTexture(GLuint texture)
{
    glBindTexture(GL_TEXTURE_2D, texture);
    glTexStorage2D(GL_TEXTURE_2D, 1, GL_R8UI, m_shadingRateImageWidth, m_shadingRateImageHeight);
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, m_shadingRateImageWidth, m_shadingRateImageHeight, GL_RED_INTEGER, GL_UNSIGNED_BYTE, &m_shadingRateImageData[0]);
}

void setupShadingRatePalette()
{
    GLint palSize;
    glGetIntegerv(GL_SHADING_RATE_IMAGE_PALETTE_SIZE_NV, &palSize);
    assert(palSize >= 4);

    GLenum *palette = new GLenum[palSize];

    palette[0] = GL_SHADING_RATE_NO_INVOCATIONS_NV;
    palette[1] = GL_SHADING_RATE_1_INVOCATION_PER_PIXEL_NV;
    palette[2] = GL_SHADING_RATE_1_INVOCATION_PER_2X2_PIXELS_NV;
    palette[3] = GL_SHADING_RATE_1_INVOCATION_PER_4X4_PIXELS_NV;

    for (int i = 4; i < palSize; ++i)
    {
        palette[i] = GL_SHADING_RATE_1_INVOCATION_PER_PIXEL_NV;
    }

    glShadingRateImagePaletteNV(0, 0, palSize, palette);
    delete[] palette;

    GLenum *paletteFullRate = new GLenum[palSize];
    for (int i = 0; i < palSize; ++i)
    {
        paletteFullRate[i] = GL_SHADING_RATE_1_INVOCATION_PER_PIXEL_NV;
    }

    glShadingRateImagePaletteNV(1, 0, palSize, paletteFullRate);
    delete[] paletteFullRate;
}
unsigned int cubeVAO = 0;
unsigned int cubeVBO = 0;
void renderCube()
{
    if (cubeVAO == 0)
    {
        float vertices[] = {
            // back face
            -1.0f, -1.0f, -1.0f, 0.0f, 0.0f, -1.0f, 0.0f, 0.0f, 1.0f, // bottom-left
            1.0f, 1.0f, -1.0f, 0.0f, 0.0f, -1.0f, 1.0f, 1.0f, 1.0f,   // top-right
            1.0f, -1.0f, -1.0f, 0.0f, 0.0f, -1.0f, 1.0f, 0.0f, 1.0f,  // bottom-right
            1.0f, 1.0f, -1.0f, 0.0f, 0.0f, -1.0f, 1.0f, 1.0f, 1.0f,   // top-right
            -1.0f, -1.0f, -1.0f, 0.0f, 0.0f, -1.0f, 0.0f, 0.0f, 1.0f, // bottom-left
            -1.0f, 1.0f, -1.0f, 0.0f, 0.0f, -1.0f, 0.0f, 1.0f, 1.0f,  // top-left
            // front face
            -1.0f, -1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 0.0f, 0.0f, 1.0f, // bottom-left
            1.0f, -1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 1.0f, 0.0f, 1.0f,  // bottom-right
            1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f, 1.0f,   // top-right
            1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f, 1.0f,   // top-right
            -1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 0.0f, 1.0f, 1.0f,  // top-left
            -1.0f, -1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 0.0f, 0.0f, 1.0f, // bottom-left
            // left face
            -1.0f, 1.0f, 1.0f, -1.0f, 0.0f, 0.0f, 1.0f, 0.0f, 1.0f,   // top-right
            -1.0f, 1.0f, -1.0f, -1.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f,  // top-left
            -1.0f, -1.0f, -1.0f, -1.0f, 0.0f, 0.0f, 0.0f, 1.0f, 1.0f, // bottom-left
            -1.0f, -1.0f, -1.0f, -1.0f, 0.0f, 0.0f, 0.0f, 1.0f, 1.0f, // bottom-left
            -1.0f, -1.0f, 1.0f, -1.0f, 0.0f, 0.0f, 0.0f, 0.0f, 1.0f,  // bottom-right
            -1.0f, 1.0f, 1.0f, -1.0f, 0.0f, 0.0f, 1.0f, 0.0f, 1.0f,   // top-right
                                                                      // right face
            1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 0.0f, 1.0f,     // top-left
            1.0f, -1.0f, -1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 1.0f, 1.0f,   // bottom-right
            1.0f, 1.0f, -1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f,    // top-right
            1.0f, -1.0f, -1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 1.0f, 1.0f,   // bottom-right
            1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 0.0f, 1.0f,     // top-left
            1.0f, -1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 0.0f, 1.0f,    // bottom-left
            // bottom face
            -1.0f, -1.0f, -1.0f, 0.0f, -1.0f, 0.0f, 0.0f, 1.0f, 1.0f, // top-right
            1.0f, -1.0f, -1.0f, 0.0f, -1.0f, 0.0f, 1.0f, 1.0f, 1.0f,  // top-left
            1.0f, -1.0f, 1.0f, 0.0f, -1.0f, 0.0f, 1.0f, 0.0f, 1.0f,   // bottom-left
            1.0f, -1.0f, 1.0f, 0.0f, -1.0f, 0.0f, 1.0f, 0.0f, 1.0f,   // bottom-left
            -1.0f, -1.0f, 1.0f, 0.0f, -1.0f, 0.0f, 0.0f, 0.0f, 1.0f,  // bottom-right
            -1.0f, -1.0f, -1.0f, 0.0f, -1.0f, 0.0f, 0.0f, 1.0f, 1.0f, // top-right
            // top face
            -1.0f, 1.0f, -1.0f, 0.0f, 1.0f, 0.0f, 0.0f, 1.0f, 1.0f, // top-left
            1.0f, 1.0f, 1.0f, 0.0f, 1.0f, 0.0f, 1.0f, 0.0f, 1.0f,   // bottom-right
            1.0f, 1.0f, -1.0f, 0.0f, 1.0f, 0.0f, 1.0f, 1.0f, 1.0f,  // top-right
            1.0f, 1.0f, 1.0f, 0.0f, 1.0f, 0.0f, 1.0f, 0.0f, 1.0f,   // bottom-right
            -1.0f, 1.0f, -1.0f, 0.0f, 1.0f, 0.0f, 0.0f, 1.0f, 1.0f, // top-left
            -1.0f, 1.0f, 1.0f, 0.0f, 1.0f, 0.0f, 0.0f, 0.0f, 1.0f   // bottom-left
        };
        glGenVertexArrays(1, &cubeVAO);
        glGenBuffers(1, &cubeVBO);
        // fill buffer
        glBindBuffer(GL_ARRAY_BUFFER, cubeVBO);
        glBufferData(GL_ARRAY_BUFFER, sizeof(vertices), vertices, GL_STATIC_DRAW);
        // link vertex attributes
        glBindVertexArray(cubeVAO);
        glEnableVertexAttribArray(0);
        glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, 9 * sizeof(float), (void *)0);
        glEnableVertexAttribArray(1);
        glVertexAttribPointer(1, 3, GL_FLOAT, GL_FALSE, 9 * sizeof(float), (void *)(3 * sizeof(float)));
        glEnableVertexAttribArray(2);
        glVertexAttribPointer(2, 3, GL_FLOAT, GL_FALSE, 9 * sizeof(float), (void *)(6 * sizeof(float)));
        glBindBuffer(GL_ARRAY_BUFFER, 0);
        glBindVertexArray(0);
    }
    // render Cube
    glBindVertexArray(cubeVAO);
    glEnable(GL_SHADING_RATE_IMAGE_NV);
    createTexture(fov_texture);
    uploadFoveationDataToTexture(fov_texture);
    glBindShadingRateImageNV(fov_texture);
    glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
    glDrawArrays(GL_TRIANGLES, 0, 36);
    glBindVertexArray(0);
    GLenum err;
    while ((err = glGetError()) != GL_NO_ERROR)
    {
        std::cerr << "OpenGL Error: " << err << std::endl;
    }
}

void createTexture(GLuint &glid)
{
    if (glid)
    {
        glDeleteTextures(1, &glid);
    }
    glGenTextures(1, &glid);
}

Vertex shader:

#version 460 core

#extension GL_NV_primitive_shading_rate: require
#extension GL_NV_viewport_array2 : enable

in layout(location=0) vec3 aPos;
in layout(location=1) vec3 aNormal;
in layout(location=2) vec3 aColor;

uniform mat4 model;
uniform mat4 view;
uniform mat4 projection;

out vec3 fragPos;
out vec3 color;

void main() {
    fragPos = vec3(model * vec4(aPos, 1.0));

    gl_Position = projection * view * model * vec4(aPos, 1.0);
    color = aColor;
    gl_ShadingRateNV = 0;

}

Fragment shader:

#version 460
#extension GL_NV_shading_rate_image : enable
#extension GL_NV_primitive_shading_rate : enable

out vec4 FragColor;

in vec3 color;

void main() {
    int maxCoarse = max(gl_FragmentSizeNV.x, gl_FragmentSizeNV.y);

    if (maxCoarse == 1) {
        FragColor = vec4(1,0,0,1);
    }
    else if (maxCoarse == 2) {
        FragColor = vec4(1,1,0,1);
    }
    else if (maxCoarse == 4) {
        FragColor = vec4(0,1,0,1);
    }
    else {
        FragColor = vec4(1,1,1,1);
    }
    //FragColor = vec4(color, 1.0);
}

[Dark_Photon]

@zMendes

I’m trying to implement a project using Nvidia’s VRS OpenGL extension. …

Howver, when trying to implement VRS in my own project I can’t seem to get it working properly.

main.cpp:

    ...
    glEnable(GL_SHADING_RATE_IMAGE_NV);
    glEnable(GL_SHADING_RATE_IMAGE_PER_PRIMITIVE_NV);
    ...

If I were you, I’d get it working without the 2nd enable first. That is, with just GL_SHADING_RATE_IMAGE_NV enabled. Then just call glShadingRateImagePaletteNV() with a constant rate enum. Make sure this basic form works.

Then once you get that working, go full-out and try the per-primitive special sauce.

It could be that you have something really basic not setup which keeps even the first, simple VRS test case from working. If so, this’ll point that out.

Note that you can take advantage of Issue 5 to avoid even having to provide a shading rate image.

(5) Does a shading rate image need to be bound in order to use the shading
    rate feature?

  RESOLVED:  No.  The behavior where there is no texture bound when
  SHADING_RATE_IMAGE_NV is enabled is explicitly defined to behave as if a
  lookup was performed and returned zero.  If an application wants to use
  a constant rate other than SHADING_RATE_1_INVOCATION_PER_PIXEL_NV, it
  can enable SHADING_RATE_IMAGE_NV, ensure no image is bound, and define
  the entries for index zero in the relevant palette(s) to contain the
  desired shading rate.  This technique can be used to emulate 16x
  multisampling on implementations that don't support it by binding larger
  4x multisample textures to the framebuffer and then setting a shading
  rate of SHADING_RATE_1_INVOCATION_PER_2X2_PIXELS_NV.

[zMendes]

Thanks for the help! After simplifying my implementation, I finally got it to work! I realized I had missed the part in the documentation stating that VRS cannot be applied directly to on-screen rendering, I needed to create a framebuffer object first. Now, VRS is being applied correctly.

Thanks again!

[Dark_Photon]

That’s great! Glad you got it figured out.