Is there a way to get a pixel data mid-rendering?

r-jaoui · February 24, 2021, 4:16pm

Hello !

I’m kind of new to OpenGL, and for a certain program, I need to be able to manipulate (gather, modify and update) a pixel array. The issue is that I’ve already written loading and updating functions, but as far as I understand, they only work on the last rasterized framebuffer, and therefore can’t be gathered during rendering in the way that I want to. Here are the loading and updating functions I’ve written (of course if anything seems wrong feel free to correct me, and note that this is JOGL, so written in Java) :

public void loadPixels() {
	IntBuffer buffer = IntBuffer.allocate(width * height);
	gl.glReadPixels(0, 0, width, height, GL2.GL_BGRA, GL2.GL_UNSIGNED_BYTE, buffer);
	pixels = buffer.array();
}

public void updatePixels() {
	IntBuffer bufferedPixels = IntBuffer.allocate(3*width * height);
	bufferedPixels.put(pixels);
	gl.glDrawPixels(width, height, GL2.GL_BGRA, GL2.GL_UNSIGNED_BYTE, bufferedPixels);
}

An example of the way I want to use them is for example rendering something (a few triangles…), then gathering all pixels WITH ONLY these shapes rasterized, and then being able to draw something else on top (let’s say a loading bar) whose render doesn’t affect the pixels that are gathered.

I know that the glReadPixels and glDrawPixels functions only work on the last rasterized framebuffer, and I’d guess that without rasterization, there is no way to gather any pixel information, so is there a way to force rasterization mid-rendering ? If not, is there a different way to gather/update pixel data mid rendering (even at the cost of another rendering step)

Thanks !

Alfonse_Reinheart · February 24, 2021, 4:37pm

I don’t understand what you mean by “last rasterized framebuffer”. glReadPixels reads from whatever the current glReadBuffer is. If you’re rendering to the back buffer, and you have set the glReadBuffer to GL_BACK, then glReadPixels will read from that buffer.

r-jaoui · February 24, 2021, 4:48pm

Well let’s say that in the display function of the GLEventListener I call a glClear and then draw a rectangle with glBegin / glEnd, then call glReadPixels with the correct buffer and finally draw a triangle on top, is the rectangle going to be present and not the triangle ? I’ll try writing an example program to test it out and if it doesn’t I’ll post the code here so you can see what I’m trying to do.

For now though, glReadBuffer reads pixel data from the last rendered frame, not from the current one at its current state, and the results are weird (see this following image, where I gather pixel data from a square region (denoted by the black square) and draw it in the center. The square is rendered AFTER the call to glReadPixels and is still gathered by it, and when the square overlaps with the center, I get copies of the square (black line in the bottom of the zoomed section is the top of the rectangle, if it makes sense) :

r-jaoui · February 24, 2021, 4:49pm

Here is the second image :

r-jaoui · February 24, 2021, 4:50pm

And here is what I’d like (no rectangle visible in zoomed square, and the image is the only thing being loaded basically) :

Alfonse_Reinheart · February 24, 2021, 5:01pm

OpenGL operates as if it were synchronous. So each command acts as if only the previously issued commands have executed completely. And later commands cannot affect the execution of previous commands.

So if you actually do the things in the way you describe, the “triangle” should not appear in the pixel data.

r-jaoui · February 24, 2021, 5:18pm

Hm alright I’ll try it, perhaps the problem was just with this specific code, I’ll try it out.

Thanks for your help

r-jaoui · February 24, 2021, 5:41pm

Hm so I tried it, doesn’t seem to work but perhaps something is wrong with my code :

Main.java

public class Main {
	public static void main(String[] args) {
		@SuppressWarnings("unused")
		Frame frame = new Frame(1280, 720);
	}
}

Frame.java

import java.awt.IllegalComponentStateException;
import java.awt.MouseInfo;

import javax.swing.JFrame;

import com.jogamp.opengl.GL2;
import com.jogamp.opengl.GLCapabilities;
import com.jogamp.opengl.GLProfile;
import com.jogamp.opengl.awt.GLJPanel;

public class Frame extends JFrame{
	
	public static int mouseX = 0;
	public static int mouseY = 0;
	
	public static GLProfile PROFILE;
	public static GLCapabilities CAPABILITIES;
	public static GLJPanel panel;
	public static PanelListener listener;
	public static boolean stop = false;
	
	//SERIALIZATION ID
	private static final long serialVersionUID = 1L;
	
	private boolean glLoaded = false;
	
	public Frame(int width, int height) {
		super();

		PROFILE = GLProfile.get(GLProfile.GL2);
		CAPABILITIES = new GLCapabilities(PROFILE);
		
		CAPABILITIES.setSampleBuffers(true);
		CAPABILITIES.setNumSamples(4);
		CAPABILITIES.setStencilBits(8);
		
		panel = new GLJPanel(CAPABILITIES);
		listener = new PanelListener(panel);
		panel.addGLEventListener(listener);
		
		this.setContentPane(panel);
		
		this.setVisible(true);
		this.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		this.setSize(width, height);
		
		run();
	}

	private void updateGraphics() {
		if(panel.getGL() != null) {
			if(panel.getGL().getGL2() != null) {
				
				GL2 gl = panel.getGL().getGL2();
				
				glLoaded = true;
				
				gl.glClearColor(
						0.95f,
						0.95f,
						0.95f,
						1.0f);
				
				gl.glClear(GL2.GL_COLOR_BUFFER_BIT | GL2.GL_DEPTH_BUFFER_BIT | GL2.GL_STENCIL_BUFFER_BIT);
			}
		}
	}
	
	private void waitForGlLoad() {
		while(!glLoaded) {
			
			updateGraphics();
			
			try {Thread.sleep(1);}
			catch (InterruptedException e) {e.printStackTrace();}
		}
	}
	
	public void run() {
		while(!stop) {
			waitForGlLoad();
			
			try {
				mouseX = MouseInfo.getPointerInfo().getLocation().x - panel.getLocationOnScreen().x;
				mouseY = MouseInfo.getPointerInfo().getLocation().y - panel.getLocationOnScreen().y;
			}
			catch(IllegalComponentStateException e) {
				System.out.println("Component data isn't available : window was closed");
				break;
			}
			
			panel.repaint();
			
			try {Thread.sleep(1);}
			catch (InterruptedException e) {e.printStackTrace();}
		}
	}
}

PanelListener.java

import java.awt.Point;
import java.awt.image.BufferedImage;
import java.awt.image.ColorModel;
import java.awt.image.DataBufferInt;
import java.awt.image.Raster;
import java.io.File;
import java.io.IOException;
import java.nio.IntBuffer;

import javax.imageio.ImageIO;

import com.jogamp.opengl.GL;
import com.jogamp.opengl.GL2;
import com.jogamp.opengl.GLAutoDrawable;
import com.jogamp.opengl.GLEventListener;
import com.jogamp.opengl.awt.GLJPanel;

public class PanelListener implements GLEventListener{
	
	GLJPanel panel;
	GL2 gl;
	
	public PanelListener(GLJPanel panel) {
		super();
		this.panel = panel;
	}
	
	@Override
	public void display(GLAutoDrawable displayable) {
		gl.glDrawBuffer(GL.GL_BACK);
		
			gl.glClearColor(
					1f,
					1f,
					1f,
					1.0f);

			gl.glClear(GL2.GL_COLOR_BUFFER_BIT | GL2.GL_DEPTH_BUFFER_BIT | GL2.GL_STENCIL_BUFFER_BIT);
			
			gl.glColor3f(1.0f, 0.0f, 0.0f);
			gl.glBegin(GL2.GL_QUADS);
			gl.glVertex2f(-0.4f, -0.4f);
			gl.glVertex2f( 0.4f, -0.4f);
			gl.glVertex2f( 0.4f,  0.4f);
			gl.glVertex2f(-0.4f,  0.4f);
			gl.glEnd();
			
			gl.glReadBuffer(GL.GL_BACK);
			
			IntBuffer pixels = IntBuffer.allocate(panel.getWidth() * panel.getHeight());
			gl.glReadPixels(0, 0, panel.getWidth(), panel.getHeight(), GL.GL_BGRA, GL.GL_UNSIGNED_BYTE, pixels);
			saveImage(pixels);
			
			gl.glColor3f(0.0f, 0.0f, 1.0f);
			gl.glBegin(GL2.GL_TRIANGLES);
			gl.glVertex2f( 0.45f,  0.0f);
			gl.glVertex2f(-0.5f ,  0.5f);
			gl.glVertex2f(-0.5f , -0.5f);
			gl.glEnd();
	}
	
	public void saveImage(IntBuffer pixels) {
		//GENERATING THE RASTER
		DataBufferInt bufferedPixels = new DataBufferInt(panel.getWidth() * panel.getHeight());
		int[] data = pixels.array();
		for(int x = 0; x < panel.getWidth(); x++) {
			for(int y = 0; y < panel.getHeight(); y++) {
				bufferedPixels.setElem(x + y * panel.getWidth(), data[x + (panel.getHeight() - 1 - y) * panel.getWidth()]);
			}
		}
		pixels.clear();
		Raster r = Raster.createRaster(ColorModel.getRGBdefault().createCompatibleSampleModel(panel.getWidth(), panel.getHeight()), bufferedPixels, new Point(0, 0));
		
		//CREATING THE IMAGE
		BufferedImage img = new BufferedImage(panel.getWidth(), panel.getHeight(), BufferedImage.TYPE_INT_ARGB);
		img.setData(r);
		img.flush();
		try {
			ImageIO.write(img, "png", new File("text_image.png"));
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
	
	@Override
	public void dispose(GLAutoDrawable displayable) {
		
	}

	@Override
	public void init(GLAutoDrawable displayable) {
		gl = displayable.getGL().getGL2();
		
        gl.glEnable(GL2.GL_LINE_SMOOTH);
        gl.glEnable(GL2.GL_POINT_SMOOTH);
        gl.glEnable(GL2.GL_SMOOTH);
        
		gl.glHint(GL2.GL_LINE_SMOOTH_HINT, GL2.GL_FASTEST);
		gl.glHint(GL2.GL_POINT_SMOOTH_HINT, GL2.GL_NICEST);
		gl.glShadeModel(GL2.GL_SMOOTH);
		
        gl.glEnable(GL2.GL_BLEND);
        gl.glBlendFuncSeparate(GL2.GL_SRC_ALPHA, GL2.GL_ONE_MINUS_SRC_ALPHA, GL2.GL_ONE, GL2.GL_ONE);
        
		gl.glClearColor(0.3f, 0.3f, 0.3f, 1f);
		gl.glClear(GL2.GL_COLOR_BUFFER_BIT | GL.GL_DEPTH_BUFFER_BIT | GL.GL_STENCIL_BUFFER_BIT);
	}

	@Override
	public void reshape(GLAutoDrawable displayable, int x, int y, int w, int h) {
		panel.repaint();
	}
}

And here is the saved image :

Dark_Photon · February 24, 2021, 6:21pm

Yes. If you really need to save off or access the intermediate rasterization result, you can split your rendering up into separate phases. Between phases, you can ask the driver to save off the rasterized content into another texture or renderbuffer, or rebind the rendered result (if rendered to a texture) to a shader so you can read the rasterized data into the shader, etc. There are also some GL extensions that will let you read from the render target you are in the process of rasterizing to (with limitations).

And you’re targeting OpenGL here, right? Not OpenGL ES/mobile?

r-jaoui · February 24, 2021, 8:35pm

Yes, OpenGL with JOGL in Java.

TBH I’ve seen the notion of RenderBuffer come around but I don’t really know how exactly I’d implement that and how this could force rasterization

I really hope this is possible though. If you have any amount of code or documentation that could help me I’d really appreciate it

GClements · February 24, 2021, 9:56pm

See the wiki page for FBOs.

A framebuffer object (FBO) allows you to render to off-screen surfaces (renderbuffers or textures). This allows you to have a format which differs from the physical screen format or dimenions which exceed those of the screen, and also allows you to preserve the rendered image indefinitely (the contents of the default framebuffer’s back buffer become undefined after a buffer swap, while the contents of its front buffer can be overwritten by the windowing system at any point).

But so far as I can tell, your existing code ought to work. The glReadPixels call should obtain the framebuffer contents after the execution of the glClear and the first glBegin/glEnd block but before execution of the second block. In the absence of errors (specifically, the glClear failing), I can’t see how the blue triangle can end up in the data returned by glReadPixels. Have you checked for errors?

r-jaoui · February 25, 2021, 7:11am

Well I tried changing the color of the glClear and it definetly seems to be working :

Thanks for the ressource on FBOs, I’ll check it out today.
It’s kind of sad, I keep getting told that this should work :’(
If anything seems wrong in my code (there should be everything here) please tell me :3

r-jaoui · February 25, 2021, 7:17am

Oh and by the way, my hunch is that the image rendered is technically the last frame rendered to the screen, which would explain the presence of the triangle.

This was especially clear in the last program where the zoomed out render would lag out 1 frame behind the main screen.

r-jaoui · February 25, 2021, 2:48pm

Well, I also tried exiting this program :

after the first rendered frame and as I expected the saved image is completely black…

r-jaoui · February 25, 2021, 4:03pm

Alriiiight, after 2 long days of trial and error I managed to finally locate the line that made it impossible to read the color buffer. This line is the following :

For anyone having this issue, make sure to NOT have this line active (or to have the line CAPABILITIES.setSampleBuffers(false)). Basically, this disables the allocation of sample buffers for full-scene antialiasing, which for some reason makes buffer reading impossible.

Alfonse_Reinheart · February 25, 2021, 4:45pm

You shouldn’t describe this that way. You seem to be speaking as if the behavior you’re seeing is correct, and that the bug is in your code.

It isn’t. The OpenGL driver is buggy; what you’re doing is working around the driver bug. The presence of multiple samples in the buffer should have no effect on whether you can properly read from it. So if it does, that’s a driver problem, not a user problem.

You still need the workaround, but you shouldn’t frame it as though it’s a code problem.

r-jaoui · February 25, 2021, 4:50pm

Oh… I just thought it was impossible to read the buffer if MSAA was enabled (which actually annoys me since I need both at the same time…).

So you don’t think that there’s a workaround that would allow me to get pixel data AND antialiased rendering (let’s say without multisampling but using post-processing AA) ?

r-jaoui · February 25, 2021, 5:41pm

Alright so another twist in this story : disabling MSAA makes Buffer Reading possible, but Anti-Aliasing isn’t activated.

I’ve figured out (just by stambling onto it) that activating P-Buffering (for off-screen rendering ?) allows to both use Anti-Aliasing and Buffer Reading (does it make sense ?)

PROFILE = GLProfile.get(GLProfile.GL2);
CAPABILITIES = new GLCapabilities(PROFILE);

CAPABILITIES.setSampleBuffers(true);
CAPABILITIES.setNumSamples(8);
CAPABILITIES.setStencilBits(8);
CAPABILITIES.setPBuffer(true);

So both anti-aliasing and buffer reading seem to be working flawlessly, but for one I don’t understand why, and secondly, I hope that this won’t cause anything more to go wrong -_-

Alfonse_Reinheart · February 25, 2021, 5:55pm

It’s a driver bug; “why” is not a thing you’re supposed to understand. You report it to the IHV responsible for your driver, you implement a workaround as best you can, and you move on.

I mean, it’s good to do some research to find out the specific things that cause the bug to manifest. But you shouldn’t do so with the expectation that it will make sense on some level. It’s a bug in code that you don’t own and have no control over.

Treat it like a pothole: you point out that it’s there, maybe put a warning around it, and tell the people responsible for fixing the street that it exists.

GClements · February 25, 2021, 6:28pm

Or maybe it’s something to do with JOGL. From what I’ve managed to figure out, GLJPanel uses an off-screen surface (FBO or PBuffer) then copies the data to the window using glReadPixels and Java’s own rendering APIs. If it’s using a FBO, it’s possible that GL_READ_FRAMEBUFFER refers to a different FBO than GL_DRAW_FRAMEBUFFER. If it uses two FBOs and swaps them, the read framebuffer might be the FBO holding the previously-rendered frame, which would explain this behaviour.

Also, GL_BACK isn’t valid for a FBO; you have to use one of the GL_COLOR_ATTACHMENTn targets. Note that this should result in a GL_INVALID_OPERATION error.

In future, it may be better to suggest that questions relating to third-party language bindings are taken to a more appropriate forum.