I have read all about GL_POLYGON_SMOOTH, and it seems simple. I am rendering some simple projected shadows to an FBO, so I thought this type of antialiasing would be perfect.
The important parts of my code are this:
glHint GL_POLYGON_SMOOTH_HINT, GL_NICEST
// Now draw stuff
But it looks like this (using GL_NEAREST for the filter so I can see it better):
If you want to get soft shadows GL_POLYGON_SMOOTH won’t help you.
Are you sure that your FBO has an alpha channel ?
Trenki, for oldskool projected shadows it should work.
The FBO is format GL_RGBA.
I am guessing this is just an old techique that isn’t even supported on current drivers.
@ZbuffeR: I think the shadow in the image from the link you posted is just a normal planar shadow which does NOT use a texture at all! GL_POLYGON_SMOOTH won’t make the shadow image in the texture soft!
PCF shadows is what you need.
@Trenki: my understanding of OP technique is to render shadow caster object with polygon smooth to a an FBO (instead of rendering to main framebuffer using stencil), and then using the FBO as darken texture.
@Leadwerks : speaking about basic tricks, maybe you can simply replace polygon smooth by multisampling if supported, or render to larger FBO and smooth it yourself. Whether these hacks are better than proper PCF is another question …
This is for the old renderer, so it isn’t using shadow maps at all. The new one already uses PCF.
Multisampling in the FBO is probably the best bet.
Multisampling in an FBO is not supported on all cards (e.g. ATI)! And Multisampling alone won’t make the shadows less blocky. For that you would have to increase the size of the rendered to texture.
I just tried, I got GL_POLYGON_SMOOTH working perfectly on FBO.
Started with this sample app :
http://www.reallyslick.com/src/fbo.tar.gz (thx to mogumbo)
Then added your settings for the part rendered to FBO.
It did not work with glBlendFunc GL_SRC_ALPHA_SATURATE,GL_ONE, dunno why.
A simple glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA); did the trick instead.
An RGBA FBO is not needed, RGB is enough (only src component sampled).
Tested on Ubuntu Edgy with quite oldish drivers:
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce 6800 LE/AGP/SSE2/3DNOW!
OpenGL version string: 2.0.2 NVIDIA 87.76
Oh, come on
Did you try to simply replace the blendfunc in your own code ?
True but http://www.opengl.org/documentation/specs/man_pages/hardcopy/GL/html/gl/blendfunc.html
search for GL_POLYGON_SMOOTH.
So I guess the API reference is wrong … or I really failed to get dest alpha on my FBO.
Anyway I hope Leadwerks’ software is better now