Trimming or cutting windows (holes) in polygons

Im coding a project right now which basically will become a detailed 3D model of a building.

My question is, when it comes to texturing my building, what is the best way to map the wall textures around the doors and windows? I understand how to make translucent obejcts etc, but from what I have read, there doesnt seem to be way to draw, say a rectangle with a rectangular hole in it, ie a window. I know opengl doesnt support the ‘object-intersection’ features ive read from other APIs that can be used to cut holes in objects.

So will I have to draw a wall as many, smaller rectangles and map the textures onto each one? There must be an easier way.

Thanks ahead of time

Better put: can you/how would I map a texture with a hole in it?

I am trying to reformulate your questions :

Q) how do I model complex geometry ?
A) OpenGL is a drawing API, not a general purpose modeling tool. Use Blender or OpenCascade (good for boolean ops) or whatever instead.

Q) how do I generate texture coordinates around complex objects ?
A) By hand it can be way too complex. Use an appropriate tool for this. Blender has great unwrap tools: automatic, semi-auto, manual. Then export your object as .obj for example, as it is easy to parse and provide texture coordinates.

Q) how to make a hole in a textured face ?
A) Enable alpha testing and a RGBA texture. A=0 for the hole, A=255 for the opaque parts.

No Im doin it myself, all in C++. This is my mistake, I was asking for help and I didn’t really know what questions to ask yet because I was frustrated. My project is more about a flexible interface for the user to view an enviroment. But I noticed the tess functions would be very handy in my project, which is mostly drawing a detailed building. So I wanted to use the tess functions to draw my walls with all the doors and windows cut out, so I could map my textures onto the entire wall, etc.

But the problem is now I really can’t figure out why I cant make the simplest of polygons with the tessellation functions. I havent found many topics online specifically about this.

I’ve taken most of this code from the red book and I’ve been toying with it in several ways, but if you can see anything obviously wrong, please elaborate.

This is just supposed to draw a big wall, with 1 small hole in it, but nothing is drawn at all. I draw an outline around where the wall should be, and its fine, so I know Im not using the tess functions properly.


GLdouble rec1[3] = {0.0, 0.0, 0.0};
GLdouble rec2[3] = {0.0, 65.0, 0.0};
GLdouble rec3[3] = {0.0, 65.0, -250.0};
GLdouble rec4[3] = {0.0, 0.0, -250.0};

GLdouble win1[3] = {0.0, 5.0, -5.0};
GLdouble win2[3] = {0.0, 25.0, -5.0};
GLdouble win3[3] = {0.0, 25.0, -25.0};
GLdouble win4[3] = {0.0, 5.0, -25.0};

GLvoid tcbBegin(GLenum type)
{
	glBegin(type);
}

GLvoid tcbVertex(GLvoid *vertex)
{
	const GLdouble* pointer;
	pointer = (GLdouble*)vertex;
	glColor3dv(pointer+3);
	glVertex3dv((GLdouble*)vertex);
}

GLvoid tcbEnd()
{
	glEnd();
}

GLvoid combineCallback(GLdouble coords[3], GLdouble *vertex_data[4], GLfloat weight[4], GLdouble **dataOut )
{
	GLdouble *vertex;
	int i;
	vertex = (GLdouble *) malloc(6 * sizeof(GLdouble));
	vertex[0] = coords[0];
	vertex[1] = coords[1];
	vertex[2] = coords[2];
	for (i = 3; i < 7; i++)
		vertex[i] = weight[0] * vertex_data[0][i] 
	+ weight[1] * vertex_data[1][i]
	+ weight[2] * vertex_data[2][i] 
	+ weight[3] * vertex_data[3][i];
	*dataOut = vertex;
}

GLvoid errorCallback(GLenum errorCode)
{
	const GLubyte *estring;
	estring = gluErrorString(errorCode);
	fprintf(stderr, "Tessellation Error: %s
", estring);
	exit(0);
}

void init()
{
	glClearColor(0.0, 0.0, 0.0, 0.0);
	
	glLoadIdentity();
	glMatrixMode(GL_PROJECTION);
	gluPerspective(fov_degrees, xy_ratio, near_clip, far_clip);
	gluLookAt(eye_x, eye_y, eye_z, focus_x, focus_y, focus_z, 0.0, 1.0, 0.0);
	glMatrixMode(GL_MODELVIEW);
	glShadeModel(GL_FLAT);

	wall1 = gluNewTess();
	gluTessCallback(wall1, GLU_TESS_VERTEX, (GLvoid)glVertex3dv);
	gluTessCallback(wall1, GLU_TESS_BEGIN, (GLvoid)tcbBegin);
	gluTessCallback(wall1, GLU_TESS_END, (GLvoid)tcbEnd);
	gluTessCallback(wall1, GLU_TESS_COMBINE, (GLvoid)combineCallback);
	gluTessCallback(wall1, GLU_TESS_ERROR, (GLvoid)errorCallback);
	
	glShadeModel(GL_FLAT);
}

void window()
{
	glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

	gluTessProperty(wall1,GLU_TESS_WINDING_RULE, GLU_TESS_WINDING_ODD);
	gluTessBeginPolygon(wall1, NULL);
		gluTessBeginContour(wall1);
			gluTessVertex(wall1, rec1, rec1);
			gluTessVertex(wall1, rec2, rec2);
			gluTessVertex(wall1, rec3, rec3);
			gluTessVertex(wall1, rec4, rec4);
		gluTessEndContour(wall1);
		gluTessBeginContour(wall1);
			gluTessVertex(wall1, win1, win1);
			gluTessVertex(wall1, win2, win2);
			gluTessVertex(wall1, win3, win3);
			gluTessVertex(wall1, win4, win4);
		gluTessEndContour(wall1);
	gluTessEndPolygon(wall1);

	glFlush();
}
 

I suspected that the combineCallback function was not appropriate for this, but I also loaded the red book’s “tess.c” verbatum, modified slightly to compile, and that also displayed nothing. It was supposed to display a quad with a triangular hole and a contour 5 pointed star.

Anyway, see anything obvious? Thanks ahead of time

oh man a reply, even if it was “I have no idea” would be fantastic!

You wanted to visually cut-out a part of a wall, and ZbuffeR told you what you need. “Tesselating” a flat surface is always useless (except for adding vertex-lightmapping). Tesselation is for adding detail, so that round surfaces (like a human head, or a ball) don’t look like a pyramid when viewed up close.
If you have used any 3D modeler, you’d have seen that cutting-out parts of meshes is not a trivial thing and there are multiple possible results.
It is much easier to use shaders, where you put the to-be-cut-out rectangles as uniforms, and in your pixel-shader compute whether the current fragment is visible or inside of a “window/door” rectangle.
Or you could edit the alpha-channel of textures at runtime and use the alpha-test.

yes Ive been looking into what he said. But I had NO idea tesselation was inappropriate for flat polygons, I haven’t read that ocne anywhere. In fact the red book’s examples are using the tessellation on flat shapes to such as the square with a triangle cut out of it. I just dont see why that cant be translated to work in my scenario.

And I’m refering to tessellation because thats what the functions are called. Its not ‘cutting’ a hole out of anything, theres no ‘cut’ action. Its supposed to draw two polygons, 1 inside the other and the tess functions are supposed to know I want the area of the larger polygon filled, minus the area of the smaller polygon within it.

Is that not the exact description of these functions in the red book or am I going crazy??

I believe Ilian Dinev is defining tessellation in terms of subdivision, much like PN triangles (ATi) and RT patches (Nvidia). The gluTessellator you are referring to, however, is only useful for flat surfaces.

Here’s a part from the spec:

coords give the coordinates of the vertex in 3-space. For useful results, all vertices should lie in some plane, since the vertices are projected onto a plane before tessellation. vertex data is a pointer to a user-defined vertex structure, which typically contains other vertex information such as color, texture coordinates, normal, etc. It is used to refer to the vertex during rendering.

Like it says, the vertices are projected onto a plane before rendering. Since your vertices are in the Y-Z plane, it will project the vertices on the Y-Z plane looking from either the positive or negative X-axis. Your winding rule will be different whether it decides to project from the positive or the negative axis. So be sure to check that your winding rule is OK. You can control it with the tess normal:

To supply the polygon normal call:
gluTessNormal(…)
All input data will be projected into a plane perpendicular to the normal before tessellation and all output triangles will be oriented CCW with respect to the normal (CW orientation can be obtained by reversing the sign of the supplied normal). For example, if you know that all polygons lie in the x-y plane, call gluTessNormal(tess,0.0,0.0,1.0) before rendering any polygons.

In your case it should be
gluTessNormal(tess,+1.0,0.0,0.0);
or
gluTessNormal(tess,-1.0,0.0,0.0);

N.

Yeah, my terminology was wrong, sry ^^". Been reading nVidia docs too much, and they’ve used “tessellation” as curved-surface-subdivision.

On the topic, things can be done easily using the following way, imho:

First, your wall is a quad in 2D (I suggest you internally keep walls as 2D with some rotation for visualization) like this:

Also, each wall has a struct like this:
RECT WallRect;
RECT* MyRects;
int NumRects=0;
RECT* VisQuads;
int NumVisQuads=1;

Initially, the above data is:
WallRect = {0,0,500,200};
MyRects = NULL;
NumRects=0;
VisQuads[0]={0,0,500,200};
NumVisQuads=1;

When adding a door/window with coordinates in some RECT, we add it to the MyRects[] and increase NumRects. Then, we call a GenerateVisQuads() procedure, with local arrays:
int MidpointsX[]; int NumMPX=0;
int MidpointsY[]; int NumMPY=0;

From each rectangle in MyRects[], we add the X and Y midpoints in those two arrays, and sort them.

We empty the VisQuads[] array, and set NumVisQuads=0;
When NumRects==1, we’ll have 9 resulting cells (quads). For each cell, we compute the coordinates into “RECT CurCoords;”. And try to hit-test this CurCoords with all MyRects[]. If it doesn’t overlap any of the MyRects[], then we add this CurCoords to VisQuads[], and increment NumVisQuads. For any vertex-attributes the CurCoords quad needs (UV coordinates, color), we simply interpolate.

We could post-process VisQuads[] to reduce the number of necessary quads and vertices, and even convert to triangles.

And finally, draw VisQuads[] in OpenGL. (don’t forget the UV coords, too)

If you’ll have only one vertex attribute (the UV coords), then make a struct:
typedef struct{
RECT rect;
float U0,V0;
float U1,V1;
}TexRECT;

and change the types of WallRect[], VisQuads[] and CurCoords from RECT to TexRECT.


Here’s how adding a second “door” will visualize:

If you add doors/windows that overlap in 2D or overlap in 1D in either X or Y, the above algorithm will automatically visualize correctly. But with a bit more subdivided quads.

Yeah the problem with the subdivided quads is texture mapping. I can apply the texture coordinates in the gluTess process too but I still have look into that.

The problem I have is resulting from the changes from the original tess.c to c++ code, I think.

After debugging, I saw that gluTessVertex is not calling my vertex function so that is why the primitives are never being drawn. Can anyone tell me how I can do this?

 
GLdouble rec1[3] = {0.0, 0.0, 0.0};
GLdouble rec2[3] = {250.0, 0.0, 0.0};
GLdouble rec3[3] = {250.0, 65.0, 0.0};
GLdouble rec4[3] = {0.0, 65.0, 0.0};

GLdouble win1[3] = {5.0, 5.0, 0.0};
GLdouble win2[3] = {25.0, 5.0, 0.0};
GLdouble win3[3] = {25.0, 25.0, 0.0};
GLdouble win4[3] = {5.0, 25.0, 0};

GLdouble* outer[4] = {rec1, rec2, rec3, rec4};
GLdouble* inner[4] = {win1, win2, win3, win4};

GLvoid tcbBegin(GLenum type)
{
	glBegin(type);
}

GLvoid tcbVertex(GLvoid *vertex)
{
	glVertex3dv((GLdouble*)vertex);
}

GLvoid tcbEnd()
{
	glEnd();
}

GLvoid combineCallback(GLdouble coords[3], GLdouble *vertex_data[4], GLfloat weight[4], GLdouble **dataOut )
{
	GLdouble *vertex;
	vertex = (GLdouble *) malloc(3 * sizeof(GLdouble));
	vertex[0] = coords[0];
	vertex[1] = coords[1];
	vertex[2] = coords[2];
	*dataOut = vertex;
}

GLvoid errorCallback(GLenum errorCode)
{
	const GLubyte *estring;
	estring = gluErrorString(errorCode);
	fprintf(stderr, "Tessellation Error: %s
", estring);
}


//init()
	wall1 = gluNewTess();
	gluTessCallback(wall1, GLU_TESS_VERTEX, (GLvoid)tcbVertex);
	gluTessCallback(wall1, GLU_TESS_BEGIN, (GLvoid)tcbBegin);
	gluTessCallback(wall1, GLU_TESS_END, (GLvoid)tcbEnd);
	gluTessCallback(wall1, GLU_TESS_COMBINE, (GLvoid)combineCallback);
	gluTessCallback(wall1, GLU_TESS_ERROR, (GLvoid)errorCallback);
	glShadeModel(GL_FLAT);


//display()
	gluTessNormal(wall1, 0.0, 0.0, 1.0);
	gluTessProperty(wall1, GLU_TESS_WINDING_RULE, GLU_TESS_WINDING_ODD);
	gluTessBeginPolygon(wall1, NULL);
		gluTessBeginContour(wall1);
		for (int i = 0; i < 4; i++)
			gluTessVertex(wall1, outer[i], outer[i]);
		gluTessEndContour(wall1);
		gluTessBeginContour(wall1);
		for (int i = 0; i < 4; i++)
			gluTessVertex(wall1, inner[i], inner[i]);
		gluTessEndContour(wall1);
	gluTessEndPolygon(wall1);

I suspect it has something to do with the (GLvoid) casts I have to add to the gluTessCallBack function parameters to eliminate the compile errors. Specifically:
error C2664: ‘gluTessCallback’ : cannot convert parameter 3 from ‘GLvoid (__cdecl *)(GLvoid *)’ to ‘void (__stdcall *)(void)’

I dont really understand what __stdcall is. Other than this problem, I cant see anything else wrong with this code. Suggestion?

I’ve been convicned to scrap gluTess for this type of project and look into using the stencil buffer for the same effect

o_O but you can do the calculations yourself. If the walls are rectangular, it’s just 3 interpolations to get a texture-coordinate. Just use the ideas of software-rasterization of textured triangles.

Come again? I are you saying do the divison of rectangles into the 2 sub-triangles myself and map the texture? That seems fine, but say I want to use a texture that is…128x128 and my wall certainly isnt going to be a power of 2 in dimensions, how will be able to make the texture look seamless, and uniform?