I have a question, let’s say that I design an opengl application around a viewport of 320x480. When you resize the window size your design size doesn’t change but the actual window will get larger, smaller, etc.

There’s a few different ways that this could be remedied.

have a max height and scale the width as wide as possible.
for example if you have a vertically scrolling application where things are positioned just outside the height of the window, then making sure that height is a constant size is a good thing.

have a max width and the height can be as tall as it needs to.
this would be for a horizontally scrolling application.

just stretch the image to the new size
This might work in certain situations.

How do I properly setup letter and pillow boxing to say keep the window at a fixed height and allow the width to scale or a fixed width and allow the height to scale?

For an orthographic projection, you’d typically have minimum bounds for each direction. For a given window size, you compute separate scale factors (window size divided by bounds) for each direction, choose the smallest scale factor, and apply that to both directions. So if the minimum bounds were the same in each direction, if the window was taller than it was wide, the width would determine the scale factor and you’d get extra at the top and bottom. If the window was wider than tall, the height would determine the scale factor and you’d get extra at the left and right.

This doesn’t work so well for a perspective projection, because wide angles result in significant distortion near the edges. So you need to ensure that the view angle in the larger direction doesn’t exceed some threshold, even if that means that the view angle in the smaller direction is narrower than you would otherwise like.

Given that you presumably want the aspect ratio to be correct, you get to choose one other parameter. If you have multiple constraints, then you find the limiting value for each constraint separately, then choose a value which satisfies all of them.

[QUOTE=GClements;1281347]For an orthographic projection, you’d typically have minimum bounds for each direction. For a given window size, you compute separate scale factors (window size divided by bounds) for each direction, choose the smallest scale factor, and apply that to both directions. So if the minimum bounds were the same in each direction, if the window was taller than it was wide, the width would determine the scale factor and you’d get extra at the top and bottom. If the window was wider than tall, the height would determine the scale factor and you’d get extra at the left and right.

This doesn’t work so well for a perspective projection, because wide angles result in significant distortion near the edges. So you need to ensure that the view angle in the larger direction doesn’t exceed some threshold, even if that means that the view angle in the smaller direction is narrower than you would otherwise like.

Given that you presumably want the aspect ratio to be correct, you get to choose one other parameter. If you have multiple constraints, then you find the limiting value for each constraint separately, then choose a value which satisfies all of them.[/QUOTE]

I forgot to mention that I was talking about only orthographic projection. With that being said, is there any sample code that I can see?

float scale_w = 1.0f * window_w / target_w;
float scale_h = 1.0f * window_h / target_h;
float w, h;
if (scale_w < scale_h)
w = target_w, h = window_h / scale_w;
else
h = target_h, w = window_w / scale_h;
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluOrtho2D(-w/2, w/2, -h/2, h/2);

[/QUOTE]

looking at this code and taking a shot. The line

if(scale_w < scale_h)
w = target_w;
h = window-h / scale_w;

so that’s basically saying that if the screen is in a vertical layout scale the width by that scale percentage, the other line says scale the height right?