GL_RAY_virtual_textures extension

Raystonn · February 23, 2006, 12:26am

Hi all,

I am looking for comments, contributions, and criticisms for the extension I am drawing up. Once complete I will run it by nVidia, ATI, and 3D Labs to see if they are interested in implementing an EXT version. This extension gives us some of what will be available in the hardware designed for DX10, but I have tried to keep the impact minimal. (The ARB hates huge API changes, and rightly so.)

Without further ado:

Name
 
    RAY_virtual_textures
 
Name Strings
 
    GL_RAY_virtual_textures
 
Contact
 
    Samuel Vincent (raystonn 'at' hotmail.com)
 
Status
 
    XXX - Not complete yet!!!
 
Version
 
    Last Modified Date: February 22, 2006
    Author Revision: 0.1
 
Number
 
 
Dependencies
 
    OpenGL 1.3 is required.
    The extension is written against the OpenGL 2.0 Specification.
 
Overview
 
    When this extension is advertised, the OpenGL implementation virtualizes
    all texture units and memory under its management.  This has the effect
    of: a) allowing the presentation and use of much more memory in a single
    rendering pass, including textures with no dimensional limits, b) allowing
    more efficient management of high-speed cache memory present on many
    graphics adapters, and c) allowing the use of more textures during a
    single rendering pass than there are texture units on the underlying
    implementation's hardware.
 
    System memory, GPU-based memory (when present), and the operating system's
    swap file (when present) will all be used to hold data that may be used
    at any time.  This means that a much larger working set will be possible.
    Performance may be adversely affected if an application uses too much data
    in a single rendering pass, causing some of it to be loaded from a virtual
    memory swap file for example, but unused data should have nearly no
    performance impact.  One benefit of having this potentially-unused data
    available is that it allows the GPU and its shader programs to decide
    which data to use on the fly when the data set may not entirely fit in the
    high-speed cache memory.  This gives unprecedented flexibility.  Large
    textures well beyond the capabilities of current hardware may be used,
    with the portions of the texture that are needed being paged into high-
    speed cache when it is available.  The only limit to the size of a texture
    would be due to the exhaustion of the operating system's virtual memory
    space.
 
    The implementation will be free to manage memory at a finer granularity.
    It will have the ability to keep only a portion of a texture in high-speed
    cache memory rather than being forced to keep it either fully in or fully
    out of the cache.  More space is thus freed to cache other data.  A system
    of organizing memory into pages that are managed independently will allow
    textures of unlimited size, bounded only by the availability of operating
    system virtual memory.
 
    Texture units will now be virtualized as well, allowing an application to
    reference a nearly unlimited number of textures in a single pass.  The
    implementation can map these virtual texture units to real texture unit
    hardware as needed, but the application will remain unaware of the
    mapping.  This allows an application to make many more textures available
    and allow a shader to choose which it will use on the fly.
 
    This virtualization will allow shaders to choose the data most appropriate
    for the situation.  As shaders get more complex, the application itself is
    less able to determine in advance which data will be required.  It is
    important that the application be able to offer the entire data set and
    allow the shader to choose at runtime.  This is an important step toward
    allowing increased complexity of shader programs.
 
IP Status
 
    No known IP claims.
 
Issues
 
  * Effects on other extensions and core components of OpenGL must be detailed
    in this specification.
 
New Procedures and Functions
 
    As the GL_TEXTURE0 family of constants cannot be extended indefinately,
    their use will be discouraged.  In place of this, all functions that take
    these constants will have another version that takes a uint indicating the
    virtual texture unit number.  The old constants will still work when used
    with the old functions.  For example, GL_TEXTURE0 used with the old
    functions will reference the same virtual texture unit as 0 used with the
    new functions.
 
    void glActiveTextureu(uint texture);
    void glClientActiveTextureu(uint texture);
    void glMultiTexCoordu1d(uint target, double s);
    void glMultiTexCoordu1dv(uint target, const double *v);
    void glMultiTexCoordu1f(uint target, float s);
    void glMultiTexCoordu1fv(uint target, const float *v);
    void glMultiTexCoordu1i(uint target, int s);
    void glMultiTexCoordu1iv(uint target, const int *v);
    void glMultiTexCoordu1s(uint target, short s);
    void glMultiTexCoordu1sv(uint target, const short *v);
    void glMultiTexCoordu2d(uint target, double s, double t);
    void glMultiTexCoordu2dv(uint target, const double *v);
    void glMultiTexCoordu2f(uint target, float s, float t);
    void glMultiTexCoordu2fv(uint target, const float *v);
    void glMultiTexCoordu2i(uint target, int s, int t);
    void glMultiTexCoordu2iv(uint target, const int *v);
    void glMultiTexCoordu2s(uint target, short s, short t);
    void glMultiTexCoordu2sv(uint target, const short *v);
    void glMultiTexCoordu3d(uint target, double s, double t, double r);
    void glMultiTexCoordu3dv(uint target, const double *v);
    void glMultiTexCoordu3f(uint target, float s, float t, float r);
    void glMultiTexCoordu3fv(uint target, const float *v);
    void glMultiTexCoordu3i(uint target, int s, int t, int r);
    void glMultiTexCoordu3iv(uint target, const int *v);
    void glMultiTexCoordu3s(uint target, short s, short t, short r);
    void glMultiTexCoordu3sv(uint target, const short *v);
    void glMultiTexCoordu4d(uint target, double s,double t,double r,double q);
    void glMultiTexCoordu4dv(uint target, const double *v);
    void glMultiTexCoordu4f(uint target, float s, float t, float r, float q);
    void glMultiTexCoordu4fv(uint target, const float *v);
    void glMultiTexCoordu4i(uint target, int s, int t, int r, int q);
    void glMultiTexCoordu4iv(uint target, const int *v);
    void glMultiTexCoordu4s(uint target, short s, short t, short r, short q);
    void glMultiTexCoordu4sv(uint target, const short *v);
 
    The only difference between all of these new functions and the existing
    ones (without the 'u' in the function name) is the first argument.  It is
    now a uint which gives the texture unit number directly, without use of an
    enum.  Texture units are virtualized, and thus there is no upper limit on
    their number beyond that of the maximum value of the uint data type.
 
Revision History, 2006/02/22
 
  * Revision 0.1 - Initial revision

zeoverlord · February 23, 2006, 2:45am

I like it, but i would like it to extend a little bit further, perhaps into fragment/vertex programs, DrawArrays and VBOs.
allso more details on how binding textures and sutch is affected by this extention could be usefull.

And as i see it this extention does little in extending the API, i would like to see some more “advances” in how to do things, to realy take advantage of virtual textuing and not just enable more multitextures, Stuff like removing the restriction of glBind within glBegin/glEnd or even enabling you to pick a new texture for every vertex.
And instead of using glBind this might work equaly well

void glTexCoordu2f(uint texture, float s, float t);

within the begin/end statements to switch the texture for every polygon, something that is not advisable if you do not have virtual texturing.
Allthough it’s slightly redundant, it becomes more logical when you start messing with VBOs and draw arrays.
Extending glEnableClientState to accept GL_TEXTURE_ARRAY could be one other ting to possibly add.

Zengar · February 23, 2006, 5:20am

I think there is one more problem about this approach…

OpenGL’s state stores the current value for each vertex attributes, so the current texture coordinates are stored within the texture units. You hovewer move the texture coordinates from the unit state to the texture state. Even more, you move the texture target to the texture object! It is not large problem, however, each texture object will just have to store some bytes more, but how will it cope with the existing state?
When to use texture-own coordinates and when the unit-own coordinates?

What if I do something like

ActiveTexture(tex);
TexCoord3(1, 1, 1);

Raystonn · February 23, 2006, 8:36am

zeoverlord, if you want to switch textures for each vertex, I suggest a 3d texture. This will even allow you to interpolate between two or more 2d texture layers for proper blending.

I had originally written up a spec that was much more invasive with changes to the OpenGL API, removing the requirement to bind textures at all, and referencing them through their texture names, as returned by glGenTextures. But this required too many changes, including new data types and built-in functions in GLSL, and was very unlikely to gain widespread support. Thus, I sat back down and rewrote it to save as much of the existing API as possible, while still giving us access to the new hardware features that should be coming soon.

I welcome any and all contributions. My intention is to get the new hardware features supported as quickly as possible. OpenGL can take advantage of being the only API to support these features on most operating systems, including versions of Windows prior to Vista. (DX10 will only be available to those who purchase Vista.)

-Raystonn

Korval · February 23, 2006, 10:59am

I don’t understand what the need for an API change is. Texture units (things that sample textures) and texture coordinate sets are already unbound from each other, by right of ARB_fp and ARB_fragment_shader.

Each fragment program already has samplers that determine which textures they access. And the number of available samplers is already limitted by an implementation-defined value. All an implementation needs to do is make this suitably large. A glslang shader (the language) is not bound to any particular number of bound textures. An application can bind as many textures as the implementation allows.

Raystonn · February 23, 2006, 12:07pm

the number of available samplers is already limitted by an implementation-defined value

An application can bind as many textures as the implementation allows.
This is true. But this is also the problem. Forcing an application to query what the hardware supports is not in the spirit of OpenGL. Beyond that, it imposes limitations due to the requirement that the application know in advance which set of data will be used by a shader, because all of the data will not be accessible if the number of bound textures is limited. Thus, the application will need to know which subset of textures to bind each pass. As shaders get more complex, it is becoming increasingly difficult to know in advance which data will be needed. So we need the ability to offer all data to a shader each pass and allow it to grab what it needs. Texture unit virtualization offers this. The added functions simply fix the issue of the texture unit enums having a limited range of values. This was poor API design from the beginning.

Raystonn · February 23, 2006, 12:15pm

An alternative to virtualizing the texture units would be to add support for binding textures in shaders, thus enabling the shaders to choose their own data set. But this is far less elegant and would require much more work from implementation vendors.

The basic requirement I am trying to fill for this point is the ability of shaders to choose their data at runtime. Texture unit virtualization and texture memory virtualization fill this requirement.

-Raystonn

zeoverlord · February 23, 2006, 12:17pm

The Important thing about virtual texturing is that you have full access to all textures at all times and the idea of just being able to use more than 32 textures while multitexturing is not enough.
Listen to john Carmacs last keynote spech at last years quakecon, he explains things so well.

Anyway, virtual texturing is sort of redundant in opengl (it only makes the batch problem a little less problematic), unless you use it in shaders that is, to be able to sample freely from a large array of textures without mutch preformance loss is a big thing.

Raystonn · February 23, 2006, 12:23pm

While I am, indeed, thinking mostly about shaders, I disagree that virtual texturing outside of shaders is redundant. Virtual texturing removes the limitations of dimensions and allows you to use a texture that would never fit entirely in a video card’s dedicated texture memory. Very often only a small portion of a texture is required, rather than the whole thing. Virtual texturing will allow implementations to load small portions of textures rather than requiring the loading of entire textures, most of which may never be used. This is an important step forward for 3D hardware and OpenGL better be there with support when the hardware vendors offer it for DX10.

-Raystonn

system · February 23, 2006, 12:39pm

When this extension is advertised, the OpenGL implementation virtualizes
all texture units and memory under its management.
If you want to virtiualize, glGetIntegerv(GL_MAX_TEXTURE_IMAGE_UNITS, xxx)
would return a higher value than supported by the hw and so would MAX_TEXCOORD and the already existing API doesn’t need to change.

Sorry, but I don’t see the point of your functions.

Example:
“void glActiveTextureu(uint texture);”

Why would I want to activate a texture?
I would want to bind a texture to a virtual texture unit.

Example:
void glMultiTexCoordu1d(uint target, double s);

Why would I want to switch texture targets here?

Perhaps if you could provide an example, I would get a better picture.

zeoverlord · February 23, 2006, 12:50pm

Yea but that has noting done with multitexturing.
What i meant with redundant was that when you start aproaching situations where you need virtual texturing you probobly allready are using shaders.

Raystonn · February 23, 2006, 12:59pm

If you want to virtiualize, glGetIntegerv(GL_MAX_TEXTURE_IMAGE_UNITS, xxx)
would return a higher value than supported by the hw and so would MAX_TEXCOORD and the already existing API doesn’t need to change.
It seems my point has been lost on you. Forcing an application to query what the hardware supports imposes limitations due to the requirement that the application know in advance which set of data will be used by a shader, because all of the data will not be accessible if the number of bound textures is limited. Thus, the application will need to know which subset of textures to bind each pass. As shaders get more complex, it is becoming increasingly difficult to know in advance which data will be needed. So we need the ability to offer all data to a shader each pass and allow it to grab what it needs. Texture unit virtualization offers this.

Sorry, but I don’t see the point of your functions.

Example:
“void glActiveTextureu(uint texture);”

Why would I want to activate a texture?
I would want to bind a texture to a virtual texture unit.
You answered your own question there. Binding a texture would be one reason to set the active texture. None of this is new. glActiveTexture and glMultiTexCoord* already exist and are used for multitexturing. The new extension makes no changes to these functions beyond making the first argument a uint rather than an enum. Your question should be asked to those who designed the multitexture extension instead. I am merely keeping consistency with the old API.

-Raystonn

Zengar · February 23, 2006, 3:57pm

Raystonn, you say that your expension will allow the shaders to use any texture, right? But you still need to pass uniform samplers to the shader, don’t you? Or can is just acces any texture by it’s id?

Suppose, your app renders a mesh and defines 18 texture coordinates for it. The shader could possibly access 2000 textures, there is no way to determine it from the start. Must the driver send all 2000 to the shader for each vertex? I don’t really get it. It makes the GL state overcomplicated.

I think, it would be better to provide virtual texture samplers(units), where you can allocate them from GL by the unique name and use that names in the shader. Still, texture unit and texture object should be separated.
Even more, teh texture should hold only the texel data, while the unit will hold the state(filters, coordinates, etc.) It makes more sense for me.

Brolingstanz · February 23, 2006, 4:20pm

those interested should download and read the d3d10 overview in the dx sdk.

in d3d10, all memory is virtualized (no more “locks”) . shaders will have resource “views”, which effectively allow a shader to map a “view” of a resource and do whatever it wants with it. this allows for selectable textures from withing shaders and all sorts of other uses. also, improvements in the api are going to make function call overhead in d3d10 much less of an issue, but batching will continue to be a big deal.

the thing of it is, who’s to say the arb isn’t working on this as we speak? i guess we won’t know until it happens.

p.s. anyone vote for the pc version of opengl|es in the poll on the main page? i looked for the “heck yeah” option, but all they had was yes and no

Raystonn · February 23, 2006, 4:23pm

Zengar, thanks for bringing this to my attention. Indeed, in its current state the extension would still require a sampler for each texture. This would mean possibly thousands of uniforms, which is not acceptable (and probably not supported on any implementation right now). In my efforts to leave the API as unchanged as possible, I missed this problem.

This specification needs to be modified. My preference at this point would be to use the texture name returned by glGenTextures for everything, to skip the glBindTextures completely, and to introduce new texture2D(), etc. built-in functions that take the texture name as an argument instead of a sampler. The set of functions listed in the current specification would need to be modified to take a texture name rather than a virtual texture unit number.

What do you think of this approach?

Thanks,

Raystonn

P.S. I will add your name to the contributors section of the next revision if you like. I’ll just need your real name.

Raystonn · February 23, 2006, 4:29pm

the thing of it is, who’s to say the arb isn’t working on this as we speak?
It’s possible. But the ARB usually prefers new features to be implemented as an extension first. Besides, I don’t like waiting around to follow someone else’s lead. If you want something done right (according to what you think is right, anyway!) then you have to do it yourself.

-Raystonn

system · February 23, 2006, 4:35pm

Raystonn,

If you want to virtualize, all it means is that you want those MAX values to be “higher”. What would you like “higher” to be? What value do you want it to have?

Ok, I had misundertood glActiveTextureu. I thought you wanted to pass the texture ID to this. Nevermind.

Or can we just access any texture by its id?
That would be far more interesting. Imagine if we could access any texture using the texture ID from within a shader.
Keep the texture coord count the same as what the hw supports. Just give people the ability to access the texture pool.

For GLSL, they could just define new functions for this.

vec4 texture2D_ID(GLuint textureID, vec2 texcoord);
and so on.

Raystonn · February 23, 2006, 4:46pm

It’s interesting that both of you like the idea that I originally scrapped because I thought it was too invasive on the API. That’ll teach me!

V-man, I really want the maximum values in the API to be removed entirely. Just as virtual memory allowed programmers to no longer worry about how much RAM is installed in a system, there is no reason to make everyone worry about how many texture units or how much physical texture memory is on a video card either. A well-designed API will abstract these concepts.

I will modify this specification with these ideas, clean it up a bit, and update the specification in the first post of the thread later tonight.

Thanks again,

Raystonn

Zengar · February 23, 2006, 5:11pm

Ok, let me summarize, just to be shure I got it right

We can use virtual textures only with shaders(it doesn’t make much sense otherwise). The texture coordinates are not part of the GL state anymore, but instead, they will be specified as custom vertex attributes(so we get through all the coords issue I spoke about). We abandon the samplers and the shaders get the ability to address any texture by it’s id. The filter and othe rattributes is owned(specified) by the texture itself.

Ok, sounds nice…

The big downside of the ability to access any texture is that we hard-code our shaders upon the particular texture. We will still have to pass some uniform to the shader to identify the texture that should be used for the current mesh. Of cource, nice possibilities like texture addressed by texture(oh hell ) appear, but will it really be so usefull? Besides, it would be a great mess, because we will have to change the whole GL texturing pipeline :-/

I’ll try to describe the idea that come to me as I was reading this thread, the one with virtual texture units.
A texture unit is merely a logical interface between the data and the shader, a “view” to a texture, as leghorn nicely said. For example, the unit “bumpmap” has to refer to a bumpmap, a “color” unit will refer to a usual object texture.
What texture data a unit links to is of no interest to a shader. It just needs one to exist.
Imagine an api like:

int CreateVirtualSampler(format : int, pchar name )

where int is TEXTURE_2D, etc

void BindVirtualSampler(int sampler, int texture)

void DeleteVirtualSampler(int sampler)

in a shader, we can refer to a “name” as to a texture sampler of the chosen format, as if it was a uniform(if we use Uniformi on it, we must get an error, because the name is always reserved by the sampler). There can be unlimited count of such samplers

This is exacly what you want, isn’t it? And there is absolutely no collision with old API or states(well, almost ) And it is very elegant.

What do you think?

p.s. sorry about my spelling, it is late at night and I am really tired, but somehow I can’t sleep

Korval · February 23, 2006, 7:37pm

This is true. But this is also the problem. Forcing an application to query what the hardware supports is not in the spirit of OpenGL.
Glslang has enough problems with implementations randomly running in software rather than hardware; it doesn’t need more.

Indeed, the original glslang spec required that implementation defined limits were made totally transparent to the user. This was quickly (and rightfully) smacked down by the members of the ARB who weren’t 3DLabs. Such an implementation would also have required driver-defined multipass for shaders (which was also rightfully smacked down).

Beyond that, it imposes limitations due to the requirement that the application know in advance which set of data will be used by a shader, because all of the data will not be accessible if the number of bound textures is limited. Thus, the application will need to know which subset of textures to bind each pass. As shaders get more complex, it is becoming increasingly difficult to know in advance which data will be needed. So we need the ability to offer all data to a shader each pass and allow it to grab what it needs.
Wait. You’re suggesting that a shader (the actual shader code itself) decides which textures it wants to use? That’s nuts, and not necessary.

For shaders that might want to dynamically decide which texture to access (which is a very wierd thing to begin with), such a decision is fundamentally no different from picking samplers from an array. Now, maybe you can formalize this array by building it into the language itself (allowing the user to bind a texture to a specific index, rather than having to give the array’s name and index), so that everything is more standardized, but even that is just syntactical sugar.

And your rationale for this feature is dubious at best. It is not difficult at all for a developer to have an object stored with each shader that points to the list of textures that the shader uses, as well as the samplers that these shaders need to be bound to. You’re talking about maybe an hour of developer time, tops.

Only in the case of a truly dynamic texture fetch, where the shader is deciding which texture to fetch from by doing some kind of dynamic computation is this feature ever necessary.

While I am, indeed, thinking mostly about shaders, I disagree that virtual texturing outside of shaders is redundant. Virtual texturing removes the limitations of dimensions and allows you to use a texture that would never fit entirely in a video card’s dedicated texture memory.
And where, precisely, in the OpenGL specification does it say that the entire texture needs to be in memory at once? It doesn’t.

OpenGL’s texture object paradigm has always allowed for the possibility of “megatextures” that can’t ever fit into video memory all at once. Indeed, the OpenGL specification only talks about texture storage in the most nebulous terms (resident vs. non-resident). An implementation that can support megatextures, that can automatically page bits of textures into and outof video memory, simply says that all textures are resident, since it doesn’t make that distinction.

The spec allows for what you want. When the hardware allows for it and implementers implement it, then you will have it. And you don’t even need to know about it.

More importantly, this has nothing to do with the shader/texture binding stuff you’re proposing. These are two entirely different concepts.

BTW, I took a cursory glance at D3D. I found nothing at all there that seemed like it provided any more virtualization of how textures get bound than OpenGL’s glslang does. I’m looking at the 2005 December tech preview documentation.