Originally posted by knackered:
Trees, grass, clouds, rocks.
Not teapots, granted - but most serious users of OpenGL are interested in more than buggering around with bumpmapped teapots and rabbits.
I refer you once again to the arguments for procedural textures and geometry. It’s the same argument for instanced geometry.
No my argument is GLSL yours is phong fragment shading. And please show me that its so much faster. Show me thats not possible now.
Originally posted by Korval:
Have any specific evidence of that?
How about common sense? If you have a bottleneck somewhere else, it’s not going to help speeding up this particular piece of the pipeline. You don’t exactly expect performance increases by optimizing your vertex shader when you’re fillrate limited either.
Originally posted by Korval:
The onus isn’t on us to provide a reason for this extension; we already have one (worst case, it does nothing. Best case, non-trivial performacne gains. Ergo worthwhile). The onus is on you to provide some specific evidence that shows how it would not be useful.
That’s an incredible backwards way of working. You don’t start by expecting everything put forth to be implemented unless it explicitely is proven to be useless by those you expect to implement it. It’s up to those suggesting a particular features to convince the implementors that the feature is useful.
I don’t think I realized that you could replicate vertex data from indices, thus avoiding replicating the actual vertex data and thus potentially blowing your cache.
???
It isn’t that bad (or bad at all). It simply requires a different kind of per-vertex fetch operation. It doesn’t screw up the cache unless the hardware has some horrible limitation.
It think you’re expecting too much from current hardware. Most hardware if not all see a slowdown when using multiple streams because of this. That’s also why you should use interleaved arrays rather than separate streams unless you have a good reason otherwise. The difference is of course small though unless you’re limited by vertex fetch performance.
Considering that you work for ATi, this means that it’s ATi’s problem, not a problem with the concept as a whole or hardware in general. They should have made a real graphics card this go around with real features, rather than a simple knockoff of the R300.
To begin with a clarification, what I’m saying here should be interpreted entirely as my own personal opinion and not a official ATI opinion. It’s entirely possible that the driver team disagrees.
What I’m referring to is not just ATI’s problems. I have very good reasons to believe the same applies to nVidia. Their instancing performance is worse than ours. Both in absolute and relative terms.
And yet, it is immediate mode which gives OpenGL some semblence of instanced rendering.
Immediate mode as in glBegin()/glEnd() etc., not as in the ability to provide uniform values across a bunch of primitives. There are glColor calls for instance in OpenGL ES IIRC, but there’s no glBegin()/glEnd(). Good riddance I say.
Performance should always be priority #1.
Stability is #1. Bug fixes go before performance. But even if you’re working on performance, where do you spend your resources on optimizations? A niche features that’s hardly useful on current hardware, or on say boosting shader performance, resource allocation etc?
Originally posted by Korval:
Equally importantly, it isn’t a difficult thing to implement. If the hardware supports it directly, then it is trivial.
And you know this by experience I presume?
You don’t exactly expect performance increases by optimizing your vertex shader when you’re fillrate limited either.
Vertex transfer is limitted by the bus between the CPU’s memory and the graphics card. This bus rarely increases in size, and when it does, it isn’t terribly much. By contrast, fillrate doubles quite frequently, and vertex processing isn’t that far behind.
We may be bound on these things now (and, it isn’t that tough to be bound on vertex transfer; just have simple shaders), but as time goes on, we will not be nearly so bound on them.
Oh, and don’t forget that any attempt to alieviate a CPU burden is good; we need to find ways to increase parallelism and improve on what the CPU can do. This functionality provides this, and it is a help for those situations.
Lastly, it doesn’t hurt to have it there in terms of performance either.
That’s an incredible backwards way of working. You don’t start by expecting everything put forth to be implemented unless it explicitely is proven to be useless by those you expect to implement it. It’s up to those suggesting a particular features to convince the implementors that the feature is useful.
I did. It will, worst case, not impair performance. Best case, it will improve performance. Ergo, it is worthwhile.
QED.
This isn’t something to the level of complexity of glslang or even RTT (with the infinitely many ways of combinding textures). It’s one entrypoint with very well-defined behavior.
What I’m referring to is not just ATI’s problems. I have very good reasons to believe the same applies to nVidia. Their instancing performance is worse than ours. Both in absolute and relative terms.
As a guess, their instancing performance is likely due to having to copy the indices directly into the command stream. NV20 and previous hardware were unable to directly use indices from AGP memory (or, doing so caused some slowdown or something); that’s why VAR didn’t let you do it. It’s, also, likely why the VBO spec suggest putting your index VBO’s in a different object from your mesh VBO’s, so that they can optimize that circumstance. Since your indices have to be repeated pretty massively for instancing (number of instances * indices in an instance), this is likely going to make things slower than the theoretical maximum.
By contrast, ATi’s never had a problem with pulling directly from a buffer in AGP memory.
And it’s still faster than batching.
Immediate mode as in glBegin()/glEnd() etc., not as in the ability to provide uniform values across a bunch of primitives. There are glColor calls for instance in OpenGL ES IIRC, but there’s no glBegin()/glEnd(). Good riddance I say.
I can’t say I disagree. I wonder if, in the relatively near future (2-3 years), PC IHV’s will just start implementing OpenGL ES instead of regular OpenGL. Except for the heavy palatting options, it seems to be a much nicer, cleaner OpenGL.
A niche features that’s hardly useful on current hardware, or on say boosting shader performance, resource allocation etc?
You haven’t provided any real evidence that the performance gain will be minimal. Saying it doesn’t make it true.
And you know this by experience I presume?
Experience is not required. If the hardware actively supports instancing, then the specific calls simply translate into writing the proper token into the command stream. The only way the extension becomes more complicated is if you have to manually “uninstance” it, because the hardware isn’t capable of handling it directly.
Korval,
I actually looked into implementing an instancing API in the open-source R200 driver on Linux. I worked through what it would take to implement it, what it would take for developers to use, and what the potential performance gains would be.
My understanding of instancing is that it allows you to draw the same mesh (with identical state) multiple times with a different tranformation matrix. The simple API that I used added two functions. I think the intended usage of both functions is pretty obvious.
void MatrixPointer(int size, enum type, sizei stride, void *pointer);
void DrawInstancedRangeElements(enum mode, uint start, uint end, sizei count, enum type, const void *indices, uint instances);
When used with all data in on-card VBOs, I found the following:
[ol][li]The theoretical performance gain would be roughly equal to the API overhead of calling PushMatrix, MultMatrix, DrawRangeElements, PopMatrix. That overhead, with properly setup VBOs and moderate sized meshes, was very small.[/li] [li]The theoretical performance gain over compiling a display list with the sequence of PushMatrix, MultMatrix, DrawRangeElements, PopMatrix on a card like the R200 (probably not the best example) was nil (see way, way below).[/li] [li]Requiring developers to calculate their own matricies and store them in an array instead of using GL functions to create the matrix sucks (see below).[/ol][/li]I did also consider an API that added a parameter to DrawInstancedRangeElements to specify whether to load or multiply the matrix and whether the matricies were “standard” or transpose. That still takes away some of the flexability of GL matrix operations, but I think it covers the 90% case.
There are three cases where I was able to convince myself that there would be significant performance gains. However, I wasn’t able to convince myself that I cared enough about those cases to continue with the exercise.
[ol][li]Very small meshes (the hypothetical 1,000,000 cubes case). These cases would see similar performances gains as with using NV_primitive_restart . [/li] [li]Non-VBO vertex arrays.[/li] [li]With some modification to the proposed API, immediate mode.[/ol][/li]Basically, with OpenGL instancing would only help in cases that are API call bound. That leaves two questions that I haven’t seen answered (either “yes” or "no). Are there important cases of that today? Are there important cases of that moving forward?
My other finding was that modifying the display list rules WTR vertex arrays provided more flexability with the same performance improvement potential. Basically, display lists copy data out of the vertex array when the display list is compled. If display lists sourced the vertex data when the display list was executed, you could trivally do instancing without adding extra API entry-points (other than perhaps a BeginListArrays or something) or reducing GL flexability. There are some other potential problems with it, but it’s something I’ve been mulling over for a awhile now.
My understanding of instancing is that it allows you to draw the same mesh (with identical state) multiple times with a different tranformation matrix.
It is more general purpose than that.
The way instancing works is that you state that one or more attributes are not indexed the way other attributes are. Instead, they are indexed by a frequency parameter. Their index starts at 0, and, whenever the current index’s index hits a multiple of the frequency, then the index for the modified attribute(s) increases.
This creates instancing depending on your vertex shader. If each instance contains, say, 45 indices, then you set the frequency parameter to 45, and you have an attribute array just contains a list of positions. Now, you create a large index array, with 45 * the number of instances entries in it. It repeats every 45 indices. You make a single draw call with that vertex array.
Very small meshes (the hypothetical 1,000,000 cubes case). These cases would see similar performances gains as with using NV_primitive_restart.
Well, how small do they need to be? The number being thrown about in this thread is 100 vertices. You can do quite a bit in 100 vertices: rocks, grass tuffts, bushes, simple windows on buildings, mesh-tiled walls, etc. Anything you want to create by step-and-repeat can be built with this.
Are there important cases of that today? Are there important cases of that moving forward?
There’s no way to answer that. Programs avoid being vertex transfer bound because they do not use large numbers of small meshes. They would like to, but they know this will cause performance problems. Until they have an efficient way to do it, it simply won’t be done by developers.
We’re talking about window-dressing here (unless you’re making an asteroids-like game). It isn’t large geometry. It isn’t huge stuff. It would certainly help in terms of creating better graphics, but it isn’t preventing anyone from making a game or other application. It is just preventing them from fully utilizing their hardware’s power to achieve a greater level of depth in terms of graphics.
Well, at least someone, who has experience with this, is clearing things up a bit.
Thanks, idr!
If OpenGL is really already that fast, that instancing couldn´t improve much/anything, then we really don´t need it.
However, the problem is, that we, as “end-users” really cannot know such stuff, and that this feature has been denied to us, with the lame excuse “this is already possible in immediate mode”.
It would be nice, if we could simply get a bit more detailed information, why such a feature is not necessray / would not be useful for us.
Jan.
The way instancing works is that you state that one or more attributes are not indexed the way other attributes are. Instead, they are indexed by a frequency parameter. Their index starts at 0, and, whenever the current index’s index hits a multiple of the frequency, then the index for the modified attribute(s) increases.
Interesting. You basically specify a modulous value in addition to the usual type, count, stride, etc. parameters to the *Pointer functions. The arrays are then indexed with (i % M) instead of i. Can you only use instancing with DrawArrays-like commands? I guess it could work with DrawElements, but you’d have to explicitly replicate and bias the index data. Hmm…
I don’t see how this would apply to the transformation matrix (especially since it isn’t indexed per-vertex). You’d have to use DrawElements and ARB_matrix_palette , which sounds unpleasant.
Well, how small do they need to be? The number being thrown about in this thread is 100 vertices.
The actual number for my system aren’t terribly valid. The speed of my CPU is way overbalanced to the speed of my GPU. Using some profile data to guide back-of-the-envelope calculations, I saw that “reasonable” performance improvement would drop off around 40 vertices. Again, different CPU / GPU combinations would have that mark at different places.
There’s no way to answer that. Programs avoid being vertex transfer bound because they do not use large numbers of small meshes. They would like to, but they know this will cause performance problems. Until they have an efficient way to do it, it simply won’t be done by developers.
That’s the thing, though. In the optimal VBO case, there should be no difference in the vertex transfer bandwidth required with or without instancing. In fact, if index data has to be replicated and biased (I asked about this above), instancing requires more index transfer bandwidth. What it does save is API call overhead and, on cards with “optimal” support for instancing, command transfer overhead to the card (hence my comment about NV_primitive_restart in my previous post).
When I get a chance, I’ll have to experiment with this some more…
The arrays are then indexed with (i % M) instead of i.
It isn’t quite (i % M). It’s actually the index of i (the index into the index array) that gets modded by M and used as the index into the instanced vertex attribute(s).
I don’t see how this would apply to the transformation matrix (especially since it isn’t indexed per-vertex).
It doesn’t. Instancing is a way to work around stalls created by frequent vertex state changes like transform matrices or other such things.
I guess it could work with DrawElements, but you’d have to explicitly replicate and bias the index data.
You do replicate the index data (one per rendered instance), but no biasing is needed. Regular attributes follow the standard rules; it is only the instance attributes that follow the new rules. It does need more overall memory, and it sucks for the pre-T&L cache.
Originally posted by idr:
Interesting. You basically specify a modulous value in addition to the usual type, count, stride, etc. parameters to the *Pointer functions. The arrays are then indexed with (i % M) instead of i. Can you only use instancing with DrawArrays-like commands?
More.
Modulo addressing or div addressing, per attribute pointer.
You put instance mesh data into attribute stream that are modulo adressed (index_n’=index%stream_frequency_n).
You put per-instance data into other attribute streams that are div addressed (index_n’=index/stream_frequency_n; integer division).
“Ultimate” indexing would allow different frequencies per attribute pointer, and also index biases, but that’s probably not necessary. 99% of cases could be covered with a single frequency for all attributes and only a div/modulo/flat choice per attribute.
This allows you to use vertex positions as “instance mesh data” (the actual model of e.g. a rock), and other attributes as “per-instance data” (per-instance position, rotation, scale, color modifiers, texcoord scale factors and offsets, blend factors between mutliple textures, specular and gloss modifiers, whatever).
Also note that you don’t really need direct matrix support. Just support attributes. A vertex program is required to make proper use of it w/o direct matrix support, but this way the whole instancing mechanism becomes far more versatile (not just position, scale, rotation). And vertex program support is widespread enough to pull it off.
You can then render a bright, small, flat rock, a darker, big, grainy rock, another mossy rock, etc, in a single API call – and a single command buffer transaction if you have hardware support.
And if you don’t really need full matrices, but can do with position and scale (say, particle systems), you just use one attribute, not four.
Or use a quaternion and another vec4 for rotation, position and uniform scale. You know the drill.
Originally posted by idr:
I guess it could work with DrawElements, but you’d have to explicitly replicate and bias the index data. Hmm…
You’d have to if you implement this as a pure in-ICD software feature. If you implement this to expose a feature of the vertex fetch hardware, you don’t have to do any replication and biasing of indices. That’s the point. If the hardware you’re working with doesn’t support the required addressing modes, I wouldn’t bother.
Originally posted by idr:
I don’t see how this would apply to the transformation matrix (especially since it isn’t indexed per-vertex).
That’s why it shouldn’t be part of any instancing mechanism IMO. Matrices (or perhaps vertex program constants) are state, and while they can be implemented on top of attributes, that’s very ugly and non-orthogonal. You’d get huge problems if the current vertex program already uses up all attributes the vertex processing hardware allows, but you need four more to implement your matrix. No go.
Programmers shouldn’t be encouraged to believe that they can use the same number of attributes along with an instanced matrix. That’s why I strongly suggest this should be limited to vertex attributes.
Originally posted by Korval:
[QB]Vertex transfer is limitted by the bus between the CPU’s memory and the graphics card.
If you have your model in a VBO, which I expect people to have, then the transfer over AGP is exactly the same for real and shader constant based instancing (except maybe some tiny difference in command stream).
I did. It will, worst case, not impair performance. Best case, it will improve performance. Ergo, it is worthwhile.
QED.
No, you should prove that it DOES improve performance. Not that it doesn’t reduce it. If you end up with zero gain it’s not worthwhile. Besides, at worst is does impair performance by adding overhead to glDrawPrimitive for the non-instanced case. That’s what it does in D3D.
You haven’t provided any real evidence that the performance gain will be minimal. Saying it doesn’t make it true.
Well, you haven’t exactly proven that the performance gain will be huge so we’re pretty even then. I’ve intended to bring home an instancing app I got at work and quickly switch it to OpenGL to compare, but I have so far forgot it. But my experience with instancing on the D3D side, and knowing that the overhead of DrawIndexedPrimitive() is much higher than glDrawPrimitive() is convincing enough for me to make it not worthwhile on the GL side on current hardware.
Originally posted by Korval:
You do replicate the index data (one per rendered instance), but no biasing is needed. Regular attributes follow the standard rules; it is only the instance attributes that follow the new rules. It does need more overall memory, and it sucks for the pre-T&L cache.
No you don’t. You only need one copy of the index data. If you had to duplicate it instancing would definitely be useless, and I assume shader constant based instancing would beat it soundly in all situations.
Modulo addressing or div addressing, per attribute pointer.
Really? I’d apparently gotten my information confused.
This is, in fact, better than I thought. With a mechanism as flexible as this, you can, in theory, change the actual meshes for each instance, by simply using a different set of indices into the same vertex array. Trees with physically different branches that all pull from the same set of vertices. This is good stuff. Very good stuff.
If you have your model in a VBO, which I expect people to have, then the transfer over AGP is exactly the same for real and shader constant based instancing (except maybe some tiny difference in command stream).
Once you compare instance-based methods to state-change based methods, you’re comparing two different things. The state-change method will induce stalls in the pipe, while the instance one will not.
No, you should prove that it DOES improve performance. Not that it doesn’t reduce it.
Why does VBO include usage hints? There’s no proof that they will improve performance. By your logic, they shouldn’t exist.
They exist for the purpose of giving the driver vital information that will allow it to improve performance where possible. The same goes here.
Besides, at worst is does impair performance by adding overhead to glDrawPrimitive for the non-instanced case. That’s what it does in D3D.
A simple conditional is not overhead. And, by all rights, you can negate this (and any other ‘if’ overhead) by using a simple v-table/function pointer.
Plus, if that overhead wasn’t in the driver, it’d be in the user’s application. Whether the underlying driver is the one adding the constants to the command stream or not, someone has to.
Well, you haven’t exactly proven that the performance gain will be huge so we’re pretty even then.
Except that it doesn’t need to be huge; it just needs to be non-trivial. It doesn’t need to be the difference between immediate mode and VBO’s; it just needs to be able to consistently beat the uninstanced case by a non-trivial margin. 5-10% would be sufficient. Getting rid of the CPU overhead alone would be quite helpful.
If you had to duplicate it instancing would definitely be useless, and I assume shader constant based instancing would beat it soundly in all situations.
How can you expect me to believe that a mechanism that, by its very nature, induces stalls in the pipeline (ie, state changes) is going to be slower than one that works exactly like the hardware wants to?
Oh, and BTW, this thread already contains substantial evidence (not proof because we have no nVidia results) that the “shader constant” instancing method is weaker under OpenGL in both tests (by a 6:1 margin) than using attributes. MZ’s uniform-vs-attribute test (on the first page) attests to this fact. So, clearly, using uniforms for instances is a bad idea, compared to attributes.
Originally posted by Korval:
[b] [quote]Modulo addressing or div addressing, per attribute pointer.
Really? I’d apparently gotten my information confused.
This is, in fact, better than I thought.[/b][/QUOTE]Actually, your information may have been more correct than mine. After looking at what Microsoft put in DirectX Graphics , it looks like they didn’t bother with the modulo mode, which is IMO both vital to get the full bang out of the technique, and quite easy to add if you already have the div mode.
If you have already spent the resources to compute the result of an integer division (which is quite hefty btw), you can certainly also deliver the remainder of the corresponding integer division without adding much extra complexity. x86 be my witness: it’s not possible to divide without computing the remainder; you get it for free. I don’t understand how this could have been left out. This crippled form of instancing is really much more limited than it needs to be.
Anyway, instancing in DXG is not as flexible as I described. My apologies.
Originally posted by Korval:
Once you compare instance-based methods to state-change based methods, you’re comparing two different things. The state-change method will induce stalls in the pipe, while the instance one will not.
Yes, but this has nothing to do with what I wrote. You implied that there would be a difference in the amount of AGP traffic. There’s not.
Why does VBO include usage hints? There’s no proof that they will improve performance. By your logic, they shouldn’t exist.
They exist for the purpose of giving the driver vital information that will allow it to improve performance where possible. The same goes here.
Uhm, the difference is that we know for a fact that these flags improve performance if used correctly. Until we know for a fact that an instancing API will improve performance there’s no reason to add such an API. If we knew that, I would of course support the inclusion, but until we know that I remain sceptical. There are litterally hundreds of things you could add to the API that might improve things in certain situations if hardware and drivers would be done for it etc. But we don’t just add stuff without knowing it’s useful.
A simple conditional is not overhead. And, by all rights, you can negate this (and any other ‘if’ overhead) by using a simple v-table/function pointer.
A simple condition is overhead unless you expect 100% cache hits. Actually, even with 100% hits you still have to execute the actual comparison.
Overhead might not be large, but it’s not for free.
Plus, if that overhead wasn’t in the driver, it’d be in the user’s application.
??
There’s no overhead of something that’s not even there.
How can you expect me to believe that a mechanism that, by its very nature, induces stalls in the pipeline (ie, state changes) is going to be slower than one that works exactly like the hardware wants to?
You mean faster, right? Well, because that’s what it does in practice in many situations already as it is. Even though you don’t need multiple index copies as you thought for real instancing, while constant based instancing needs to duplicate both vertices and indices, it still beats real instancing in many situations, in particular on small models, like < 20 triangles. If you would need multiple copies of the indices for real instancing, that would make it slower, and thus constant based instancing would beat it in more situations. Plus, a few stalls aren’t really hurting much. It’s already reducing the number of state changes enough to hide the cost of that.
Oh, and BTW, this thread already contains substantial evidence (not proof because we have no nVidia results) that the “shader constant” instancing method is weaker under OpenGL in both tests (by a 6:1 margin) than using attributes. MZ’s uniform-vs-attribute test (on the first page) attests to this fact. So, clearly, using uniforms for instances is a bad idea, compared to attributes.
Uhm, MZ’s test was not a what I mean with shader constant based instancing. That’s what I call naive one-instance per call implementation. It did prove one thing though. Even with such a naive implementation he reached performance close to theortechical maximum, which would support that instancing hardly would improve performance at all.
as intresting as discussion is, I’d like to jump in here and ask if anyone wants to give an update on the kinda time table we are likely to work towards for the specs and implimentation for the framebuffer_object extension?
I wouldn’t expect much until March 
.
Uhm, the difference is that we know for a fact that these flags improve performance if used correctly.
We only know that because the implementers decided to. They could have decided to ignore the flags and simply stick everything into video memory. Or, they could have moved the memory around based on how you use it.
There is nothing intrinsic in the spec that makes the flags advantageous, performance wise. Only implementations make them useful.
Actually, even with 100% hits you still have to execute the actual comparison.
Not if they use a v-table. And, considering the current number of conditionals that have to live inside glDraw* calls (among others), I wouldn’t be surprised if they did invoke v-table calls. Various state changes would simply swap v-tables in and out.
There’s no overhead of something that’s not even there.
As in, the user’s application will have to do the instancing stuff. It will have to make decisions about how to render stuff, and how to convert a sequence of instances into a sequence of draw calls.
You mean faster, right?
You’re right. I meant faster.
Even though you don’t need multiple index copies as you thought for real instancing, while constant based instancing needs to duplicate both vertices and indices, it still beats real instancing in many situations, in particular on small models, like < 20 triangles.
How does it beat something that invokes O(n) stalls (where n is the number of instances)?
Plus, a few stalls aren’t really hurting much. It’s already reducing the number of state changes enough to hide the cost of that.
Having no state changes is faster than having state changes. The only potential performance negative is the need to walk a large index buffer (which means indices won’t be pre-T&L cached), but that’s not nearly as bad as stopping the entire pipeline to wait until the post-T&L cache is flushed (if not more).
Stalls waste performance. Not having stalls does not.
Uhm, MZ’s test was not a what I mean with shader constant based instancing.
His slower method may have had more state changes than yours, but it does demonstrate that state changes are bad.
That’s what I call naive one-instance per call implementation. It did prove one thing though. Even with such a naive implementation he reached performance close to theortechical maximum, which would support that instancing hardly would improve performance at all.
But that method (the one that did well) used attributes (specifically, constant attributes per draw call), not state changes.
More importantly, it only demonstrated this on ATi cards (you know, the ones that don’t have hardware instancing). It proves nothing about nVidia cards, some of which actually do provide hardware instancing support.
Originally posted by ffish:
I wouldn’t expect much until March.
It migth also mean that the spec will be released at the same time. ****tttttttttttt!
Originally posted by Korval:
Only implementations make them useful.
Exactly what I’m saying while you’re arguing that stuff should be implemented only because it could improve performace.
Not if they use a v-table.
v-tables are even slower in most cases unless the number of paths are large and the conditions can be linearized.
How does it beat something that invokes O(n) stalls (where n is the number of instances)?
Stalls impair performance, but there are hundreds of other factors that could be equally or more important.
His slower method may have had more state changes than yours, but it does demonstrate that state changes are bad.
But is it the worst thing in the equation. Apparently not.
But that method (the one that did well) used attributes (specifically, constant attributes per draw call), not state changes.
So? He still reached near theorethical maximum performance with currently available APIs. What more do you need? Instancing can’t make it go faster than the hardware.