nVidia drivers & 3dmark2003

OpenGL OpenGL: Advanced Coding
61

THATS WHAT A BENCHMARK IS FOR! NOT TO BE INTELLIGENT CODED, BUT TO TEST OUT IF SOME HW CAN PERFORM WELL!!
Let me remind you:
a) 3Dmark shows of “Simulating gaming environment” label
b) 3Dmark consists of:
. - Four “Game” Tests
. - “Theorethical” Tests
c) 3Dmark (by design!) does not include results of the “Theorethical” tests in final result.

The “stressing hardware” in Futuremark’s way, means running “stupid” (naively inefficient) algorithms - just to give more (useless) workload for GPU.

This design is in obvious contradiction with the “Simulating gaming environment” slogan. Real games don’t do this (at least not intentionally ).

There is supposed to be excuse for it: the benchmark is said to try to give premise of performance also in future games. But this claim is obviously invalid - why should games start to use “stupid” algorithms in future? There will always be better uses for spare GPU power than wasting it.

Conclusion

If there exists any place for “stupid” algorithms, then it is in “Theorethical” test. Not in “Game” tests. 3Dmark’03 is simply inconsistent with its own assumptions.

3dmark gave a quest. nvidia cannot win that

I think it would be easy to create benchmark which “proves” nv3x is faster than R3xx. it could use shaders with complex swizzles (this would increase ins. count) , or use screen space derivatives, or “require” >=33 constants, or “require” >=33 texture samples (each would force R3xx to multipass).

Finally, at the end of the frame, it could cover half of the screen with DX7/8 game scene

But this is not the point. Real game engine programer would not “give quest” to IHVs, but would do completely opposite: in his own interest he would do his best to design and optimize code for HW he is targeting.

If “Game” benchmark is meant to be valid and fair, it should follow this way. Which 3DMark’s “Game” test doesn’t.

[This message has been edited by MZ (edited 05-29-2003).]

62

But think about this:
If an IHV could get the card to perform at high speeds with an inefficiently designed benchmark, just think how fast it would be with an efficient one!

Edit:
Before I get beat in the face with a trout, I know that this isn’t the point of all this talk. I’m just saying.

[This message has been edited by NitroGL (edited 05-29-2003).]

63

Originally posted by DopeFish:
And tell me, what reason would this wrapper have to change how it is rendered based on executable filename? The wrapper which was made to run the dawn demo on ATI hardware having special cases for quake3.exe and 3dmark.exe?

Yes! Have you even seen what these “special cases” are?

Without renaming the executable, Dawn on a Radeon looks more or less normal, except for the issues reported earlier with the hair and the eyelashes. When you rename the executable to Quake3.exe, Dawn’s leaves disappear. When you rename it to 3DMark03.exe, both the leaves and the wings disappear, leaving you with a naked human.

If you rename the executable when running on an NVIDIA, nothing happens. Ergo, it must be the wrapper that’s doing the filename detection, and not the demo itself.

Get it now? The hair and eyelashes artefacts on ATI cards occur regardless of whether the executable has been renamed or not, and the whole renaming thing is nothing more than a nude patch built into the wrapper.

– Tom

64

What MZ just said is along the lines of what I was thinking.

Sometimes I even wonder whether Futuremark ever knew what they really wanted to do with 3DMark.

“Lets make a synthetic benchmarks that runs games tests and gives a synthetic result that will tell how well all other games run. Oh, and wouldn’t it be nice if it could make your coffee in the morning too?”

65

I still don’t understand how GT4 is biased towards ATI. . . GeforceFXs have poor floating point performance, but complaints regarding the requirement of FP precision would be better directed at Microsoft – or at NVidia for designing the GPUs that way. In any case, the scene must be rendered the exact same way regardless of what video card is used, so how is this not fair? Also, if GeforceFXs are horrible at FP precision, doesn’t that just plain make them bad for PS2.0? It’s not bias, simply observation of the obvious.

66

undoubtfully NV’s will be faster in DOOM3 and other upcoming games

JC doubts it. Indeed, he said that, under the ARB_vp/fp path, ATi wins, but under the NV-path, nVidia wins. Why? 24 vs 32-bit floats.

If there would be stress for ATIs hierarchial Z buffer and NVIDIAs Z oclussion culling, I fell there would be another story, but WE CAN’T DO THE TEST THE WAY IT HURTS OUR BETA MEMBERS, CAN WE?

The jury is still out about whether or not ATi’s z-test method is better or worse than nVidia’s. Indeed, some posts on this board have asserted that ATi’s z-tests are better. Historically, in game benchs that have low overdraw (generally, Unreal-based games), ATi’s cards tend to do much better than in low-overdraw circumstances.

I don’t know where the rampant nVidia fanboy-ism is coming from. Quite frankly, the GeForceFX line before the 5900 was significantly weaker than the ATi equivalents. Only the 5900 gives ATi any real competition.

As for 3DMark not using benchmarks that hurt their beta members… this doesn’t somehow justify nVidia’s cheating. If the test is unfair, show that it is unfair. Keep harping on the idea that it is unfair. However, nVidia must consider it a legitimate test, since they devoted driver development resources to cheating on it.

If there exists any place for “stupid” algorithms, then it is in “Theorethical” test. Not in “Game” tests. 3Dmark’03 is simply inconsistent with its own assumptions.

So, how does this justify nVidia’s cheating?

If nVidia is able to detect 3DMark, they should also be able to prevent rendering in the application, too. As such, they should just do that to prevent people from benchmarking 3DMark.

67

However, nVidia must consider it a legitimate test, since they devoted driver development resources to cheating on it.

I wouldn’t go that far. . . I’m guessing that NVidia just felt that 3dMark03 scores would have enough of an influence on sales as to justify spending time and resources developing ways to use 3dMark to their own advantage, despite their assertions against the benchmark’s validity.

[This message has been edited by Ostsol (edited 05-29-2003).]

68

Korval:

I Like how you assume (as are a lot of people) that it took a whole lot of effort for nvidia to do these cheats. I doubt they spent more than 2 days on this.

69

Originally posted by MZ:
I think it would be easy to create benchmark which “proves” nv3x is faster than R3xx. it could use shaders with complex swizzles (this would increase ins. count) , or use screen space derivatives, or “require” >=33 constants, or “require” >=33 texture samples (each would force R3xx to multipass).

well. and then explain me how to fit that into a dx9 test. you could as well write a test wich uses floatingpoint textures, and voilà, nvidia could not run any. thats no benchmark for dx9 then. 3dmark is. and the test shows, nvidia sucks at it. last time ati did. this time nvidia does. its fun how last time ati got blamed to suck, but this time you all stand behind nvidia, and bitch over futuremark.

their test is valid, and their test is good. there is no problem with 3dmark at all. it does not need to be a supertweaked optimized till the last engine. it runs fine on my radeon the way its made. i don’t except anything more. i buyed a dx9 card, i run a dx9 test, and it runs well. tell me any problem i shall have.

btw, the nude dawn is funny

70

OK, I have to put my two cent in.

I’m not defending nVidia’s actions, but am I the only one who sees a conflict of interest in Futuremark’s beta developers program? They take large amounts of money from hardware manufacturers whose products their software evaluates. Would it be appropriate if a hardware review website accepted hundreds of thousands of dollars directly from the companies whose hardware they were reviewing? And why do companies need a sneak peak at the 3dmark software (and the code!) before it is released? So they can tweak their drivers to suit the tests, that’s why. IMO, 3dmark has exceeded its usefulness.

And shame on you, nVidia. Not so much for cheating, but for doing it so stupidly. How could they not have been caught? Perhaps not as easily spotted as executable filename identification, but still pretty bad. It’s like writing the answers to a quiz on the top of your hand.

There, now I’ve posted my obvious and redundant comments, and all I have to show for it is $-0.02

71

Originally posted by davepermen:
well. and then explain me how to fit that into a dx9 test.

By using ps_2_x.

72

3DMark is a toy benchmark; I don’t pay too much attention to how a synthetic benchmark comes up with its own set of magic numbers. You can run any number of synthetic benchmarks and prove almost anything you want because it all comes down to how you collapse different results from different tests into the final number. Ever seen the Simpsons epsiode to see who’s the best bar keeper? The chick won the first two tests but lost the third test; but because the third test was weighted 95%, Moe won in the end. It’s the same story.

I think graphics vendors face an up-hill battle. I’m sure everyone would agree that graphics chips are complicated devices, but the tests are trying to enforce a uniform execution model across all hardware vendors, and that is, in my opinion, Not A Good Thing.

For example, CPUs are also complicated devices but CPU tests are ~targetted~ for a particular architecture. How so? Well, some the tests involve compiling code on a CPU and running it to see how well the code executes. The code that is being used to test the cpu has been ~scheduled for that cpu~ by the compiler. Instruction scheduling (+ cache line prefetching + a myriad of other optimisations that rely on knowing the architecture of a CPU) is ~not cheating~, but it IS hardware specific optimisations. I am not saying that this is a bad thing for CPUs. My point is that 3DMark’s approach to trying to come up with the same execution flow to all hardware pipes is not a fair test.

At the end of the day, the only results a buyer should be interested in is how well the card performs for what they ask of it.

FYI, I use both an ATI and an nVidia card.

cheers
John

73

Finally we are close to conclusion!

One thing is for sure, the more complicated pipes we get, the harder becomes task to write unified tester, it’s a bit like Intel Celeron vs Intel Pentium vs AMD MHz value (and matching FSB’s values), but the pipe structure for CPU is a bit more clearer than one of GPU.

Talking about unsuccesfull FX line, I’d prefer FX5200 over Radeons entry lavel card, & FX 5600 isn’t that bad either (especcialy with new drivers), check tech-report, for more info, reliable site based not only on 3DMark.

[This message has been edited by M/\dm/
(edited 05-29-2003).]

74

I say NVidia should have just kept it simple and done everything in floating point precision. Just stay with two precisions, as is supported in ARB-extended OpenGL and in DirectX. FP16 may not be quite as fast as FX12, but at least you have the advantage of a high dynamic range.

75

Geeze madman, try not to live up to your name.

This isn’t about one vendor vs another.

ALL vendors are tainted (IMHO not by this episode but by others).

There’s no need to act as an apologist for anyone, it doesn’t matter what hardware your current www.opengl.org html is being rendered on, screw it, you don’t owe these guys a dime. Just hitch your wagon to the fastest tow and enjoy the ride. That’s what these guys set out to create, take them up on the offer. Pony up $500 and get the bad assest graphics known to man, love it, REALLY love it but don’t take sides, because any side would tear you a new one financially if they thought they could. Love the competition, that’s your REAL friend.

If someone cheats then call them on it, yea there’s some shady practices but I’ve been introspective and thoughtful in other threads already(don’t laugh), why bother here, you know where I stand on this.

[This message has been edited by dorbie (edited 05-30-2003).]

76

Originally posted by john:
For example, CPUs are also complicated devices but CPU tests are ~targetted~ for a particular architecture. How so? Well, some the tests involve compiling code on a CPU and running it to see how well the code executes. The code that is being used to test the cpu has been ~scheduled for that cpu~ by the compiler. Instruction scheduling (+ cache line prefetching + a myriad of other optimisations that rely on knowing the architecture of a CPU) is ~not cheating~, but it IS hardware specific optimisations.

Is the D3D HLSL compiler built into D3D itself, or is it in the drivers? If it’s in the drivers, using HLSL would go a long way towards this goal, no? Of course we don’t know if 3DMark uses it or not…

– Tom

77

I like your new attitude, dorbie!
Pretty soon you will succumb, and join C++ and myself on the dark side. We’re having jelly for tea.

78

The Direct3d HLSL compiler lives inside the microsoft written D3D-runtime, the driver just gets the pixel/vertex shader assembler. One of the big differences between the Direct3d HLSL and glslang.

79

When glslang comes around which directly targets the underlying hardware we have the perfect tool for fair comparisons. It’s all up the the hardware vendors to make it run as fast as they can on their platform.

80

knackered, jedi’s light saber is looking for yer traitor’s soul! (OT)