Conformance Test

hi,
i think that a conformance test for OpenGL 3 implementations would help developers design their applications for a wider field of plaftorms. they mainly would not have to worry about pitfalls of specific implementations to work around.

this test would not have to be mandatory. parties that took the test successfully could get some kind of official signature/label. this signature could evolve in some kind of quality seal for the devlopers of the implementations and its users. so it also has some kind of public awareness outcome.

i think the khronos group could be the individual to perform such tests and give out the conformance signatures.

what do you all think about that?

Conformance tests already exist. You can’t claim conformance to a Khronos API without passing the conformance test.

Apologies for the long post. Two issues I’d like to raise:

  1. mandatory detailed conformance reporting
  2. community-assessed OpenGL implementation performance/quality

I’d like to see Khronos require all implementors of the OpenGL spec release to the public the full conformance test results for every driver release. Here’s why…

In a thread in “OpenGL.org » Forums » DEVELOPERS » OpenGL coding: advanced » Memory usage”, the issue of texture proxies came up:

I wouldn’t rely on proxies… We had some reports suggesting that IHV don’t take them too seriously.

The poster is clearly trying to do the right thing. Unfortunately, without considerable effort, there’s no way for a reader to verify or refute the claim (which vendors, how badly broken, how recently, etc)

So texture proxies are required by the spec, but because of a report that one or more implementors decided not to support it in a useful way, developers may stop using part of standard OpenGL functionality.

This isn’t just about proxy textures. There are many places the spec leaves behaviour unspecified. Similar doubts may arise elsewhere. If the standard is undermined, everyone loses.

I couldn’t find any information about if or how well texture proxies are tested in the OpenGL conformance suite. I couldn’t find any test results either. If the average reader didn’t have time to develop their own test and run it on a variety of cards, they’d understandably accept the wisdom of the poster, rated as “OpenGL Pro” by the forum software.

If Khronos required that the full OpenGL conformance test results be made public for every driver version (or other implementation) OpenGL implementors released, situations like this might be avoided.

I suspect any competent OpenGL implementor runs the conformance tests prior to a driver release in any case, so not much extra work would be required.

If test results were collated in one place by Khronos (c.f. SPEC and its results), public scrutiny of the results by developers and consumers might “encourage” less “meticulous” implementations in the “right direction”.

If you believe in free markets. :slight_smile:

But conformance to the specification is sometimes not enough…

Perhaps the conformance tests wouldn’t have revealed that in some implementations, texture proxies conform to the API but provide little useful information, create performance issues, etc.

The OpenGL spec is deliberately non-specific in some places, to allow implementors to adopt approaches that match their priorities, architecture, etc. It’s not clear that adding a minimum functional requirement for texture proxies would have been possible or would have helped in the example above.

Developers (and ultimately the consumers they serve) often need more than just mandatory conformance information. Other issues like coverage of optional features (e.g. which texture formats) and performance/quality of implementation (e.g. texture proxies) are important.

Perhaps developers frustrated by continually having to write feature and performance tests of their own could pool their code to create a community-based conformance and performance tests available to all. Add a website for the results, forums for discussion (e.g. right of reply by OpenGL implementors, clarification by specification authors, etc) and you’d have a really useful resource.

This should be applied to extensions too. The extension registry doesn’t give any indication of how widely, how recently or how effectively an extension is supported.

  1. mandatory detailed conformance reporting
  2. community-assessed OpenGL implementation performance/quality

The conformance tests are not public so we don’t know what is tested. Perhaps it tests some basic GL 1.0 features like triangle filling, blending, line rasterizing, point rasterizing.
I’m sure it is not a fine grained debugging process.

#2 happens on these boards and on other forums as well.

Texture proxies : yes, someone wanted to know if the texture will fit into memory. He was getting a false positive.

So texture proxies are required by the spec, but because of a report that one or more implementors decided not to support it in a useful way, developers may stop using part of standard OpenGL functionality.

Personally, I don’t see why proxy textures exists because you can just attempt to create the texture, call glGetError.

All this comes down to a need for a newer, cleaner API.

Conformance testing is a great idea in general but is by no means exhaustive, nor could it be, realistically. You could easily have an implementation pass a conformance test and yet exhibit anomalies under the right conditions.

Golden standards and quality control, OK; but simulating all real world conditions would be next to impossible I imagine…

The best way to ensure that GL 3.0 implementations are good is to make GL 3.0 implementations as simple to write as possible. That will be far more effective than any conformance test.

Are these issues not connected?!
The OpenGL 2.1 API has to be simplified to write a reliable conformance test (there are too many shady formulations in the current spec). To write a good implementation the API has to be simplified (there are too many (and too many outdated) options available for the developer).
To decide when an implementation is good, we need a reliable conformance test - instead of the community sits here and tests every possible combination and whines for years about broken drivers, and eventually decides collectively that the ATI implementation is broken. This opinion may be true for their older drivers but it looks like ATI has been improving a lot lately (and although NV seems better, it is not the golden standard). I’m sure ATI and all other relevant parties (IHVs, OS- and application developers) are interested in both a reliable conformance tests and simpler implementations - and so we just have to continue waiting patiently.

I agree w/ smileyj: “If test results were collated in one place by Khronos”

My humble newbee suggestion:
A site to end all sites where the defacto standard is voted to the top and broken/obsolescence bubble sorts into the bilge.
Or maybe “voting” is actually benchmark tests that are run on every OpenGL ap?

A log/debugger mode that collects all of what was successfully ran on what system specs, to be uploaded to the public site…
Like Steam’s list only a lot more detailed. That would be killer.

Hey, it worked for God and Darwin. Their users are the conscripted into conformance testing too. :slight_smile:

For the record, here is a (non-official) test suite for OpenGL :
http://cgit.freedesktop.org/piglit

1 Like