It seems that the Futuremark debate and the, so-called, driver optimization war is still going strong. Following a brief truce, both sides have resumed the campaign, with Futuremark initiating the current bout of aggression with a 3D Mark '03 patch released on tuesday. The company was very careful not to point any fingers but claimed that this new patch disables all driver optimizations. Finger pointing however, cannot be avoided when reviewing the way the patch affected 3DMark '03 scores. nVidia boards experienced drops of up to 26.4 per cent while ATI's offerings experienced more performance increases than decreses with the latter peaking at 2.4 per cent. An example of how badly nVidia results were affected can be seen by considering the 5950 Ultra which had a score of 5886 3DMarks and dropped to 5081 following the application of the patch.
So if we were to accept Futeremark's statements about the patch then nVidia were clearly the company that was utilising driver optimizations. nVidia, of course, replied with a statement, only for Futuremark to respond with one of their own. So this will go on for some time and those with less than a relevant PhD will not get much. No one can be blamed however for gaining some form of perverted enjoyment out of this mess.
Here is the full nVidia statement:
With the introduction of the GeForce FX - we built a sophisticated real-time compiler called the Unified Compiler technology. This compiler does real-time optimizations of code in applications to take full advantage of the GeForce FX architecture.
Game developers LOVE this - they work with us to make sure their code is written in a way to fully exploit the compiler.
The end result - a better user experience.
One of the questions we always get is what does this compiler do? The unified compiler does things like instruction reordering and register allocation. The unified compiler is carefully architected so as to maintain perfect image quality while significantly increasing performance. The unified compiler a collection of techniques that are not specific to any particular application but expose the full power of GeForce FX. These techniques are applied with a fingerprinting mechanism which evaluates shaders and, in some cases substitutes hand tuned shaders, but increasingly generates optimal code in real-time.
Futuremark does not consider their application a "game". They consider it a "synthetic benchmark". The problem is that the primary use of 3DMark03 is as a proxy for game play. A website or magazine will run it as a general predictor of graphics application performance. So it is vital that the benchmark reflect the true relative performance of our GPUs versus competitors.
And, while they admit that our unified compiler is behaving exactly the way it behaves in games and that it produces accurate image quality, they do not endorse the optimizations for synthetic use. Hence, Futuremark released a patch that intentionally handicapped our unified compiler.
So, we advocate that when reviewers are using 3DMark as a game proxy, they must run with the unified compiler fully enabled. All games run this way. That means running with the previous version of 3DMark, or running with a version of our drivers that behave properly.
And here is the Futuremark reply:
3DMark03 does not talk to graphics driver, it talks to the DirectX API, which then talks to the driver. Thus, it is impossible for the application to disable GPU compiler!
The only change in build 340 is the order of some instructions in the shaders or the registers they use. However, new shaders are mathematically equivalent with previous shaders. A GPU compiler should process the old and the new shader code basically with the same performance. Of course, if there are application specific optimizations in the driver that depend on identifying a shader or parts of it, then you might see performance differences because these optimizations will not work if the driver is not able to detect the shader.
Let's also repeat that 3DMark specific driver optimizations are strictly forbidden in our run rules because they invalidate the performance measurement and the resulting score is not comparable to other hardware.
Thus, the right conclusion is that the new version of 3DMark03 is now very suitable for objective performance measurement between different hardware. Futuremark's Benchmark Development Program is an open program where we also work a lot with drivers and actually help IHVs in getting more performance via generic enhancements that benefit all applications. We would like to invite you to apply to the program.
