AquaMark 3 - IntroductionIntroduction
Recently we received our preview copy of Massive Development's
upcoming and much anticipated DirectX 9 (DX9) benchmark called, AquaMark 3 - The reality benchmark
. It is based on the Krass engine which is used with such games as Aquanox and Aquanox 2.The main purpose of this benchmark which will be released to public for free on the 15th of this month is to measure the gaming performance of current and future graphics cards for the next 12 months based on DX9 technology - such as NV35 and R350.This benchmark comes at a rather interesting time, actually. Not only has benchmarking in the tech community been put under the spotlight over the past 8 months with the need for more accurate benchmarks, if you have been keeping up with the news over the past week, you would have heard problems facing nVidia with Valve's Half Life 2 game and their not-so-brilliant GPU DX9 performance against ATI's Radeon 9800 Pro which shined at ATi's recent Shaders Day event as much as Michael Schumacher would if he just won a F1 race in Germany. At this event, Gabe from Value showed various media benchmarks of Half Life 2 with various cards from nVidia and ATI.In certain benchmarks, ATI's Radeon 9800 Pro was twice as fast as nVidia's GeForce FX 5900 Ultra with nVidia's GeForce FX 5200 and 5600 cards being at such low frame rates which would render the game unplayable. Shortly after the event, nVidia issued an official response about the claims asking why Gabe and the rest of the team from Valve did not use the latest beta Detonator 50 drivers which they apparently had over two weeks before the event, which nVidia say are optimized for better DX9 performance - the same technology Half Life 2 and AquaMark 3 are based on. We will not even bother guessing why Valve did not use the latest drivers as it is not our job to do so, and really it is irrelevant at this point in time.While taking a preview look at AquaMark 3, we thought it might be interesting to compare a reference GeForce FX 5900 Ultra 256MB direct from the folks at nVidia with the latest official drivers on their website at time of publishing, Detonator 44.23, and the upcoming Detonator 50 series (which will be released at the end of the month which nVidia's Brian Burke promised in their response would increase DX9 performance) right up against a PowerColor Radeon 9800 Pro with only 128MB of RAM onboard. As well as performance figures, we have many different screenshots captured through AquaMark 3's intelligent screen capture service with AA (Anti-Aliasing) and AF (Anisotropic Filtering) disabled and enabled for those graphics gurus out there who which wish to see the difference offered up by Massive.Let's get to it - read on!
AquaMark 3 - The reality benchmarkThe reality benchmark
Before getting to the issue at hand regarding nVidia GPU performance under DX9, we will take a brief look at what AquaMark 3 is all about and what Massive have to offer.As mentioned in the introduction, AquaMark 3
is a DX9 benchmark based on the Krass engine which is used with such games as Aquanox and Aquanox 2 which will become available to the public, free of charge, on the 15th of this month by German based company Massive Development
.Since this article is not purely focused on AquaMark 3, we will not be spending a lot of time looking deep into the program, just the basics. Massive offer AquaMark 3 for free (basic version) as mentioned, however if you wish to access advanced options in the benchmark you have the option of paying for four different levels of licenses, these include:- AquaMark 3 Professional- AquaMark 3 Professional Plus- AquaMark 3 Commercial- AquaMark 3 Commercial PlusThe basic version
, which will be for free, still includes a lot of options but you are very limited in what you are able to test. You will only be able to test with the default settings - 1024x768x32, with no AA and 4x AF.The professional version
is similar to the basic version except you have much more freedom to choose how you wish to test; in fact, your options are unlimited. You also get access to more special features such as the screen capture service which automatically takes a screenshot every 500 or 1000 frames into the test and dumps them to your HDD in high quality *.TGA format at around 5MB a pop.Then we have the professional plus
version which offers the same features as professional but with the ability to run multiple tests as once aka batch testing.Next up is the commercial version
which offers the complete range offered by AquaMark 3 as well as the rights to publish your results in media publications.The final version is commercial plus
and this is the exact same as commercial but gives the ability to run the benchmark on 10 different computers - perfect for websites such as ours.No pricing information is available at this stage; however, we do know that the top of the line commercial plus version will cost $199.99 US.
AquaMark 3 - The reality benchmark continuedThe reality benchmark continued
After installing the 63MB AquaMark 3 file, and placing the provided preview commercial plus license under "\My Documents\AquaMark3\" which tells the game which license you have, we fired up the benchmark in much anticipation. We are greeted with this menu screen after two welcome screens:
To run the default benchmark, we then click the "Select Measurement" option which takes us to the following screen:
Clicking "Start Measurement" under AquaMark Score Measurement begins the benchmark at its default settings, which are 1024x768x32 with no AA and 4x AF. By the way, thankfully Massive disabled the use of the Windows key which can be good and bad - bad when you are trying to take many screenshots like here.If you decide to buy a license, you are able to change settings under the "Options" menu on main menu:
This section allows you to choose all graphics and sound related options for the benchmark which you wish to run. As for color depth, you can choose from 16-bit or 32-bit - no surprises there. Resolution options are 640x480, 800x600, 1024x768, 1280x1024 and 1600x1200. AA can be set from 0x to Level 4 while AF can be set from 0x to 8x in 2x increments. We have a couple other options there which we missed, but they are self-explanatory as well as the "Light" and "Detail" menus which let you choose specifically which DX9 drawing methods you want enabled and disabled.Alright, it must be time to run an actual benchmark! Since most of you, for starters, will only be using the basic version which only offers default benchmarking, we will only concern ourselves with this test today.After selecting "Start Measurement" under AquaMark Score Measurement you are taken to the following screen which takes as the window while the benchmark is loading:
The benchmark begins and we find ourselves deep under the ocean fighting a battle with the unknown with many exciting explosions and as far as we can tell: fisherman fished all the fish out of this part of the ocean.Depending on what configuration you use and the specs of your PC, the benchmark takes anywhere from 2 minutes to 5 minutes (and longer with older systems) per run to complete the entire 5202 frames which encapsulates the entire test - which is much less if you compare it to 3DMark 2001 or 03, however given, AquaMark 3 is not as complex.So, as an example say you average 24.35 frames per second, the benchmark will take a little over 3 and a half minutes to complete - simple math, 5205 divide 24.35 divide 60. The really good thing about this benchmark is that there is very little (a second or less) time to wait between each different style of DX9 element testing, such as many particles, masked environment mapping, large scale vegetation and so on, which simply carries on and does not load a new scene, like 3DMark.Once you have completed the default benchmark, you are presented with the following screen which details the benchmark results, which are very easy to understand. The results are also outputted to a text file in an automatically created folder with time and date under "\My Documents\AquaMark3\". The really funky thing is the Excel results generator which Massive include with every license - you simply choose to import one of the saved text files into their Excel spreadsheet which displays the results very nicely for future reference, as well as to send to friends and publish online, etc.
It is also worth noting that if you click "Submit Results Online", besides uploading your results to the AquaMark 3 database, it generates a HTML website with your results and test system specs which you can save and use how you please.
AquaMark 3 - ScreenshotsScreenshots
I have included four screenshots of the benchmark below for your viewing pleasure. Please click on the images below for a larger version and allow time to load - they are quite large.
Now you have a basic understanding of AquaMark 3, it is time to get back to the issue we began this article with - nVidia GPU performance under DX9!
AquaMark 3 - Benchmarks - NV under the spotlight and Test System SetupsNV under the spotlight
From the beginning after receiving our preview copy of AquaMark 3, I never had the intention of naming this article "AquaMark 3 - Time for nVidia to face the reality?
" nor to have a dig at our good friends from nVidia. I was purely going to tell you about AquaMark 3 and if it serves as a reliable DX9 benchmark for modern graphics cards. However, I found something far more interesting I could talk to you about today.It was after nVidia gave us access to their upcoming Detonator 51.75 beta drivers that I thought it might be interesting to compare a reference GeForce FX 5900 Ultra with 256MB of RAM with the latest official drivers on nVidia's website at time of publishing, Detonator 44.23, and the upcoming Detonator 50 series which will be released at the end of the month as told in the introduction, which nVidia say will improve DX9 performance. We put the NV35 up against a PowerColor Radeon 9800 Pro with only
128MB of RAM with ATI's latest drivers, Catalyst 3.7 - the 128MB less RAM on the Radeon must be kept in mind throughout the rest of this article for reasons which will become more obvious why later on.We decided to use an Intel based system to test DX9 performance on both cards with a Pentium 4 "C" 2.4GHz processor under AquaMark 3 and the Mother Nature DX9 test under 3DMark03. We didn't use a higher clocked CPU because not many people can afford to use the latest 3GHz or 3.2GHz processors from Intel, but more likely clock speeds around the 2.4GHz range which will give you a better chance to compare our results with your system. The full system specs are below for both ATI and nVidia cards:ATI Test System SetupProcessor
: Intel Pentium 4 2.4GHz "C" (800MHz FSB)Motherboard
: ABIT IC7 MAX3 (Supplied by ABIT
: Kingmax 2x 256MB DDR-400 (Supplied by Kingmax Australia
: Maxtor 80GB SATA-150 (Supplied by Altech Computers
: PowerColor Radeon 9800 Pro 128MB
(Supplied by Altech Computers
: ATI Catalyst 3.7Operating System
: Microsoft Windows XP Professional SP1nVidia Test System SetupProcessor
: Intel Pentium 4 2.4GHz "C" (800MHz FSB)Motherboard
: ABIT IC7 MAX3 (Supplied by ABIT
: Kingmax 2x 256MB DDR-400 (Supplied by Kingmax Australia
: Maxtor 80GB SATA-150 (Supplied by Altech Computers
: Reference nVidia GeForce FX 5900 Ultra 256MB
(Supplied by nVidia
: Detonator 44.23 and Detonator 51.75 BETAOperating System
: Microsoft Windows XP Professional SP1As far as the system setup went, we left all driver settings at their defaults. The only driver option change we made was disabling V-Sync and AA and AF settings were left application specific. The memory controller was set to run at DDR-400 with the Kingmax RAM running SPD. No ABIT performance acceleration options were enabled in the BIOS. The AGP aperture size was set to 256MB for both systems. For all tests under AquaMark 3, triple buffering and V-Sync were disabled with the gamma option being left at default. Since the benchmark is new, we ran most of the same tests two or three times to check for consistency. Massive have done a fabulous job with this benchmark - most repeats of the benchmarks showed exact same results and when they were different, they were under one frame per second different from the previous run. Thanks Massive!
Can nVidia make up ground in the DX9 department by relying only on software optimizations in the Detonator 50 series of drivers? Read on and find out just what the story is for nVidia as we weave our way down the Yellow Brick Road.
AquaMark 3 - Benchmarks - AquaMark 3 - DefaultAquaMark 3 - Default- 1024x768x32
In the first of our benchmarks we see that the new Detonator 51.75 drivers offer around 8% gain over the older drivers with no AA. The Radeon 9800 Pro 128MB is only about 2% behind the 256MB monster FX.- 1280x1024x32
Here we see the increase in performance in the new drivers is 8% with Radeon being about 5% slower.- 1600x1200x32
In this benchmark, as we enter the high resolution range, the new drivers offer a fairly significant 8% increase in performance with the Radeon almost 10% behind.We can see with these three benchmarks that the Radeon becomes slower and slower compared to FX as we increase the resolution.
AquaMark 3 - Benchmarks - AquaMark 3 - High DetailAquaMark 3 - High Detail- 1024x768x32
In the first of our all important AA benchmarks, we see that the Detonator 51.75 drivers offer no improvement over the older drivers - in fact, the new drivers are almost 9% slower! This could indicate that nVidia has sacrificed performance for image quality or they have not optimized their AA part of their drivers under AquaMark 3 very well.The highly optimized and trusty Radeon 9800 with 128MB less than the FX is a little over 6% faster, which tells us that ATI have done the far better job of their product under DX9 with AA. FX with only 128MB would be much slower than Radeon.- 1280x1024x32
The difference between the FX with older drivers and the Radeon this time is only about 3% faster with the FX with new drivers struggling to keep pace being a massive 22.7% slower.- 1600x1200x32
The difference between the FX with old drivers and the Radeon is almost nothing at only 0.8%, which could almost be the difference between one benchmark run and another. The story doesn't get any better for nVidia here either as they are almost 26% slower than the Radeon with the new drivers.These sets of AA benchmarks under AquaMark 3 tell us that either nVidia has stuffed up big time with their Detonator 50 series of drivers when it comes to AA or have not yet optimized their drivers for AquaMark 3. To give us some sort of answer to this, we decided to run the Mother Nature test of 3DMark03 which is DX9 compatible - that's on the next page.
AquaMark 3 - Benchmarks - 3DMark03 - Default3DMark03 Mother Nature DX9 - Default
3DMark03 is the latest version of the highly favored 3DMark series. By combining full DirectX9.0 support with completely new tests and graphics, 3DMark03 continues the legacy of being industry standard benchmark.- 1024x768x32
In 3DMark03's only DX9 test, Mother Nature, the difference between the new and old drivers is very small. The Radeon is about 7% slower, what does this mean? Keep reading.- 1280x1024x32
In this benchmark we see our first tie - the FX with old drivers is performing exactly the same as the FX with the new drivers. The Radeon is behind by almost 5%.- 1600x1200x32
Again we see the FX with both new and old drivers performing exactly the same with the Radeon only a mere .4 frames per second slower.Keeping in mind the Radeon has 128MB less than the FX, in this set of benchmarks it has been able to stand up very strong with its head held high. What does 3DMark03 AA have in store for us? Read on!
AquaMark 3 - Benchmarks - 3DMark03 - High Detail3DMark03 Mother Nature DX9 - High Detail- 1024x768x32
In the first of our high quality tests of AA under 3DMark03 and possibly the most important of the lot, we see the Radeon killing the FX with new drivers by an amazing 25%.The increase in performance of the new drivers over the old drivers is only about 5% - a nice increase, but is it enough to make nVidia competitive against ATI?- 1280x1024x32
As we increase the resolution, the Radeon is still way ahead by 24%. The new drivers offer a slightly higher performance increase of 5.3% over the older drivers - slowly getting better.- 1600x1200x32
When we increase the resolution right up to 1600x1200, the 128MB Radeon is still able to maintain its lead of a little over 25% against the 256MB FX. The Detonator 50 drivers earn a little bit of respect giving around 6% increase in performance over the older drivers in this test, though it is hardly enough to conquer the ATI gladiator.Considering the Radeon, with ONLY 128MB, is 25% faster than the FX with 256MB, nVidia really needs to get to work and improve their driver performance by another 2 or 3 times, at least
, to be anywhere near ATI when we are talking about maximum detail graphics - AA and AF enabled - which will become more and more important from now on. There is only so much performance increase attainable through software optimization. It really may be time for nVidia to face the reality
.The slowest the Radeon ever was against the FX with either driver, in any test, was never anymore than 8% while the FX at its worst was up to 26% slower in some tests. When the FX was slower than the Radeon, it tended to be a lot higher than 8%. The only other thing we can attribute the slow performance to is better image quality - and this just happens to be what we are looking at on the next page. Maybe it's nVidia's answer to poor frame rates, let's find out if this is the case or not.
AquaMark 3 - Image QualityImage Quality Testing
Massive created a very cool feature in AquaMark 3 which allows you capture screenshots at certain frame points in the benchmark - every 500 or 1000 frame points to be exact - an extremely handy feature for hardware reviewers. So, if you choose every 500, you will end up with 10 screenshots and if you choose every 1000, you will end up with 5 since the benchmark has 5202 frames in total.The screenshots are saved in the "\My Documents\AquaMark3\" section as a high quality *.TGA files weighing in at about 5MB each when testing at 1600x1200.Following on from our benchmarks, we took a bunch of different screenshots, 230MB in total, running 1600x1200 with the highest possible quality offered by AquaMark 3 - Level 4 AA and 8x AF with Triple Buffering enabled. We also captured screens at the same resolution at the lowest possible image quality - no AA and trilinear filtering only with Triple Buffering disabled.Let's get the image quality tests under way. Just remember these screenshots do not represent the actual quality of the game since we have saved these images as JPG which does not carry the quality of the TGA format.Image Quality - Highest Possible
Below are three screenshots taken at the 1000 frame point in the benchmark at the highest possible quality level. Please click on the image for full size image;- FX 44.23 - FX 51.75 - Radeon
As far as I am concerned, AA image quality from the old drivers to the new drivers has increase a little bit with the Radeon graphics being a tad worse than nVidia, but only by a small margin. This may explain the drop in performance with the new drivers, but for such a small increase in image quality it seems fairly unlikely.I am not going to give much commentary on image quality since interpretation varies from person to person. Instead open up the images yourself and decide for yourself. And while you're at, decide for yourself if you would really want to sacrifice frames per second for slight image quality improvement. I know what I would choose at this stage.
AquaMark 3 - Image Quality ContinuedImage Quality - Lowest Possible
Below are three screenshots taken at the 1000 frame point in the benchmark at lowest possible quality level. Please click on the image for full size image;- FX 44.23 - FX 51.75 - Radeon
Again, you decide for yourself which is better or worse. I cannot be the judge of that.
AquaMark 3 - ConclusionConclusion
There is no more appropriate time than to say "the benchmarks speak for themselves". Massive's AquaMark 3 does not paint a particularly good picture for nVidia, at all, in the DX9 department.It would be convenient to say the Radeon 9800 Pro we tested had 256MB of RAM onboard but it only had 128MB and that is the scary part. Minus only a few benchmarks, the Radeon was able to knock down the GeForce FX 5900 Ultra with 256MB of RAM onboard, with either driver, time after time in both low and high detail tests, sometimes by very high margins. The new Detonator 50 drivers increase AA performance under 3DMark03 by a fair margin but when it comes to AquaMark 3, something horrible is happening and we aren't 100% what - and nVidia cannot blame it on increased image quality, because as we saw - it has only improved a little and not enough to account for the drop in frame rate. Maybe nVidia has not yet optimized their drivers for AquaMark 3 but the benchmark, albeit unreleased to the public, isn't exactly new to developers and manufacturers, and I would be very surprised if nVidia has not been working in conjunction with Massive to improve their driver support for their card owners.Non-AA performance of the Detonator 50 drivers has increased performance fairly well across the board which nVidia should be congratulated on but it is nowhere near enough to compete with ATI and the raw power and DX9 efficiency the Radeon 9800 Pro is generating. nVidia had it their way for quite some time with DX7 and DX8, but as more DX9 related games and benchmarks start emerging, ATI are going to be able to produce a better name for themselves with the R350, as it is clearly better optimized for DX9 compared to nVidia's flagship which seemingly just can't stand the heat at the present time.I very much doubt nVidia are not aware of this situation, in fact I'd say right now it would be at the top of their list of stuff needing fixing ASAP before it starts affecting their bottom line too much. Until nVidia come out with their next-gen GPU, ATI will lead the graphics card performance market with a GPU which has proven itself here today as being designed much better for DX9 games and applications than nVidia did with their NV35.