WTH is wrong with NVidias 6x.xx drivers ? (solved)

This topic is 4801 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

Recommended Posts

OK, the story: a customer reports heavy performance problems with our 3D system on a GF3. I'm bored, so I decide to investigate the thing. I take my second PC (the only one with Windows), and pop in an old GF4 Ti4800 I still had lying around. OK, that's not a GF3, but our engine uses the same codepath for both, so the problem should be reproducable. AMD XP 2800, Windows 2000, 1GB of RAM, and the said GF4, running on older 53.03 drivers. I load the customers scene, and get around 130 fps. Hmm, seems fine to me. So I check the customers logs, and notice that he used the newest drivers, 66.93, also under Windows 2000. OK, no problem, I download and install them. I restart the engine - and get 11 fps on the exact same view !! WTF. I downgrade to 61.77: 17 fps... Finally back to 53.03: 130 fps. Peformance gets worse, the newer the driver. Great. Something is seriously messed up here. It seems to be a problem with VAR and/or VBO. When using standard vertex arrays, performance is approximately the same on all drivers. When using VAR, performance is near optimal on 53.03, but decreases exponentially within the 6x drivers. VBO again is near optimal on 53.03, and decreases also, but not nearly as bad as VAR. With the VBO codepath, I get around 40 fps on the 66.93 drivers - still better than the 11 fps with VAR, but still lightyears away from the 130 fps of the 53.03 drivers. Did anyone else notice very bad performance on GF3/GF4 cards with the 6x driver series ? That's really a problem, I need to find a workaround asap - other than telling our customer to install 53.03... Any help or hints are appreciated ! [Edited by - Yann L on December 30, 2004 1:58:53 PM]

Share on other sites
I'm windows xp sp2, GF2, with 66.93 installed. If it helps, and if you have some application I can run to test, throw me a linky.

Share on other sites
Quote:
 Original post by Yann LDid anyone else notice very bad performance on GF3/GF4 cards with the 6x driver series ? That's really a problem, I need to find a workaround asap - other than telling our customer to install 53.03...

I noticed a much worse thing with these damn 6x drivers - none (I mean not a single!) of my IM programs worked! I use IM for some proof-of-concept things sometimes and I was shocked. Instead of the expected scenes I just got weird polygon garbage... I only noticed this because I didn't had my Radeon card for a few days and thus installed a GF4 with the latest (and obviously totally messed up) drivers.
Now that I have my Radeon back, everything works as expected again. Before I got this card I also had a GF3 with older drivers and never experienced any problems.

Share on other sites
Quote:
 Original post by owlI'm windows xp sp2, GF2, with 66.93 installed. If it helps, and if you have some application I can run to test, throw me a linky.

Thanks for the offer, but it's a commercial application, so I can't give it out.

Quote:
 I noticed a much worse thing with these damn 6x drivers - none (I mean not a single!) of my IM programs worked! I use IM for some proof-of-concept things sometimes and I was shocked. Instead of the expected scenes I just got weird polygon garbage... I only noticed this because I didn't had my Radeon card for a few days and thus installed a GF4 with the latest (and obviously totally messed up) drivers.

Hmm. I tried an FX5900, and didn't experience any significant slowdown with 66.93.

I guess Nvidia is optimizing every last bit out of their Doom3 performance, while forgetting to keep good performance and compatibility with older hardware. I'm afraid that I might be using a driver codepath that used to be accelerated, but isn't anymore in the new drivers. Gah. If you don't copy Carmacks OpenGL coding style, you're basically fucked. I never had such problems with ATI.

*sigh*

I would still be interested in some more experiences with 6x drivers and GF3/GF4 cards. I would like to make sure it's not a hidden bug in my engine.

Share on other sites
Quote:
Original post by Yann L
Quote:
 Original post by owlI'm windows xp sp2, GF2, with 66.93 installed. If it helps, and if you have some application I can run to test, throw me a linky.

Thanks for the offer, but it's a commercial application, so I can't give it out.

lol, I never expected such a thing :) Is something out there that I can quickly downlad to test VBAs VBOs? I don't have anything of that sort in this machine.

Share on other sites
Can you run VTune and see where the slow-down is?

Perhaps they changed the location of your VBO and VAR memory, so it's now in VRAM instead of AGP? If that's the case, and especially if you don't overwrite every byte of the entire array sequentially, you may be suffering write combiner (fetch buffer) performance penalties when re-combining write lines.

Share on other sites
Quote:
 Original post by hplus0603Can you run VTune and see where the slow-down is?

Will do that tomorrow. I need to find the reason for this.

Quote:
 Perhaps they changed the location of your VBO and VAR memory, so it's now in VRAM instead of AGP? If that's the case, and especially if you don't overwrite every byte of the entire array sequentially, you may be suffering write combiner (fetch buffer) performance penalties when re-combining write lines.

It's always been in VRAM, on purpose. My engine works a little different from other common ones. I use an associative and adaptive caching system for onboard memory management. But for the purpose of this test, all VBO/VAR data was completely static, nothing was dynamically updated. Basically: "bind static buffer, set pointers, render, repeat". The fact that something so simple fails so miserably makes me worry. I even made it fallback to the basic fixed function OGL 1.1 combiner path, without any shaders. No difference.

I might try to strip it down until I isolated the problem in a very small piece of code, that I could upload for public testing (and eventually send to NV, should the bug be confirmed).

Although I'm now 100% positive, that the problem is with VBO and/or VAR. With simple VAs, the performance is equal among all driver versions I tried.

Bah, this thing is frustrating. I'm off to bed now, will test more tomorrow.

Share on other sites
Try the beta drivers.

while i doubt it will magically work, it is worth a shot.

Share on other sites
sorry cant help, but with a gffx5900/5200 ive never had problems with VBO's drivers 5X. -> 7X.
whats your data types, verts3/4 texcoords2 color?
did u check the performance pdf's at the nvidia website, they also have a couple of vbo's ones from memory (not sure if they recommend what data to use)
IIRC theres also a old gf3 performance pdf at the nvidia site (might have it on my HD if u cant find it)

Share on other sites
I experienced almost exactly the same problems...

Except along with poor performance with VBO, there was absolutly massive geometry corruption and problems with render-to-texture.

However, on the other hand, the 50 series drives worked almost fine... The only thing to go wrong was that MSN messenger appeared in my water reflection.. (quite the WTF?!?! [wink])

On the other hand, I've found with gefore FX's, that the performence different between 50s and 60s is absolutly utterly rediculous. Shader heavy scenes can literally run 5x faster.. getting closer to the speed of the equivalent radeon.

But on the other hand again, my former flatmate, found 60s much, much more unreliable with system hangs and reboots and the like.

I also noticed with FX's, the 60 series finally fixed clip planes.. So this may possibly have something to do with it?

yeah.

Share on other sites
Where can I download the 5x.xx drivers? I want to see that too!

Share on other sites
Question: does the scene run faster on newer cards with 53.03?

Share on other sites
If you think that's bad, I got constant BSOD in WinXP with the new 6x drivers. I had to roll back to the 5x series. (this was while playing Half-Life 2)

System:

1ghz AMD Athlon
384 megs of ram
FX5200
WindowsXP SP1

Share on other sites
Their last most recent drivers would crash so bad on me that when I booted back up, Windows couldn't find my video card. O_o Now that is a bad driver.

Share on other sites
Well... I had those drivers, and indeed there where many crashes when running 3D games. I installed the 53.03 drivers now and tuxracer (I LOVE IT!) didn't crash yet, before it crashed after 5 mins or so. I'll try now with Rome: Total War, If I can find the installation CD's. But it could be my video card too; is a FX5200 good enough?
[EDIT] Yep! seems to work!! No crashes anymore

[Edited by - LordMyth on December 30, 2004 5:30:35 AM]

Share on other sites
My GF3 runs best with 44.67 (hacked by StarStorm for extra performance). The performance penalty when downgrading to 6x.xx isn't that bad though.

I don't think I'm running 44.67 now though, I'm probably running some recent hacked Omega driver.

Share on other sites
arf, and they say ATIs drivers are bad, reading some of these horror stories i'm glad I swap years back [wink]

I have noticed however in the past a running theme of complaints about newer drivers not reacting well to older cards, it seems that when they get a couple of chips ahread that they start to muck up performance on the older cards while boosting the newer ones *shrugs*

It does seem strange however that they could screw up something as old as VAR, even on an older card, if it was just VBO i wouldnt be as shocked as they have had problems before now. Maybe they have changed something else where and the driver is mis-detecting the gfx card's ram? maybe even broken something with regards to allocating AGP ram? I'm not sure about the specifics of the VAR extension but if you were asking for VRAM/AGP ram and it couldnt supply it would it just silently fall back to system or would it moan at you?

I'd love to help with the testing however the only machine I've got with a NV card in, and a GF3 no less, currently only has linux installed on it.

Share on other sites
Hmm, seems that more people have some trouble with the 6x drivers. Cheers for the reports, guys. I will do some heavy profiling now, and see where the performance goes. My guts tell me it's either glVertexPointer and friends, or glDrawElements - ie. it gets lost somewhere in the depths of the driver. Well, let's see. For now, I told our customer to install 53.03, and as suspected, he got a tremendeous performance boost. This is ridiculous.

Quote:
 Original post by zedzeeksorry cant help, but with a gffx5900/5200 ive never had problems with VBO's drivers 5X. -> 7X.

It works fine on an fx5900, so I suspect it's a problem with the GF3/4 and >53.03 drivers only.

Quote:
 whats your data types, verts3/4 texcoords2 color?

Pretty much everything is streamed as generic vertex attributes directly into the shaders, so there's no distinction between texcoord, colours, etc. But I made it fall back to the fixed function pipeline, down to the most basic level: 3 floats positional and a packed 32bit colour. According to any old performance FAQ I could find, that's pretty much optimal. I can even try to remove the colour attribute, but I doubt it will make much difference.

Quote:
 I also noticed with FX's, the 60 series finally fixed clip planes.. So this may possibly have something to do with it?

Did they ? Well, there are no clip planes used in that test scene anyway. It's basically just a store shelf with many little packages on it.

Quote:
 Question: does the scene run faster on newer cards with 53.03?

No, it runs the same speed for VAR, and slightly slower using VBO on older drivers. Just the way it should, actually. It's really only the GF3/4 that seems to be affected.

Quote:
 I'm not sure about the specifics of the VAR extension but if you were asking for VRAM/AGP ram and it couldnt supply it would it just silently fall back to system or would it moan at you?

I am actually suspecting something like this. Normally, if the system cannot allocate the requested resources, it will return a NULL pointer. Well, it doesn't, so I assume that it would be OK. Performance, however, is very similar to standard vertex arrays, while it should be around 10x faster. Sounds like a silent fallback to sysRAM, doesn't it ? Well, it isn't - if I disable the range (ie. tell the driver to treat the allocated VRAM/AGP/whatever mystery mem as a standard vertex array), then I fall down to 0.1 fps. That is absolutely normal, if the range was allocated in VRAM, because the CPU needs to constantly push it around the slow AGP bus. So it seems to be VRAM or AGP, but it behaves like system RAM. I'm puzzled.

Share on other sites
Hmm. Since I'm a Linux guy, I don't really know too much about how Windows handles certain things by default. Question: does Win 2000 with service pack, uh, 3, I think (the one that comes with a VS.net 2003 installation) installs an appropriate AGP driver by default, or do I have to do that manually ?

Because if it doesn't, and the >53.03 allocate AGP mem instead of VRAM for whatever reason, then this might explain the slowdown.

Where can I check if an appropriate AGP driver was installed ?

Share on other sites
the AGP driver should be a function of your motherboard chipset drivers, if you've installed them then you should have an AGP driver.

Best bet would be to look under the device manager to see what it reports for the PCI=>AGP bridge (it'll be under System devices).

Share on other sites
AHA ! Now we're coming to something. The profiler showed the hog to be in glDrawElements, as expected.

But, first of all, I had no AGP drivers installed. This didn't really matter, as my engine was exclusively working on VRAM anyway. There might have been some slowdowns when updating the cache, but I never really noticed that, as I was either testing it under Linux or Windows XP - both with a working AGP driver.

Anyway, I just tried to allocate the cache in AGP instead of VRAM, on the 53.03 drivers: down to 13 fps ! So that's probably the reason of the slowdown.

But this also means, that the 6x drivers will always allocate AGP memory, even when I specifically ask for VRAM. That really sucks, since AGP is going to be slower, especially on older AGP x4 cards. But at least I can hope for a somewhat more acceptable performance. I'm downloading the driver right now, and I'll try it combined with 66.93.

Share on other sites
Yann:
I'd strongly suggest you to get in touch with nVidia about this. This might be fixed once brought to their attention, as I seriously doubt they deliberately intended their drivers to be slower. I use ATI so I can't check this issue myself.

Share on other sites
>>3 floats positional and a packed 32bit colour<<

4 color as ubytes? from memory i remember reasing somewhere this mightnt be optimal, try it without the color stuff

Share on other sites
Quote:
 Original post by zedzeek>>3 floats positional and a packed 32bit colour<<4 color as ubytes? from memory i remember reasing somewhere this mightnt be optimal, try it without the color stuff

4 ubyte RGBA is actually the optimal format for colour data on all nvidia hardware (and also ATI, btw).

Anyway, it was the AGP stuff. Following results from my tests, with installed AGP driver. I plugged in a GF4 Ti4400 with AGPx4, in order to be able to assess the speed with a x4 AGP:
           53.03 VRAM      53.03 AGP       66.93 VRAM      66.93 AGPVAR          117 fps        53 fps           56 fps          56 fpsVBO          99 fps         n/a              104 fps         n/a

The conclusion to draw from this:

* The peformance difference between 53.03 VRAM and AGP is due to the slow AGP x4 connection, and is to be expected. That's why I used VRAM on older cards. On x8 or PCI-express cards, the difference will approach zero.

* The 66.93 driver always issue AGP memory when using VAR, hence the exact same (and bad) performance values for VAR. This is a bug, and I will contact nVidia about it.

* 66.93 VBO is much faster even on AGP x4, probably because other than VAR, it (partially) uses VRAM internally. Although still not as fast as 53.03 VAR, the speed is acceptable.

OK, so the mystery is solved. Thanks for your help, and I will get in touch with nVidia about the VAR issue.

Share on other sites
youre correct i misread, i hunted up the paper again it saiz with VBO's dont use RGB color in ubyte format,