Has anyone encountered such a situation where,

* The D3D9 debug runtime reports memory leaks on application exit, listing all unfreed allocations with their AllocIDs.
* Tell D3D to break on one of the allocations using the "Break on AllocID" feature of DX control panel.
* With the D3D symbols loaded, we get a complete call stack when the break happens. It turns out the allocation is for a vertex declaration.
* We are 100% sure the very vertex declaration IS released during the shutdown process, and the ref count reaches 0 after the release.

Is it possible that the D3D runtime is reporting wrong AllocIDs? Or is the "Break on AllocID" feature unreliable?

Well, the D3D debug runtime reports exactly the same allocIDs every time I run. I think this proves that our code is[font="helvetica, arial, verdana, tahoma, sans-serif"][size="2"][color="#282828"] [/font][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

[background=rgb(250, 251, 252)]deterministic.[/background][/font]

I find it hard to find D3D leaks using allocID, it's way too undetermined.

Try to remove components from your program 1 by 1:
Remove Skybox/etc and see if leak still happens.
Remove all Terrain and test again.

This way you'll find which system causes the issue. If it's not too big you can easily recheck whole thing. If it is big, then just start disabling it's parts.

@Ripiz: Thanks for the tip. That's what I am starting to do now. I've completely lost in the irrelevance of the allocIDs.

As I continue to investigate the problem, I found something weird.

I used PIX to record a diagnostic log. At the very last frame, where all D3D objects are being released, I found out that one of our cubemaps is released this way:

Frame 000098 ........PRE: IDirect3DCubeTexture9::Release()
Frame 000098 ........POST: <0> IDirect3DCubeTexture9::Release()

While other cubemaps are released like this:


Frame 000098 ........PRE: IDirect3DCubeTexture9::Release()
Frame 000098 ............PRE: RemoveObject(D3D9 Cube Texture, 0x062AAD30, 0x0018E6E0)
Frame 000098 ............POST: <> RemoveObject(D3D9 Cube Texture, 0x062AAD30, 0x0018E6E0)
Frame 000098 ............PRE: RemoveObject(D3D9 Surface, 0x06272F60, NULL)
Frame 000098 ............POST: <> RemoveObject(D3D9 Surface, 0x06272F60, NULL)
Frame 000098 ............PRE: RemoveObject(D3D9 Surface, 0x06272ED0, NULL)
Frame 000098 ............POST: <> RemoveObject(D3D9 Surface, 0x06272ED0, NULL)
Frame 000098 ............PRE: RemoveObject(D3D9 Surface, 0x06272E40, NULL)
Frame 000098 ............POST: <> RemoveObject(D3D9 Surface, 0x06272E40, NULL)
Frame 000098 ............PRE: RemoveObject(D3D9 Surface, 0x06272DB0, NULL)
Frame 000098 ............POST: <> RemoveObject(D3D9 Surface, 0x06272DB0, NULL)
Frame 000098 ............PRE: RemoveObject(D3D9 Surface, 0x06272D20, NULL)
Frame 000098 ............POST: <> RemoveObject(D3D9 Surface, 0x06272D20, NULL)
Frame 000098 ............PRE: RemoveObject(D3D9 Surface, 0x062AAF18, NULL)
Frame 000098 ............POST: <> RemoveObject(D3D9 Surface, 0x062AAF18, NULL)
Frame 000098 ........POST: <0> IDirect3DCubeTexture9::Release()

The main difference here is that the latter call to IDirect3DCubeTexture9::Release() actually removed the texture object and all of its surfaces, but the first call did nothing.

And both of them returned a reference count of 0(<0>). That's why our sanity check on the return value of Release() can't catch this type of leak.

Of course, PIX marked the texture and the surfaces of the first cubemap as never destroyed, which means they are leaking.

Are you rendering to cube-maps at any point?

Nope. But I found that if we create the cubemap but don't use it at all(i.e. bind the cubemap using SetTexture()), it will be properly released. However, we do unbind all textures at the end of each frame.

Phew, I finally get the problem solved.

We capture the states of the whole pipeline in a StateBlock before rendering our UI, and restore the states after it's done.

However, we didn't properly call Release() on this IDirect3DStateBlock9 when the application exits.

According to DirectX docs, plus an educated guess, I believe that this StateBlock holds a reference to basically every resource that's current bound to the driver, including all vertex streams, textures for all sampling stages, vertex and pixel shaders, etc. If you fail to release the StateBlock, all these resources won't get released. This perfectly explained why some of our vertex declarations and cubemaps leak even if we called Release() on them.

Thank you guys for providing valuable opinions along the way!