QNAN

I solved it.

I forgot to change the indices of the triangles.

I figured it out, and it was not an easy one:

My rendertarget was in the format of ARGB and cleared to black. The alpha values of the plants then changed the alpha value of the rendertarget. This rendertarget was then rendered onto another rendertarget (in order to support multiple viewports), also cleared black, but with XRGB format. And then the black hits through the alpha values.

I changed the rendertarget to XRGB and the problem was solved. I was experimenting once with antialias and at some point set the rendertarget to ARGB and never changed it back, and then it hit me now like a landmine

I thank you all for your patience and incredible will to help. Even when we did not get close to the actual problem here, I learned alot:

 - Not taking anything for granted, not even a D3DX-file-loader

 - Both editors and loaders can play funny tricks to save space, like storing RGB-layer of pixels with alpha 0 as black (0).

 - I learned that good alpha textures needs similar RGB color across the border, or filtering will create an unwanted result (since I have only used other ppl's textures, this was new  to me)

 - When I go into details on mipmaps, I have to be careful with the same borders on these as well

 - I learned to use PIX!

Thank you all for your help. I wouldn't know what to do without the incredible amount of expertise on this forum.

EDIT:

phil_t:

It was actually the same scenario as before, I had just disabled rendering skybox, terrain and non-alphatested objects, so I wouldn't get an ungodly amount of textures and surfaces in the test.

I stole code for loading .tga from the ATI demo I used ealier. But it still took quite a while to implement, so I really regret that I didn't try PIX earlier.

I didn't know PIX and I guess I shied away because of that. Too bad.

Once I got it up and running, I must say that Im impressed. I can easily see myself getting addicted to it

It does not look like there is anything wrong with the D3DX-loading of the .tga. My ATI-imported code didn't solve the problem either, so I expected that. But what can it be then?

I made a collage of the PIX session I ran. If you are interested in something not shown, let me know. I am also uploading my .PixRun file, if anyone is interested. It is created by the latest DX-SDK (june 2010 - had to download it especially for this, as the DX I run on had an ancient version (DX SDK June 2005).

Here is the PIXRun-file (zipped):

http://www.samtalebasen.dk/Run7.PIXRun.zip

Here is the PIX-collage (BIG!):

I am hard at work testing different approaches. I got drowned in suggestions here

My first attempt will be to write my own loader, so I can be sure D3DX is not messing with me.

I will get back with a full report when I come out of the pixel mines again

EDIT:

Actually Im gonna install PIX first, it looks like it can save me the trouble, if D3DX is actually doing its job properly.

I tried to alter the texture filtering in the shader to Point, with this result:

 
As can be seen, this does not solve my problem. The result is just pixelated.
 
This is how the sampler looks in the shader:

sampler2D g_texturemap_sampler =
sampler_state
{
	Texture = <g_texturemap>;
	MinFilter = Point;
	MagFilter = Point;
	MipFilter = Point;
};

I also tried to plug my texture into an ATI-demo called "Alpha to Coverage", replacing it with one of the other textures. The demo also draws AlphaTesting, which I set it to in the screenshot.

This is how it looks in the ATI demo:

There is no flaws, so I believe that this is not a texture problem - at least not at the file level.

Maybe in my creation?

D3DXCreateTextureFromFileExW(   m_device,
                                "../database/texture/foliage/gniff.tga", // gniff.tga is just a testname - it is the right one
                                D3DX_FROM_FILE,			// width
                                D3DX_FROM_FILE,			// height
                                D3DX_DEFAULT,			// miplevels
                                0,				// usage
                                D3DFMT_A8R8G8B8,		// format
                                D3DPOOL_MANAGED,		// pool
                                D3DX_DEFAULT,			// filter
                                D3DX_DEFAULT,			// mipfilter
                                0,				// colorkey
                                NULL,				// srcinfo
                                NULL,				// palette
                                a_textureDX))			// texture

If it is not any of these, could it then be a renderstate, that Im unaware of? Is there any renderstate, that can cause this?

Thank you for your help guys.