Sign in to follow this  
myvraccount

Stream Leaks in SharpDX

Recommended Posts

Note:  I chose SlimDX for a topic prefix, but actually I'm using SharpDX (that wasn't an option in the list), but the code is virtually identical.  I'm using C# in VS.NET, and no other API that would be relevant to these graphical issues.

 

Anyway, the problem I'm having is that when I run my program for a couple minutes, it crashes with an exception that says I have a memory leak.  It's a very educated guess that it's being caused by these stupid DataStream objects, because there's really nothing else in the code that could possibly be causing it.

 

I need to be able to change the vertices within the stream at any time, so I tried setting the position to 0 at every frame and rewriting all the data, but that doesn't seem to work.  So I tried disposing of the stream and recreating it and rewriting the data every frame, but I still get the memory leak.

 

How exactly am I supposed to use these streams and still be able to modify my data?  Isn't there a better way to store the data as just an array or something and then use that instead of a stream, and change the values whenever I want?  I NEED to be able to move vertices in any way I want at any time.

 

Unfortunately, I really can't provide source code, so please don't request it.  But this is more of a conceptual issue, not specifically about code.

Share this post


Link to post
Share on other sites

I'm using SlimDX targeting Dx9 (your mileage may vary), and an example of how I use the DataStream object is as follows:

 

            using (var dataStream = _indexBuffer.Lock(0, sizeInBytes, LockFlags.Discard))
            {
                dataStream.WriteRange(_indices);
                _indexBuffer.Unlock();
            }

My understanding of DataStreams is that they are temporary objects that are used to get data in and out of the D3D objects. 

 

So with that in mind, I keep an in-memory copy of the raw data in an array (_indices in the above code), and when that is updated, I call the above code to commit those changes to the actual index buffer.

 

Share this post


Link to post
Share on other sites

Well you seem to do it in a totally different way than me!  I'm using it for storing triangles to display, and doing stuff something like this (I don't have the code in front of me so it's paraphrased from memory):

 

 

DataStream s = new DataStream(sizeof(float) * vertexCount);

s.Position = 0;

for(int i = 0; i < vertexCount; i++)

{

    s.Write(new Vector3(x, y, z));

    // More vectors if it's a more complicated vertex data format

}

s.Position = 0;

// Draw it here

s.Dispose();

 

 

I think this sumarizes it fairly well.  I do the whole thing inside a loop so it happens every frame.  The reason for the two s.Position = 0; lines is because it seems like I had to start at the beginning to write from there, and then again to draw starting with the first point.  I guess my worry is that it isn't being properly disposed, so it keeps making more and more memory allocations until it overflows, but I don't know why that would be.

 

Honestly, I don't even know where the exception is being thrown, and if it's happening form inside these stream functions, I can't break there, so I'm not sure how to prove exactly where it's happening (do you know how?), but I assume that it must be because of the streams, since they're the only thing in my program that keeps allocating more memory.  I can't run it for more than a few seconds at a time and it's really slowing down my progress of testing all other bugs!

 

 

Also, in your example, what data type is your _indexBuffer, and what exactly are you doing with it?  You don't ever dispose of your stream or set the position?

Share this post


Link to post
Share on other sites

s.Write(new Vector3(x, y, z));

 

 

This is likely the source of the problem. It appears you create (new) vectors without deleting them. You should be able to determine if that's the problem by using a fixed value (i.e., don't use new). It, of course, won't render properly but it will tell you if that's the source of the leak.

Edited by Buckeye

Share this post


Link to post
Share on other sites


This is likely the source of the problem. It appears you create (new) vectors without deleting them

 

Vector3 is a value type, so new in this case is just allocating on the stack. It's not the source of the leak.

 

 


DataStream s = new DataStream(sizeof(float) * vertexCount);
s.Position = 0;
for(int i = 0; i < vertexCount; i++)
{
    s.Write(new Vector3(x, y, z));
    // More vectors if it's a more complicated vertex data format
}
s.Position = 0;
// Draw it here
s.Dispose();

 

Why are you not using "using" syntax? It may not be the source of your leak, but if any exceptions are thrown in there, the DataStream won't be Dispose'd. It should be written like this:

 

using(DataStream s = new DataStream(sizeof(float) * vertexCount))

{

    // blah blah blah

}

Share this post


Link to post
Share on other sites

Buckeye, that's a good point.  phil_t, are you sure the stack isn't overflowing then?

 

I didn't know I was supposed to be using using.  Using will set the object = null when it's done, right?  But it won't actually call the Dispose method, will it, and isn't that necessary?  I was just following an example I had seen, and I didn't realize people used the using keyword to do this particular thing.

Share this post


Link to post
Share on other sites

Buckeye, that's a good point.  phil_t, are you sure the stack isn't overflowing then?

 

 

 
Using new to create a stack-allocated variable in a loop like that cannot "overflow the stack." There's only space for one Vector3 allocated on the stack and it's constantly reused in each iteration of the loop. The use of "new" here isn't the problem (it's C#; you basically must use new to create new instances).
 
Using will set the object = null when it's done, right?  But it won't actually call the Dispose method, will it, and isn't that necessary? 

 

 
No, a using block will not assign the "used" variable null at the end of the scope. It will, however, ensure Dispose is called. It's basically syntactic sugar for a try/finally block. See the documentation for more.
 
 
As for your actual problem, what is the actual type of exception thrown and the actual text of the error message? Copy-and-paste it, don't retype it, as much as possible. If you suspect you have a memory leak it's a good idea to run the program through a memory profiler, such as the one built-in to Visual Studio or the excellent third-party one provided by Red Gate Software (the ANTS memory profiler; it has a free evaluation). Also:

Honestly, I don't even know where the exception is being thrown
 

 

Debug -> Exceptions, make sure "break on throw" is checked for managed exceptions.

and if it's happening form inside these stream functions, I can't break there,
 

 

 

Enable unmanaged debugging in your project options and you can set breakpoints (or get accurate break-on-throw) results inside the SharpDX code.

 

And finally:

 

Note:  I chose SlimDX for a topic prefix, but actually I'm using SharpDX (that wasn't an option in the list), but the code is virtually identical.  I'm using C# in VS.NET, and no other API that would be relevant to these graphical issues.
 
Topic prefixes are less important than titles; you should put "SharpDX" in the title. I'll edit it for you.
Edited by Josh Petrie

Share this post


Link to post
Share on other sites

If you're not using the "using" syntax I believe you have to call Close() and then Dispose() on the Datastream to properly clean it up.

 

You don't need both. SharpDX's DataStream implements IDisposable correctly, so it's Dispose method -- where all the clean up logic is -- is sufficient on its own. It inherits from System.IO.Stream, which has a Close method that just calls Dispose. So one or the other is fine, but both is redundant (safe, but redundant).

 

Calling Close will cause the GC to suppress finalization of the DataStream (due to the explicit call to SuppressFinalize in Close), but that does nothing because DataStream doesn't have a finalizer.

 

Close is most vestigial, an artifact from the pre-IDisposable days.

Edited by Josh Petrie

Share this post


Link to post
Share on other sites

I personally use DataStream extremely irresponsibly. I only recently made sure to Close or Dispose of any of them, and I have never had this problem.

Edited by cephalo

Share this post


Link to post
Share on other sites

cephalo:  But did you keep creating new ones all the time?  My problem is I don't know how (and haven't had a chance yet to try any of these suggestions) to initialize data that I can change on the fly, so I've been just re-creating everything at every frame, so I could put it in a new position, then render it and replace it next frame.

Share this post


Link to post
Share on other sites

Now that I think about it, I think I'm using DataStream for initializing buffers, but DataBox to map /unmap the dynamic ones, maybe that's why I didn't run into obvious problems.

 

I get the feeling from your posts that you are migrating from DX9 to DX11. Let me tell you that the change is not-at-all trivial. I was pretty comfortable with DX9 and though I'm just an amateur in any case, it took me a whole year to get that comfortable with DX11. They are so different from one another you almost have to forget what you know. Maybe you should put your project aside and implement some DX11 tutorials in SharpDX. Everything is different in DX11.

Share this post


Link to post
Share on other sites

I'm unsure if you're using DX9 or DX11 so I'll just assume its DX11. I've been using DX11 + SharpDX for a few months now.

There are a few ways to update a buffer.

One way is via Context.UpdateSubResource(data, buffer). I have no idea if this works on non-writable buffers though.

So far I've used map/unmap without any problems on various buffers types (vertex/index/structured/stream, staging/dynamic, etc).

For dynamic buffers I use:

// Map
DataStream mappedData;
Context.MapSubresource(buffer, MapMode.WriteDiscard, MapFlags.None, out mappedData);

// Write data
mappedData.WriteRange(dataArray, offset, count);

// Cleanup + unmap
mappedData.Dispose();
Context.UnmapSubresource(buffer, 0);

For this to work, your (vertex) buffer needs to be setup with the following description:

    BufferDescription desc = new BufferDescription()
    {
        BindFlags =  BindFlags.VertexBuffer,
        Usage = ResourceUsage.Dynamic,
        OptionFlags = ResourceOptionFlags.None,
        SizeInBytes = _stride * elementCapacity,
        CpuAccessFlags = CpuAccessFlags.Write, //Dynamic buffers will only accept write flags
    };

   
Keep in mind dynamic buffers can only ever be written to by the CPU, and only read by the GPU. Adding CpuAccessFlags.Read to the description will cause an exception (usually "the parameters are incorrect").

The other way of updating a buffer is via a staging buffer. These are better if you'll be updating buffer data less than once per frame.
Staging buffers can be both written and read to by the CPU, but have a massive tradeoff. They cannot be used by the GPU for anything except resource transfer (staging).

 

Usually they're paired with a "default" buffer which the staging buffer will transfer to (or from):

BufferDescription desc = new BufferDescription()
{
    BindFlags =  BindFlags.VertexBuffer,
    Usage = ResourceUsage.Default, //NOTE: Default usage
    OptionFlags = ResourceOptionFlags.None,
    SizeInBytes = _stride * elementCapacity,
    CpuAccessFlags = CpuAccessFlags.None, //NOTE: no cpu access
};

For me, the FPS gain from rendering with a default buffer has been significant. But the frame rate tends to die pretty fast if you constantly update the buffer via staging buffers once per frame or more (which is where dynamic buffers are better).

 

Map/unmap can be used to fill a staging buffer as such:

// Map - Note the "Write" map mode.
DataStream mappedData;
Context.MapSubresource(stagingBuffer, MapMode.Write, MapFlags.None, out mappedData); //MapMode.Read if you want to read from it.

// Write data
mappedData.WriteRange(dataArray, offset, count);

// Cleanup + unmap
mappedData.Dispose();
Context.UnmapSubresource(stagingBuffer, 0);

Since staging buffers arn't bound to the pipeline for use, it is safe to use a Write or Read map mode. I'm unsure if WriteDiscard works on a staging buffer, but i'd expect it to wipe all of the data in the buffer.

 

The description for a staging buffer is generally the same for every staging buffer, since they are a bindless resource:

BufferDescription desc = new BufferDescription()
{
    BindFlags =  BindFlags.None,
    Usage = ResourceUsage.Staging,
    OptionFlags = ResourceOptionFlags.None,
    SizeInBytes = _stride * elementCapacity,
    CpuAccessFlags = CpuAccessFlags.Write, //You can add read flags here if you're going to be reading.
};

Before the data is usable by the GPU, it must be copied to the buffer you're going to be rendering/using via Context.CopyResource.

Context.CopyResource(stagingBuffer, vertexBuffer);

You can also use them in reverse for copying data from a buffer on the GPU, to a staging buffer. But this is more use for things like structured or stream-out buffers than vertex or index buffers.

 

If you really wanted to pull data out of a dynamic buffer, you should be able to do:

Context.CopyResource(dynamicVertexBuffer, stagingBuffer);

Then use map/unmap on the staging buffer with MapMode.Read.

 

Keep in mind that dynamic buffers can only be read by the GPU, so I'm not sure copying from a staging buffer to a dynamic buffer will work, since that would involve the GPU writing to it.

 

All of this will work the same for index buffers too (and most of the other buffer types).
 

Hopefully this helps, and apologies if I derped. Its my first post here. If you have any other questions, drop them here and I'll try my best to help if I'm around. smile.png

Edited by Syncaidius

Share this post


Link to post
Share on other sites

cephalo, yes I'm using version 11, but I didn't just migrate.  I use DX on and off and have for a long time, but I never did anything really elaborate with D3D, so I may be running into some newbie errors.

 

Syncaidius, that's a lot of info, and I'll go through it and try it when I get a chance (probably tomorrow), but thanks.

Share this post


Link to post
Share on other sites

Well so far I've verified that it is in fact the vertex buffer that's causing the memory leak, so I was definitely right about that.  So I'm using a using, sort of how Postie is, but I don't have an index buffer, but a vertex buffer instead, and I'm not actually locking at the moment.  I'll try Syncaidius' more complicated methods if this doesn't work, but right now, here's what I'm doing:

 

 

void drawLine(Vertex3 lineColor, Vertex3 lineBegin, Vertex3 lineEnd)

{

    // Set both points, converting to vector4 for size alignment.

    Point[] points = new Point[2];

    points[0] = new Point();

    points[0].location = new Vector4(lineBegin.X, lineBegin.Y, lineBegin.Z, 1.0f);

    points[0].color = new Vector4(lineColor.X, lineColor.Y, lineColor.Z, 0.0f);

    points[1] = new Point();

    points[1].location = new Vector4(lineEnd.X, lineEnd.Y, lineEnd.Z, 1.0f);

    points[1].color = new Vector4(lineColor.X, lineColor.Y, lineColor.Z, 0.0f);

    using(vertexStream = new DataStream(sizeof(float) * 8 * 2, true, true) // Set up the vertex stream for 2 points with 8 floats each

    {

        vertexStream.writeRange<Point>(points);

        context.InputAssembler.PrimitiveTopology = PrimitiveTopology.LineList;

        buffer = new Buffer(dev, vertexBuffer, sizeof(float) * 8 * 2, ResourceUsage.Default, BindFlags.VertexBuffer, CpuAccessFlags.None, ResourceOptionFlags.None, 0);

        context.InputAssembler.setVertexBuffers(0, new VertexBufferBinding(buffer, sizeof(float) * 8, 0));

        context.Draw(2, 0);

    }

}

 

 

I'm calling this function from inside the loop, once for each line for each frame.  I know for a fact that this function is causing the OutOfMemoryException.  Is there any obvious reason why?

Edited by myvraccount

Share this post


Link to post
Share on other sites

buffer = new Buffer...

 

You're creating a new buffer every frame. Use map and unmap instead. You want to create a buffer on the GPU, and reuse that every frame.

Edited by cephalo

Share this post


Link to post
Share on other sites

You appear to constantly allocate buffer objects and never release (Dispose) of them.

 

Not only will this create unbounded resource growth that could eventually lead to an OOM condition on the CPU or GPU, it's a fantastically slow and inefficient way to render lines. You should create one (dynamic) buffer containing all the line geometry and fill it up, then render it.

Share this post


Link to post
Share on other sites

Well for what I'm doing, I wasn't really worried about efficiency at the moment.  I just wanted to get it to stop crashing in the simplest way possible.  So now I gave the buffer its own using, and that fixed it.

 

Be aware that that isn't the proper way to do it. Doing it right is rather easy and a habit well worth getting into.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this