Skip to main content
GameDev.net gamedev.net
🔒 Locked

[.net] Why is Collections.Generic.List so damn slow?

Started by Scet Jul 22, 2007 at 3:01 PM 28 replies 16.1k views
Original Post
Scet
Scet
I was doing a lot of heavy indexer work in my program and noticed that accessing elements in an Array seemed a hell of a lot faster then accessing them in a List. Anyway I wrote this little test program to time how long they take.

using System;
using Collections = System.Collections.Generic;
using Threading = System.Threading;

namespace Bob
{
	
	public static class Application
	{
		
		[STAThread]
		public static void Main( string[] ArgumentArray )
		{
			int Ticks = 0;
			int[] Array = new int[320];
			Collections.List<int> List = new Collections.List<int>();
			for( int x = 0; x < Array.Length; x++ )
			{
				List.Add( x );
			}
			
			Threading.Thread.Sleep( 1000 );
			Ticks = System.Environment.TickCount;
			for( int i = 0; i < 5000000; i++ )
			{
				for( int j = 0; j < Array.Length; j++ )
				{
					Array[j] = j;
				}
			}
			Console.WriteLine( "Array performance: {0}", System.Environment.TickCount - Ticks );
			
			Threading.Thread.Sleep( 1000 );
			Ticks = System.Environment.TickCount;
			for( int i = 0; i < 5000000; i++ )
			{
				for( int j = 0; j < List.Count; j++ )
				{
					List[j] = j;
				}
			}
			Console.WriteLine( "List performance: {0}", System.Environment.TickCount - Ticks );
			
			Console.ReadLine();
			return;
		}
		
	}
	
}




The results where 2620 for the Array and 13200 for the List. I was under the impression that the .Net 2.0 List class operated similar to a C++ std::vector and that there would be very little(or no) difference between accessing it and an array. I wrote a simple class to mimic the C++ std::vector to see how it could perform.

		public sealed class Vector<T>
		{
			
			private T[] Array = null;
			
			public int Size
			{
				get
				{
					return SizeProperty;
				}
				private set
				{
					SizeProperty = value;
				}
			}
			
			private int SizeProperty = 0;
			
			public T this[int Index]
			{
				get
				{
					return Array[Index];
				}
				set
				{	
					Array[Index] = value;
				}
			}

			public Vector()
			{
				Array = new T[8];
				return;
			}
			
			public Vector( int Reserved )
			{
				Array = new T[Reserved];
				return;
			}
			
			public void Add( T Value )
			{
				Array[Size] = Value;
				Size++;
				if( Size >= Array.Length )
				{
					System.Array.Resize<T>( ref Array, Array.Length << 1 );
				}
				return;
			}
			
			public T[] ToArray()
			{
				if( Size == Array.Length )
				{
					return Array;
				}
				T[] NewArray = new T[Size];
				System.Array.Copy( Array, NewArray, Size );
				return NewArray;
			}
			
		}




Using the same style test as the program above, the result was exactly the same as the Arrays. Can any of the .Net gurus explain how the List class works and why its indexer performance is so bad? I did run the tests multiple times to make sure some other process wasn't hogging the CPU and messing it up and I got around the same results every time. I'm on an Athlon64 at 2Ghz, maybe it was something to do with the way the JIT compiled it. Perhaps other people could run the test program and post their results. Edit: Changed loop size from 10000000 to 5000000 to match other posts. [Edited by - Scet on July 22, 2007 4:18:04 PM]
Headkaze
Headkaze
Have you tried running the tests on a reference type instead of a value type?
Scet
Scet
Quote:
Original post by Headkaze
Have you tried running the tests on a reference type instead of a value type?


Not until now, here are the results from assigning a simple class(5000000 times):

Array: 13719
List: 24515
Vector: 13172

I ran the tests multiple times using 5000000, 10000000 and 20000000 loops, and every time the numbers had the same ratios to each other(10000000 loops equals ~26000 vs ~48000) so I don't think there's some number where the List would overtake them.
Spoonbender
Spoonbender
Quote:
Original post by Headkaze
Have you tried running the tests on a reference type instead of a value type?


The point in the generic list is supposed to be that it handles value types without boxing them, so that shouldn't make a difference.

And I've never benchmarked it, but I'd expected List to perform better than this... Looking forward to see if anyone can come up with explanations :D

(By the way, I assume you are running release builds?)
Scet
Scet
Quote:
Original post by Spoonbender
(By the way, I assume you are running release builds?)


Yes, although the debug version doesn't really change anything other then increasing all the results.

I've run two more tests, one using structs and the other using the same struct as a nullable type(with a ? on the end of the type), both use 5000000 loops.

Results from the struct test. This is weird, I expected it to perform similar to the first one(using an integer) since they're both value types, but the Array beats both the List and Vector.

Array: 4600
List: 12750
Vector: 12700

Results from nullable types test:

Array: 24500
List: 36000
Vector: 30000
Headkaze
Headkaze
Quote:
Original post by Spoonbender
The point in the generic list is supposed to be that it handles value types without boxing them, so that shouldn't make a difference.


That's what I thought too, but worth a try I guess.

I'm also interested in any explaination of this.
SiCrane
SiCrane
I'm putting forward a vote for crappy code generation. I broke your code into functions and looked at the ildasm output. And yes, this is release. The functions:
    static void IndexArray(int [] array) {      for (int i = 0; i < 10000; i++)      {        for (int j = 0; j < array.Length; j++)        {          array[j] = j;        }      }    }    static void IndexList(Collections.List<int> list) {      for (int i = 0; i < 10000; i++)      {        for (int j = 0; j < list.Count; j++)        {          list[j] = j;        }      }    }

IndexArray's MSIL:
.method private hidebysig static void  IndexArray(int32[] 'array') cil managed{  // Code size       35 (0x23)  .maxstack  3  .locals init ([0] int32 i,           [1] int32 j)  IL_0000:  ldc.i4.0  IL_0001:  stloc.0  IL_0002:  br.s       IL_001a  IL_0004:  ldc.i4.0  IL_0005:  stloc.1  IL_0006:  br.s       IL_0010  IL_0008:  ldarg.0  IL_0009:  ldloc.1  IL_000a:  ldloc.1  IL_000b:  stelem.i4  IL_000c:  ldloc.1  IL_000d:  ldc.i4.1  IL_000e:  add  IL_000f:  stloc.1  IL_0010:  ldloc.1  IL_0011:  ldarg.0  IL_0012:  ldlen  IL_0013:  conv.i4  IL_0014:  blt.s      IL_0008  IL_0016:  ldloc.0  IL_0017:  ldc.i4.1  IL_0018:  add  IL_0019:  stloc.0  IL_001a:  ldloc.0  IL_001b:  ldc.i4     0x2710  IL_0020:  blt.s      IL_0004  IL_0022:  ret} // end of method Application::IndexArray

IndexList's MSIL:
.method private hidebysig static void  IndexList(class [mscorlib]System.Collections.Generic.List`1<int32> list) cil managed{  // Code size       42 (0x2a)  .maxstack  3  .locals init ([0] int32 i,           [1] int32 j)  IL_0000:  ldc.i4.0  IL_0001:  stloc.0  IL_0002:  br.s       IL_0021  IL_0004:  ldc.i4.0  IL_0005:  stloc.1  IL_0006:  br.s       IL_0014  IL_0008:  ldarg.0  IL_0009:  ldloc.1  IL_000a:  ldloc.1  IL_000b:  callvirt   instance void class [mscorlib]System.Collections.Generic.List`1<int32>::set_Item(int32,                                                                                                        !0)  IL_0010:  ldloc.1  IL_0011:  ldc.i4.1  IL_0012:  add  IL_0013:  stloc.1  IL_0014:  ldloc.1  IL_0015:  ldarg.0  IL_0016:  callvirt   instance int32 class [mscorlib]System.Collections.Generic.List`1<int32>::get_Count()  IL_001b:  blt.s      IL_0008  IL_001d:  ldloc.0  IL_001e:  ldc.i4.1  IL_001f:  add  IL_0020:  stloc.0  IL_0021:  ldloc.0  IL_0022:  ldc.i4     0x2710  IL_0027:  blt.s      IL_0004  IL_0029:  ret} // end of method Application::IndexList

So the List generic version is actually making function calls in the generated code.
Headkaze
Headkaze
It would be interesting to put the ArrayList in for comparison as well.
Scet
Scet
@SiCrane:

Yes of course there has to be some extra instructions it to be slower. The question is why they're there. I know my Vector class can't do everything the List one can(Remove etc.), but I expected indexers to be at least close in performance.

@Headkaze:

Results from tests simialr to the first one except with an ArrayList

Integers(value type):

Array: 1860
ArrayList: 48400
List: 14500
Vector: 3600

Classes(reference type):

Array: 13640
ArrayList: 32000
List: 24400
Vector: 13600

Yeah, stick to generics.
Headkaze
Headkaze
I found some more tests here

http://www.experts-exchange.com/Programming/Programming_Languages/Dot_Net/Q_22020171.html

System.Diagnostics.Stopwatch looks like a handy way to measure speeds.
remigius
remigius
I can't tell you exactly why the containers generate this code, but maybe you needn't worry about the containers themselves, but your choice of container instead. As you noted yourself, the containers are optimized for typical usage patterns, such as dynamic growth and cheaper insertion/removal of objects. So you should probably use these containers if your algorithm matches this usage pattern. If you have a relatively fixed number of elements, for which fast indexed access is absolutely paramount, arrays probably are the best choice indeed.

I agree it's strange that the List doesn't work like the typical Vector and thus doesn't come closer to array performance, but if you really have as many indexed operarations as your code snippet suggests, you're probably better off using a custom container implementation which gives you more finegrained control anyway.


Quote:
Original post by SiCrane
So the List generic version is actually making function calls in the generated code.


The 1st call looks like a by-product of the fancy 'syntactical sugar' that .NET indexers are. The 2nd call probably is a call to the List.Count property to do index bounds checking. This would kinda prove my point that a container tailored to the OPs needs might be a better choice. Obviously a generic lib implementation needs bounds checking on each access to provide a robust container and sticks to .NET practices like indexers, but a specialist implementation could do this more efficiently and/or forego any checks completely for the sake of speed.


Well, hope this is of some consolation at least [smile]

[Edited by - remigius on July 23, 2007 2:38:19 AM]
Mike.Popoloski
Mike.Popoloski
Arrays are probably special-cased into the MSIL, whereas any collection written will necessarily be slower, simply because of the need to call a function somewhere in order to get at the data. I am pretty sure that arrays are specially built into MSIL, and therefore don't need this function call, but anything that you write will be just as slow, because you have to make that function call to access the data.

Not that this is a huge speed hit we are talking about here. Besides, if you need that much speed you should probably use arrays, using the "you get what you pay for" approach. Arrays give you speed, while lists allow you to add as many elements as you want without having to recreate the array.
Mike Popoloski | Journal | SlimDX
Dark_Nebula
Dark_Nebula
Yes, MSIL has special instructions for single-dimensional, zero-based arrays. For example, the stelem and ldlen instructions seen in the code SiCrane posted.
These allowes the JIT to emit optimized code for array indexing. It will most likely move the bounds-check out of the inner loop.
alex_myrpg
alex_myrpg
These are some interesting comparisons, but I'm quite curious why the Vector class performs faster than a simple array in some cases. Surely it would always be slower than just an array, being built on top of one itself? Also, I'm wondering why you're doing Array.Length << 1. This ought to double the size of the array each time you add a value - I would simple do Array.Length + 1. Perhaps I'm misunderstanding here, but they were a few things that struck me as being rather strange when looking at the test results.
Cygon
Cygon
Just checked the method in reflector, this is what happens inside List.this[int index] in C# form:
if (index >= this._size) {  ThrowHelper.ThrowArgumentOutOfRangeException();}return this._items[index];


Quote:
Original post by alex_myrpg
Also, I'm wondering why you're doing Array.Length << 1. This ought to double the size of the array each time you add a value - I would simple do Array.Length + 1.


That's a simple adaptive resizing strategy. When the vector runs out of space, it pre-allocates some more items than it needs to avoid having to allocate and copy its contents all over again when the next item is added.

If a fixed step size was used (eg. always add 64 to the vector's capacity) it might waste a lot of memory when the user has hundreds of vectors with an average size of only 10 items. On the other side, if you used a fixed step size of 4, it would not perform optimally for a single vector with 10,000 items.

So it just doubles the size. This wastes little memory for small vectors and allocations happen in larger chunks the larger the vector gets.

-Markus-
Professional C++ and .NET developer trying to break into indie game development.
Follow my progress: http://blog.nuclex-games.com/ or Twitter - Topics: Ogre3D, Blender, game architecture tips & code snippets.
alex_myrpg
alex_myrpg
Ok, thanks for the explanation. I guess I wasn't quite reading it properly. I suppose the amount by which you increase the size depends how often you're going to add items, so that seems sensible.
remigius
remigius
Quote:
These are some interesting comparisons, but I'm quite curious why the Vector class performs faster than a simple array in some cases. Surely it would always be slower than just an array, being built on top of one itself?


The differences aren't huge, so I think this only goes to show that the overhead of indexing either the Vector or the arrays is marginal, even compared to the simple operations in the provided test code.


---


Reviewing SiCrane's code snippet, I'd say the performance loss is accounted for. His code calls the get_Count property accessor for each loop iteration (j < list.Count) and the use of indexers by List<> adds another function call (set_Item). That's two virtual function calls in the inner loop you don't have to worry about when using arrays, so I think this mystery can be considered solved.

To confirm the calls are the bottleneck, one could cache the list size in a local variable (since it doesn't seem to change anyway) and check if the performance 'penalty' of using the List is roughly cut in half.
Dark_Nebula
Dark_Nebula
Quote:
That's a simple adaptive resizing strategy

Yes, and it's a good one too. Using this strategy means that n insertions will run in O(n) amortized time, giving O(1) on inserts on average. Just increasing the capacity with one on each insert would give O(n) on inserts.
alex_myrpg
alex_myrpg
remigius, that could definitely be a possible explanation, but a test would be nice to see (I'll do one myself if I have some time) - anyway I always thought accessing the Length property of an array shouldn't add much overhead (whereas GetUpperBound(0) might well add some), but I could be wrong. As you said, the differences aren't big, so I doubt there's much to worry about in this case (comparing Array and Vector).
remigius
remigius
Quote:
Original post by alex_myrpg
... anyway I always thought accessing the Length property of an array shouldn't add much overhead


The MSIL of SiCrane's code shows that accessing array.Length shouldn't add much overhead actually. So it's List.Count that requires a virtual function call (which is relatively expensive), but accessing array.Length should be ok. I was wondering why they didn't make List.Count a field like array.Length is conceptually, but then again they probably figured it shouldn't be that much of an issue for typical container use.

And you're right on writing up some test, I already felt like a slacker when I dumped the suggestion without doing the test myself. Here are some results from my quick and dirty test similar to SiCrane's:

1 - Vanilla array inserts: 1.537s2 - Array inserts with bounds check ala List.set_Item: 2.854s3 - Array inserts with bounds check & version increment ala List.set_Item: 2.86s4 - List inserts with Count in loop: 5.704s5 - List inserts with cached Count: 4.547s


Going from this the costs to SiCrane's code (that's nr 4 in my results) seems to be distrbuted something like this:

- ~20% on List.Count
- ~23% on bounds checking in List.set_Item
- ~27% on the actual array inserts
- < 1% for the version increment

That leaves ~30% for the virutal function call to set_Item itself. This is a bit higher than the List.Count call, but this makes sense since get_Count takes no parameters, whereas set_Item takes two.

That's my dodgy statistical analysis for today [smile]

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.