Jump to content

  • Log In with Google      Sign In   
  • Create Account


Big Array to String Speed


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
10 replies to this topic

#1 asdatapel   Members   -  Reputation: 111

Like
0Likes
Like

Posted 03 December 2012 - 09:14 PM

Sup guys. Ive got this really big array of about 25 thousand strings(its okay they're pretty short) and i need to get them all into one big string.
Right now Im using string += array[count] + '\n'. It works, but it takes pretty long time to do it. It pretty much about two minutes of the program looking like its crashed. So if you guys can help, thanks.

Sponsor:

#2 SiCrane   Moderators   -  Reputation: 9427

Like
3Likes
Like

Posted 03 December 2012 - 09:18 PM

Did you try a StringBuilder? Make sure to try setting the capacity first.

#3 asdatapel   Members   -  Reputation: 111

Like
0Likes
Like

Posted 03 December 2012 - 09:20 PM

just wondering, will this make a difference?

#4 RulerOfNothing   Members   -  Reputation: 1153

Like
2Likes
Like

Posted 03 December 2012 - 11:23 PM

Well, you need to allocate a destination string big enough to hold the concatenated strings, otherwise you're going to have to allocate memory and copy data more than you have to. With the solution you provided, the program needs to copy the first string about 25000 times (and the second string 1 less time than that, the third 2 fewer times and so on) so it is not a particularly efficient solution.

#5 iMalc   Crossbones+   -  Reputation: 2264

Like
2Likes
Like

Posted 04 December 2012 - 01:57 AM

What language is this written in? If it's C++ then you could use a std::stringstream to build the large string instead.

I had a similar thing once with CComBSTRs which went from 2 minutes to about 6 seconds just by building up the various substrings from certain sections separately from concatenating that onto the main string, which was a small change. Still not the most efficient, but it more than met the performance requirements, and we spent more time on it in another iteration.
"In order to understand recursion, you must first understand recursion."
My website dedicated to sorting algorithms

#6 rip-off   Moderators   -  Reputation: 7723

Like
1Likes
Like

Posted 04 December 2012 - 04:28 AM

Where do these strings come from? How are they loaded in your application?

#7 PhillipHamlyn   Members   -  Reputation: 454

Like
3Likes
Like

Posted 04 December 2012 - 03:29 PM

If its C# or any other .net language you absolutely must use StringBuilder.
Consider - all strings in .net languages are immutable - they cannot be changed once created. So;

A = A + B then
A = A + C then
A = A + D
essentially means creating a new string called A and filling it with A + C. Repeat 10,000 times and you have the memory heap fragmentation from hell.

StringBuilder keeps a list of the pointers to the original strings, calculates the size, and copies them all in one operation, meaning the intermediate instances of A shown above are not created. Hugely faster and more memory efficient.

#8 btower   Members   -  Reputation: 874

Like
2Likes
Like

Posted 04 December 2012 - 03:39 PM

How about the following:


[source lang="csharp"] //this just generates a "big string array" to play with List<string> bigList = new List<string>(); for (int i = 0; i < 25000; ++i) bigList.Add("someMediumLengthString"); string[] bigStringArray = bigList.ToArray(); //measure time DateTime start = DateTime.Now; //now we do the part that matters string ourHugeCombinedString = string.Join<string>('\n'.ToString(), bigStringArray); //print out the time it took to make the giant string TimeSpan timeToMakeCombinedString = DateTime.Now.Subtract(start); System.Diagnostics.Debug.WriteLine(timeToMakeCombinedString.TotalMilliseconds.ToString()); //prints out "2,0001" on my system, so 2 ms![/source]

#9 PhillipHamlyn   Members   -  Reputation: 454

Like
2Likes
Like

Posted 04 December 2012 - 03:49 PM

Depends on the framework implementation of String.Join. I expect its StringBuilder internally.

... and using my trusty copy of .Net Reflector I see that it does indeed use StringBuilder internally.

#10 Nypyren   Crossbones+   -  Reputation: 3743

Like
2Likes
Like

Posted 04 December 2012 - 05:26 PM

Simplified syntax for string.Join (C# knows how to do generic type inference):

string combinedString = string.Join("\n", array);

It should be noted that the latest version of .Net includes both a string.Join which takes an array AND one which works on IEnumerable<T>.

Edited by Nypyren, 04 December 2012 - 05:27 PM.


#11 asdatapel   Members   -  Reputation: 111

Like
0Likes
Like

Posted 04 December 2012 - 06:13 PM

goddamn thanks guys. Join() works perfectly!




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS