Updated marshalling performance results

Started by
4 comments, last by hplus0603 15 years, 2 months ago
Shalom Recently I updated the Windows portion of a page that compares results from the Boost Serialization library and the C++ Middleware Writer -- www.webEbenezer.net/comparison.html The updated tests use MSVC 9.0, Windows XP and Boost 1.37. The new results are somewhat more in favor of our approach than results using MSVC 8.0, Windows XP and Boost 1.34 -- www.webEbenezer.net/prevcomp.html. I hope to further update and add to the tests in the future. Brian Wood Ebenezer Enterprises www.webEbenezer.net If you haven't read anything by Jonathan Katz, http://wuphys.wustl.edu/~katz, I recommend it.
Advertisement
Do you have actual numbers or other concrete data to show the results as well as the machine specs the tests were ran on? How about OS conditions and the testing steps taken to make sure the tests weren't being biased?

I'm not quite sure how to interpret your results. You are showing serializing a lot of ints, but that is not practical for most networking applications. Most network data is going to be a combination of custom user data types, strings, floats, etc...

I'm not commenting on your actual library's performance or contesting your results, but what you are trying to present looks quite unsubstantial unless more information is added. It's nice to see test cases, but there is more to the process than just that.

So, I would suggest:
- Practical test cases showing more data types or structures that are more likely to be used.
- Actual result data rather than a textual summary of what happened.
- More information on the hardware used and other common environment settings.

I hope that does not sound too harsh, thanks for sharing [smile]
My question is why focus on serialization performance. What combination of server hardware and network set-up makes serialization ever turn up in the top 20 on a full application profile?
enum Bool { True, False, FileNotFound };
Quote:Original post by Drew_Benton
Do you have actual numbers or other concrete data to show the results as well as the machine specs the tests were ran on? How about OS conditions and the testing steps taken to make sure the tests weren't being biased?


For the test with just a list of ints, the Boost approach ranged from 2.1 to 3.2 times slower. As far as testing steps, I run one of the tests a given number of times and then the other the same number of times. Certainly I'm not starting or stopping any other programs while in the middle of the testing.

Quote:
I'm not quite sure how to interpret your results. You are showing serializing a lot of ints, but that is not practical for most networking applications. Most network data is going to be a combination of custom user data types, strings, floats, etc...


We have limited resources to invest in this area. As I said, hopefully we'll be able to do more in the future. Based on your comment though, I tested (also on XP using msvc 9.0 and Boost 1.37) marshalling a deque of standard strings. The Boost version is 2.1 times slower than the Ebenezer version. The strings were fairly short with lengths between 1 and 8.

Quote:
I'm not commenting on your actual library's performance or contesting your results, but what you are trying to present looks quite unsubstantial unless more information is added. It's nice to see test cases, but there is more to the process than just that.

So, I would suggest:
- Practical test cases showing more data types or structures that are more likely to be used.
- Actual result data rather than a textual summary of what happened.
- More information on the hardware used and other common environment settings.

I hope that does not sound too harsh, thanks for sharing [smile]


I agree with some of that and hopefully we'll be able to make progress in that area.


Brian Wood
Ebenezer Enterprises
www.webEbenezer.net
Quote:Original post by hplus0603
My question is why focus on serialization performance.


I guess it's similar to becoming a juggler, baseball player or tennis pro.

Quote:
What combination of server hardware and network set-up makes serialization ever turn up in the top 20 on a full application profile?


I don't know. I do know that IT budgets are tight right now and programmers are expected to do more with less hardware. Does anyone think the average IT budget will keep up with inflation over the next few years?

Brian Wood
Ebenezer Enterprises
www.webEbenezer.net

"All the world is just a narrow bridge. The most important thing
is not to be afraid." Rebbe Nachman
Quote:Does anyone think the average IT budget will keep up with inflation over the next few years?


In performance? Absolutely! At fixed prices, the number of cores as well as memory throughput will double every 2-3 years at least. (Note that the clock frequency per core used to do that, but hasn't for quite some time now)

However, all I've seen from corporate IT is moving (or, rather, has moved) to managed frameworks; either Java EE or .NET. And the big movement now is to outsource corporate applications entirely, using SaaS platforms like salesforce.com, Google Apps or Microsoft Azure. Before that, there were lots of large enterprise frameworks that did the same thing, but without the outsourcing part, such as BAE, IBM etc.

A typical enterprise IT application worries a whole lot about things like service busses, (high-level) message queues etc. Low-level serialization in C++ doesn't even enter into the equation! The people I do worry about those kinds of things are either network infrastructure people (Cisco, Juniper, etc), or real-time applications people (mostly your game developers or embedded systems developers). For the former, compliance with and adherence to standards is critically important, and thus they'll use some serialization mechanism that's engineered explicitly for the standard in question. For the latter, most of them want to focus on where they will get the big speed-up, and network serialization throughput isn't it -- if serialization takes more than 1% of your application runtime, you're doing something entirely wrong, and if you can remove ALL of 1% of your application runtime, then you have sped up your application by not much. Amdahl's law and all that.

I understand that you like your serialization solution, and you want others to like it and use it as much as you do. However, personally, I don't see the value. I'm not the entire world, though, so if some other user out there is using it for something real, I'll certainly welcome feedback and discussion.
enum Bool { True, False, FileNotFound };

This topic is closed to new replies.

Advertisement