quote:Original post by Aldacron
Another thing to consider when running microbenchmarks in Java that tests like these often overlook is ''warming up'' the VM. For small applications such as these tests, JIT compilation will not kick in. Making the program run longer gets the JIT going, so it''s usually a good idea to call the test function a couple of times before actually calling it for the test run. Otherwise, you''re getting results for interpreted byte code rather than run-time compiled native code. The JIT is where all of the smart optimizations and bigger performance gains come into play.

Could you elaborate? What do you mean by JIT kicking in? I was under the impression that JIT VM''s always JIT''d the code from the first bytecode?? If not, when is it determined to JIT the rest of the code?

If you''re talking about the startup time of the JIT compiler, would that not be irrelevant? Considering that the timing code would be within the code, the timer would not begin until the code had begun running, assuming he was not externally timing the programs.

quote:Original post by HenryApe

Here is an article that uses the same benchmarking flaw, measuring different things, to demonstrate that Java (although only JDK1.4.1) is infinitely faster than C++, C#, and a bunch of other languages.

Language Time taken to run a simple empty loop test for 10 000 billion iterations
Java Under one second
C/C++ approx. 1 month

This a joke, I hope! If you really go through 10E13 iterations and you use only one clock to do each iteration (impossible) you will need a 10000 GHz cpu ( probably a 100 THz ) to finish in 1sec!!!.

Note that in this test C++ has a speed of about 4M iterations/sec (int this case it is too slow bcause it is equal to 250 clocks on 1GHz CPU ).

The test simply demonstrates (a month to see this???) that java 1.4.1 skip the loop if it is empty...

Looks like this benchmark is inferior to language shootouts in every single aspect.

On the otherhand it is "mathematical" ablitity. I would have like to seen how ocaml would have performed in that

Given that the author doesn''t understand if he''s creating a managed or unmanaged executable with visual c++, I didn''t have high hopes for the article. And I was not dissapointed...

quote:Original post by CrashTheuniversE
I think there is no real basis for the results. Everybody
knows that even if interpreters (JVM, and Python) are
now very optimized with OTB(on time building/compilation),
they can''t stand results of a fully compiled programm, IMHO.

When I was taking a grad optimizing compiler course, the professor mentioned this project:
http://www.hpl.hp.com/techreports/1999/HPL-1999-78.html
http://www.hpl.hp.com/techreports/1999/HPL-1999-77.html

It''s Dynamo by HP. They created an HP-RISC interpreter that would run regular compiled HP-RISC binaries. And they ran this thing on an HP-RISC machine... so they had an HP-RISC interpreter running on an HP-RISC machine, and the interpreter analyzed programs while they were running and created further optimized code based on how the program actually ran (what branches it took, what parts of the program were run frequently, etc). They were starting with binaries that had already gone through an optimizing compiler, but they were able not only to make up the performance hit of running the interpreter, but to actually get better performance than if they just ran the binaries on the bare machine.

Pretty impressive stuff.

Now, our JIT compiled and interpreted languages aren''t doing that well, but the point is that JIT or interpreted doesn''t automatically mean slow. There are other factors that affect performance. One is language features; if your language is dynamically typed and using all sorts of runtime checking to get things done, that adds overhead that can''t be optimized away. Quality and maturity of implementation is also a big issue, and I think it''s safe to say that the interpreter for Python, for example, has not had as much time and effort poured into getting it optimized it as many of the C and C++ compilers that are out there, and a lot of this will have to do with the intended use of the language. But my guess is that the future will be JIT or interpreted with performance equal to or better than current languages with better portability. I''m not saying it will be Java 2.5 or some version of C#, but eventually it will happen.

Man it was a pain to dig up those papers on Dynamo. I couldn''t remember the name, and searching for keywords turned up all kinds of unrelated crap. But I knew if I tried to cite that system without links I''d be called a BSer.

[Edited by - The_Incubator on November 28, 2006 11:07:29 PM]

Python actually did surprisingly well in these tests. It''s main bad point was "long math", because Python''s longs can be infinitely large instead of just being 64-bit numbers. So if you rule that "unfair" test out and calculate the total of the rest of the tests, Python with Psyco is only 5 times slower than C++.