Sign in to follow this  
davenirline

[java] how do you test for efficiency in java?

Recommended Posts

davenirline    121
I test Java routines by calling System.currentTimeMillis() and compare which set of routines consumes less time. But results are not very consistent. Sometimes they are, more or less, the same, and I can not determine which is really faster. I guess this is not the good way to compare things for efficiency. Is there some other better way? Some tools to use? Please show me?

Share this post


Link to post
Share on other sites
TheBluMage    372
If you just want something simple, you can stick with the System.currentTimeMillis() idea, but have a loop inside the timed area of your code that runs each routine several (hundred, thousand, hundred thousdand - depending on how long your routine takes) times, then divide the time that it took by the number of loops. I wouldn't count on most clocks to be more accurate (though some are) than about 16 ms +/- the real time, so make sure to run your routine enough times that a result a few ms off won't taint the results.

Share this post


Link to post
Share on other sites
Optus    422
This give you a good idea of the performance of your app. Try passing -Xprof to the JVM, it will then output the profiling data for your app when it closes.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
The most important thing you have to do, and which isn't mentioned above, is to run the test for at least 10 minutes, then pause, then *start counting the time* and run it the X,000 times as mentioned by TBM.

If that annoys you, bear in mind that what you're doing is pretty much a complete waste of your time anyway, and you really shouldn't be doing it. Java is far too advanced for you, and you should just program your code well and worry about performance when your game is too slow. Which shouldn't be until it's nearly complete anyway.

Share this post


Link to post
Share on other sites
Darragh    308
Quote:
Original post by davenirline
I test Java routines by calling System.currentTimeMillis() and compare which set of routines consumes less time. But results are not very consistent. Sometimes they are, more or less, the same, and I can not determine which is really faster. I guess this is not the good way to compare things for efficiency.

Is there some other better way? Some tools to use? Please show me?


You could try using System.nanoTime(); instead, which returns the current time in nanoseconds (billionths of a second). I've no idea how accurate this is but i use it anyway and it definitely helps give some idea as to how my programs are performing. It should help give you a better idea anyhow than System.currentTimeMillis() which only returns the time in milliseconds.

I've heard older JRE's have issues with timing also, so make sure you have the latest JRE if you want the most accurate results.

Share this post


Link to post
Share on other sites
redmilamber    253
Quote:
Original post by Darragh
I've no idea how accurate this is


it's as accurate as the OS allows. That's the definition - partly influenced IIRC by those of us who were upset that currentTimeMillis was inaccurate for no actual reason, and so asked for "somethign that is as accurate as the OS allows".

Quote:

but i use it anyway and it definitely helps give some idea as to how my programs are performing. It should help give you a better idea anyhow than System.currentTimeMillis() which only returns the time in milliseconds.


No, it will make absolutely no improvement whatsoever at all. You should read the AP post above.

Quote:

I've heard older JRE's have issues with timing also, so make sure you have the latest JRE if you want the most accurate results.


Their issues are granularity, hence no effect on benchmarking.

Share this post


Link to post
Share on other sites
Darragh    308
Quote:
Original post by redmilamber

Quote:

but i use it anyway and it definitely helps give some idea as to how my programs are performing. It should help give you a better idea anyhow than System.currentTimeMillis() which only returns the time in milliseconds.


No, it will make absolutely no improvement whatsoever at all. You should read the AP post above.



No I didn't mean that it was more accurate in keeping the time.

What I meant was that the greater resoultion of the timer (nanoseconds rather than milliseconds) would help in noticing smaller changes in the speed of his code.

Suppose that a particular function takes less than 1ms to run- using milliseconds in this particular case would not be helpful in detecting any speed improvement...

Share this post


Link to post
Share on other sites
redmilamber    253
Quote:
Original post by Darragh
Quote:
Original post by redmilamber
No, it will make absolutely no improvement whatsoever at all. You should read the AP post above.


No I didn't mean that it was more accurate in keeping the time.

What I meant was that the greater resoultion of the timer (nanoseconds rather than milliseconds) would help in noticing smaller changes in the speed of his code.

Suppose that a particular function takes less than 1ms to run- using milliseconds in this particular case would not be helpful in detecting any speed improvement...


Something that takes less than a millisecond to run is unmeasurable. There are no two ways about this: the fundamental design of java, laid down more than 11 years ago, forces that a speed test is only meaningful when run over the course of minutes. The current JVM's require TEN MINUTES to come up to full "normal" speed, and must be timed for several minutes to get an accurate measurement.

Hence the "run for thousands of times" (amongst several other reasons why you also need to do that - including the fact that on default settings some jvms will not optimize something until it has been executed 10,000 times).

Welcome to Java...

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
With the Client VM, the compiler kicks in after something is called 1500 times. It's only the server VM that compiles things after 10000 invocations. That is why the Client VM 'feels' much faster than the server VM while the server VM performs better optimizations.

Share this post


Link to post
Share on other sites
tombr    148
Quote:
Original post by redmilamber
Something that takes less than a millisecond to run is unmeasurable. There are no two ways about this: the fundamental design of java, laid down more than 11 years ago, forces that a speed test is only meaningful when run over the course of minutes. The current JVM's require TEN MINUTES to come up to full "normal" speed, and must be timed for several minutes to get an accurate measurement.

Hence the "run for thousands of times" (amongst several other reasons why you also need to do that - including the fact that on default settings some jvms will not optimize something until it has been executed 10,000 times).

Welcome to Java...


Letting the code run for 10 min before doing the test is a bit silly. So what do you do in a real game. Force the player to wait for 10 mins so the vm can warm up. Microbenchmarks must be treated with scepticism and haveing a warm up phase is a good idee, but TEN MINUTES!!!

Mesuring methods can be done if you've got a timer with enough precision. The nanoTime uses the performance counter I think, and could be used to mesure a method. Sure, other treads might step in, the mothod might be compiled etc. But it can be a good indication on how the method is performing.

Share this post


Link to post
Share on other sites
redmilamber    253
Quote:
Original post by tombr
Quote:
Original post by redmilamber
Welcome to Java...


Letting the code run for 10 min before doing the test is a bit silly. So what do you do in a real game. Force the player to wait for 10 mins so the vm can warm up. Microbenchmarks must be treated with scepticism and haveing a warm up phase is a good idee, but TEN MINUTES!!!


Until that moment, you are not testing anything at all and your numbers are meaningless.

Unless you know your game will only be played for, say, less than 3 minutes start to finish (not very ambitious) then you're interested in the real performance of code, not the "random way it performed once on my machine and I don't care how fast or slow it really is in practice". Shrug.

Share this post


Link to post
Share on other sites
Darragh    308
Quote:
Original post by redmilamber
The current JVM's require TEN MINUTES to come up to full "normal" speed, and must be timed for several minutes to get an accurate measurement.


Hmmm... that certainly explains something.

When I'm testing my code I noticed that after a few minutes or so that it suddenly speeds up for no apparent reason- usually running around 15-20% faster.

I always attributed this to the garbage collector winding down a bit after cleaning up all the junk I create when my code starts. Guess i'm wrong there..

Share this post


Link to post
Share on other sites
davenirline    121
I'm actually testing efficiency for my research. I compare game algorithms to conclude which is more usable for a particular game at some circumstances. Thanks for all the replies. Now I can continue my research. I'll try that nanoseconds. It really doesn't matter if there's only a slight difference. As long as I can prove it differs significantly, that's more important.

Thanks again. Let the thread continue.

Share this post


Link to post
Share on other sites
pirate_dau    151
Quote:
Original post by Darragh
When I'm testing my code I noticed that after a few minutes or so that it suddenly speeds up for no apparent reason- usually running around 15-20% faster.

I always attributed this to the garbage collector winding down a bit after cleaning up all the junk I create when my code starts. Guess i'm wrong there..


This could be due to your code being compiled into native form by the hotspot compiler.

Share this post


Link to post
Share on other sites
tebriel    904
Quote:
Original post by Anonymous Poster
If that annoys you, bear in mind that what you're doing is pretty much a complete waste of your time anyway, and you really shouldn't be doing it. Java is far too advanced for you,


What's this? An AP telling someone that they obviously know nothing about that "Java is far too advanced" for them. Wow... it's a shame people can't ask a simple question around here without having some stupid AP accuse them of being an idiot. If it wasn't for that kind of stuff, these forums would be just about perfect, oh well.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Quote:
Original post by Tebriel
Quote:
Original post by Anonymous Poster
If that annoys you, bear in mind that what you're doing is pretty much a complete waste of your time anyway, and you really shouldn't be doing it. Java is far too advanced for you,


What's this? An AP telling someone that they obviously know nothing about that "Java is far too advanced" for them. Wow... it's a shame people can't ask a simple question around here without having some stupid AP accuse them of being an idiot. If it wasn't for that kind of stuff, these forums would be just about perfect, oh well.


It wasn't meant as an insult, just a statement of fact.

Java, the entire platform, was SPECIFICALLY DESIGNED to act in such a way that the type of measurement described WOULD NOT WORK (hence "complete waste of time") unless you do all the steps mentioned (many minutes warmup, then start measuring, measure at least X times, etc).

This is fundamental to java, and it is SO GOOD at doing it that there are few hunmans who have a chance of predicting what the hell is going on in the first X minutes before the warmup has happened (hence "far too advanced": java is doing a heck of a lot under the covers to make your code run faster than you could ever have hand-optimized it. Your CPU has the new SSE3 one day? Well, java discovers that and *re-compiles* your code in such a way to take advantage of it. This is anathema to C programmers, who have to go away and get the source out and manually re-compile with a new copy of ms-vc/gcc and manually specify different flags to get the same effect AND they have to send a copy to all players. Java "Just Does It" automaticallyk, on all players machines, without them downloading a new version of the game).

Sorry if you misinterpreted it...

Share this post


Link to post
Share on other sites
Malone1234    139
If you want something a little better than System.currentTime() there are a lot of really good (and free) profilers out there for Java. I've use a profiling plugin for Eclipse before, there are several to choose from. It'll tell you in which methods the most time is being spent and how many times they are called, etc.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Quote:
Original post by Anonymous Poster
With the Client VM, the compiler kicks in after something is called 1500 times. It's only the server VM that compiles things after 10000 invocations. That is why the Client VM 'feels' much faster than the server VM while the server VM performs better optimizations.





Damn that is bad.

Is there no way (parameter somewhere) to tell the VM to compile at a lower invocation count ?? (maybe even a flag to tell the compiler to ALWAYS compile on first use???)

You would think that numbers much lower than 1500 or 10000 would be enough of an indicator of high usage of a chunk of code....

Share this post


Link to post
Share on other sites
tebriel    904
Quote:
Original post by Anonymous Poster
It wasn't meant as an insult, just a statement of fact.

Java, the entire platform, was SPECIFICALLY DESIGNED to act in such a way that the type of measurement described WOULD NOT WORK (hence "complete waste of time") unless you do all the steps mentioned (many minutes warmup, then start measuring, measure at least X times, etc).

This is fundamental to java, and it is SO GOOD at doing it that there are few hunmans who have a chance of predicting what the hell is going on in the first X minutes before the warmup has happened (hence "far too advanced": java is doing a heck of a lot under the covers to make your code run faster than you could ever have hand-optimized it. Your CPU has the new SSE3 one day? Well, java discovers that and *re-compiles* your code in such a way to take advantage of it. This is anathema to C programmers, who have to go away and get the source out and manually re-compile with a new copy of ms-vc/gcc and manually specify different flags to get the same effect AND they have to send a copy to all players. Java "Just Does It" automaticallyk, on all players machines, without them downloading a new version of the game).

Sorry if you misinterpreted it...


Ahh, I see what you were saying. "Java is far too advanced for your purposes" is what you meant. Glad to hear it wasn't actually an attempt to be insulting.

Kinda thought it may have been a simple lack of clarity but I was in a slightly pissy mood, so I didn't really care! :D Yea, I'm bad...heh heh.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Quote:
Original post by Anonymous Poster
Quote:
Original post by Anonymous Poster
With the Client VM, the compiler kicks in after something is called 1500 times. It's only the server VM that compiles things after 10000 invocations. That is why the Client VM 'feels' much faster than the server VM while the server VM performs better optimizations.


Damn that is bad.


Actually, it's good - otherwise when you startup your program runs slow cos it's COMPILING (LEAVE ME ALONE I'M WORKING) (says your CPU - ever looked to see how long it takes to run a single compile of a linux kernel on brand new hardware? We're still talking "leave it overnight" even in 2005...).

You really only want the compilation IF the program runs for a long time, and can amortize the cost - what many people want is for the JVM to *save* the compilation data even if it's not started, so that on each run of the app, the JVM is gathering more info, and eventually starts compiling, amortized over *every run of the app*, making it that every time you play, it might speed up a little :).

Quote:

Is there no way (parameter somewhere) to tell the VM to compile at a lower invocation count ?? (maybe even a flag to tell the compiler to ALWAYS compile on first use???)


There's 5 of them, off the top of my head. Seeing as there are currently around 1,000 JVM flags on sun's JVM's, most undocumented, I suggest you JFGFI (the middle initial stands for Google, the rest I'm sure you can guess ;))

Quote:

You would think that numbers much lower than 1500 or 10000 would be enough of an indicator of high usage of a chunk of code....


Find the switches, play with it, see what happens when you start doing what would "seem" sensible to you or I (non JVM engineers).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this