Intel sponsors gamedev.net search:   
[Control Panel] [Register] [Bookmarks] [Who's Online] [Active Topics] [Stats] [FAQ] [Search]

Add Forum to Favorites |  Send Topic To a Friend | View Forum FAQ | Track this topic

Page:   1 2 »»

 Last Thread Next Thread 
 Benchmarks
Post New Topic  Post Reply 
P4 2.4 Ghz, 1GRam, win2k sp3, microsft c++ vc7 compiler 2003 edition,
nonoptimized code/release build, multithreaded libraries scrip libs
statically linked.

I ran some tests comparing SQUIRREL2_1_0 and angelscript_2.4
In a 100 runs I get folowing results.
Time is in microseconds.
---------------------------------------------------------squ--------angel----ratio
---------------------------------------------------------time in microseconds
Calling void function from c/c++ to script 10000times    3894        90453    003
Calculating first 30 fibonacii nums in script            1527859     398040   3.99
Computing 100000 float math divisions/multiplications    47549       31603    1.58
Script calls another script void function 100000 times   47347       17613   2.81                       
Integer math storing the results into 1000 int array     429811      528918  0.85
10000 calls to C/C++ function passing an integer         10056       1604    6.24
10000 calls to C/C++ function passing a 32 chars string  9891        8484    1.17
--------------------------------------------------------------------------------







[Edited by - Marius on May 26, 2006 12:00:07 PM]

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Thanks for this, it's very interesting.
I have been looking for a good comparison between AS and SQ as I am going to need a scripting interpreter in a performance-intense application very soon.

 User Rating: 1015    Report this Post to a Moderator | Link

What the!? 9ms to call a funcion in AS (from C/C++ that is)? Am I the only one who thinks that sounds kinda much? Can you please post that benchmarking code? I have to see if the same thing happends with later versions as well..

 User Rating: 1052   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

No, that's 9 microseconds.

1000 microseconds = 1 millisecond. Each call took 9 microseconds, or 0.009 milliseconds.

And, no, it's not a lot. The overhead of asIScriptContext::prepare is well known. It's doing a lot of work. And, as you can see, AS trumped it in most of the other tests - most noticably the calls back into C++.

 User Rating: 1547   |  Rate This User  Send Private MessageView ProfileView Journal Report this Post to a Moderator | Link

Yes, sorry, my bad.. I was absolutely sure it was measured in milliseconds.. While I do understand it takes time to prepare and things, 9ms seemed extreme.. 9µs however isn't very much at all.. Again, thanks for pointing this out, I'll read more closely before I make a statement next time (rating-- on self :p)..

 User Rating: 1052   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Here is the whole project: vc7(2003).

Ia a bit messy, (...) math float tests have room of improvement.

I was interested in call times between c->script
script->script and script->c
Project can be imported in code::blocks NP.

I zipped the whole structure dir with libs and
headers. The exe is gen in /_bin
All libs are generated in /_lib
If you step in debuger make sure you set the exeutable path
to ../_bin couse the dumb vc7 holds the absolute path.

BM_.rar

[Edited by - Marius on May 26, 2006 3:10:26 PM]

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Yes, this is pretty interesting-however it might be more representative and also more informative if there was some sort of wider comparison. For example, we could not only extend the current test suite but also the scripting languages supported, Lua, Perl, Python and Ruby would seem like ideal candidates for such a comparison-what do you think?

This would certainly show those areas in AngelScript and Squirrel that are superior or inferior compared to the mainstream scripting languages.

Just an idea :-)

 User Rating: 1015    Report this Post to a Moderator | Link

Pretty much a year ago, we had this thread: http://www.gamedev.net/community/forums/topic.asp?topic_id=311655&forum_id=41&gforum_id=0


 User Rating: 1015    Report this Post to a Moderator | Link

Quote:
Original post by Anonymous Poster
Pretty much a year ago, we had this thread: http://www.gamedev.net/community/forums/topic.asp?topic_id=311655&forum_id=41&gforum_id=0


Didn't see any benchmarks there...

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Cool!

This really shows me where I have to spend some time optimizing the code for AngelScript. :)

I'll also take a look at that benchmark code you have, to see if you're doing everything the optimal way. ;)

Regards,
Andreas

__________________________________________________________
AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game


 User Rating: 1702   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Interresting comparison, I would wish someone would also test a couple of other scripting languages which "compete" which AS.

I don't know much about the techniques in either AS or Squirrel, but one interresting thing to note is that the void function takes much longer in AS, so one must assume that AS does quite a lot more than Squirrel, which makes the other tests unfair. For instance AS might generate some internal byte-code, while Squirrel just store the functions in a string and doesn't parse it until it's actually run. Of course it could also be a possibility that AS just parse the function everytime it's called and Squirrel actually notes that the function have been parsed once, I don't think this is what is happening though since "Script calls another script void function 100000 times" also should be slow in AS then.

Quote:
Original post by Marius
microsft c++ vc7 compiler 2003 edition,


Might be a little off-topic, but the 2003 edition is called VC 7.1, VC 7.0 is the 2002 edition, which one do you use?

 User Rating: 1688   |  Rate This User  Send Private MessageView ProfileView Journal Report this Post to a Moderator | Link

For simple benchmarks, it is known that Lua can be faster than Squirrel. Here are Alberto's remarks regarding benchmarks and real-world applications (such as large MMO with several megabyte of data living in the VM, etc.):

From:
http://www.squirrel-lang.org/forums/827/ShowPost.aspx

Quote:

As a personal experience, I'm writing an MMOG, and we also have a huge amount of squirrel code. As a matter of fact our game is written is squirrel and there isn't any part of the C++ that is specifically tight to our current title.

As you can imagine we have a lot of entities (how we call game objects with a script attached) plus main loop, DB interface etc... So far squirrel doesn't really show up in the profiler. As any game what is consuming all our CPU is Graphics on the Client and Collision detection and Network protocol on the server.

Now talking about squirrel performances, I'm personally very satisfied at least for my purposes. Comparing squirrel 1.x and lua is quite understandable, comparing Squirrel 2.x and Lua id a bit unfair because squirrel now packs many more features(especially types) that in a dynamically typed language means a lot of runtime checks. However means better performances in different fields.

* Squirrel arrays are ways faster and more compact than tables used as arrays
* Class instances use less than half memory of the equivalent solution done with tables(and we got alot of instances).
* Weak references have been a bless from god in our project, in terms of resource management(we do resource management in squirrel).
* Proper class inheritance allows to create a more comfortable programming model for the average C++ programmer.

Also I think when looking at the performances of a language of this kind, we have to look at a real scenario, stuff like The great language Shootout is in my advice pointless. In a real app you have several megs of data living in the VM; what is going to affect your performances is not the bytecode speed (that's almost linear). Mostly, as you realized yourself , the performance problem will be memory. Memory management (if is a GCd language) plus memory aliasing and pagefaults and cache misses.

Ref counting solves the first issue by giving a linear/predictable cost in memory management. Smaller and more cache friendly data structures improve the aliasing (my attempt to solve this in squirrel are classes and arrays). To give you an example in our engine we have written a very good memory manager that keeps blocks of memory of the same size in the same virtual page(we do not use a heap, we go straight to the os virtual page and take advantage of the 4Kb memory alignment) and squirrel performs 3 times faster on my memory stress test that tries to reproduce a game loop creating and deleting stuff. It makes the system behave like a language with a copy collector.

To conclude, what I'm trying to say is that I wouldn't be too concerned if the bytecode execution speed is 30% slower in some tiny sorting test, when at the end in a real system that's not going to make any difference.

Anyway I'm still working to improve squirrel performances.


Keep in mind many games use Python, which is very slow compared to Lua/Sq/As, however it's still fast enough for real-time games.

As for Squirrel vs. AngelScript, one would need to first determine how easy/complete the integration with C++ to script is. SqPlus (for Squirrel) does everything I need, quickly and easily. Perhaps someone with an unbiased opinion could review each of the scripting langauge binding systems and ease of use (Lua, AS, Sq, Python, C#, etc.).

From the benchmarks presented for AS vs. Sq, the AS C++ to script calls are so slow that it would appear something is wrong (23 times slower?). 3 of the tests are close, with 3 showing AS 3-6x faster. A real game/app would need to be tested with script-to-C++ and C++-to-script calls to see if there is any significant difference between the two languages. For systems with lots of classes, inheritance, and many megs of resource data in script, Alberto states that Sq is 3x faster than Lua* (for his MMO project). More tests/benchmarks would be cool to see. *Estimate based on other comments vs. Lua.

For those looking for ultimate scripting speed, look no further than C#. It compiles very quickly to code that is not much slower than native C++. Using Mono or dotGNU, game related C# code should run anywere (not clear about PSP or other memory limited systems). For something like Alberto's MMO, Squirrel may still work better due to his customized memory allocator and Squirrel's design to keep memory related GC burst free.

 User Rating: 1464   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Quote:
Original post by WitchLord
Cool!

This really shows me where I have to spend some time optimizing the code for AngelScript. :)

I'll also take a look at that benchmark code you have, to see if you're doing everything the optimal way. ;)

Regards,
Andreas


1) Thank you Andreas. I hope this 'ugly bench startup' can be improved to became a real bechmark. I also started to profile lua (as seen in the code),
but I did not have time (I did it it at my job (cheating) in respiro times) to finalize it.


2) I use VC7.1
3) .. is interesting calling script to script taking less time than script to c/c++ and vicevera. I hope I did not make any mistakes in the code. You guys can doublecheck and any lua expert can finalize the lua part.


 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

I am personally happy with either of them.
As any language you find advantages and disadvantages on both sides.
I hope the test are close to be accurate and I am waiting authors
reply if I did anything wrong (function caching, template wrappers etc.)
Thx....

Here are latest results with a more realistinc calls passing 2 params
into/and from script (time in microseconds). What is interesting
is that over time one of the libs (or both) eats virtual memory and
execution times increases exponentially.

TEST                        SQUIRREL2_1_0  angelscript_2.6.0
----------------------------------------------------------------
1K calls C++    -> Script    S = 2357      A = 5716        
1K calls Script -> Script    S = 843       A = 1104        
1K calls Script -> C++       S = 1686      A = 1357        




and the scripts
//angel
int Main(int k, bstr s)
{
    int h=k;
    return h*100;
}

int CallScript(int v)
{
    int k;
    for(k=0;k<v;k++)
        Main(k,"abcdefghijklmnopqstuvxyz");
    return v;
}

int CallHost(CTest o, int n)
{
    int i;
    for (i = 0; i < n; i++) 
    {
        o.Calla(i,"asdfghjkqwertyuiopasdfghjkqwertyuiopas");
    }
    return i;
}

//squer
function Main(k,s)
{
    local h=k;
    return h*100;
}

function CallScript(v)
{
    local l = 0;
    for (local i = 0; i < v; i+=1) 
        Main(i,"abcdefghijklmnopqstuvxyz");
    return l;
}

function CallHost(o,n)
{
    for (local i = 0; i < n; i+=1) 
    {
        o.Call(i,"asdfghjkqwertyuiopasdfghjkqwertyuiopas");
    }
    return 0;
}


//cpp angel
  int r = s._pengine->RegisterObjectType("CTest", sizeof(CTest), asOBJ_CLASS);
    r = s._pengine->RegisterObjectMethod("CTest", "int Calla(int i, bstr s)", asMETHOD(CTest,Calla),asCALL_THISCALL);

    
    asBSTR pstr = asBStrAlloc(64);
    strcpy((char*)pstr,"asdfasdfasdfas2458 jhflkajsdhflkjasdhflkjadfasdf");


    BOOL b = s.Load("test.ang");

    int Main            =   s.GetFunction("test.ang","int Main(int k, bstr s)");
    int CallScript      =   s.GetFunction("test.ang","int CallScript(int v)");
    int CallHost        =   s.GetFunction("test.ang","int CallHost(CTest o, int n)");

    FFunctor    fMain(Main);
    FFunctor    fCallScript(CallScript);  
    FFunctor    fCallHost(CallHost);
    DECLARE_TIME();
    for(int i=0;i<1000;i++)
    {
        fMain(i,&pstr);
    }
    GETTIME();
...

//cpp sqer
CTest       t;
    SquirrelVM::Init();

    SQClassDef<CTest>("CTest").func(&CTest::Call,"Call");

    SquirrelObject main = SquirrelVM::CompileScript(_SC("test.nut"));
    SquirrelVM::RunScript(main);

    SquirrelFunction<int> fMain("Main");
    SquirrelFunction<int> fCallScript("CallScript");
    SquirrelFunction<int> fCallHost("CallHost");
    const char str[]="asdfasdfasdfas2458 jhflkajsdhflkjasdhflkjadfasdf";

    DECLARE_TIME();
    for(int i=0;i<1000;i++)
    {
        fMain(i, str);
    }






[Edited by - Marius on May 28, 2006 11:08:34 PM]

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

You can reduce SqPlus overhead slightly by changing:

#define sq_argassert(arg,_index_) if (!Match(TypeWrapper<P##arg>(),v,_index_)) return sq_throwerror(v,_T("Incorrect function argument"))

to

#define sq_argassert(arg,_index_)



Part of your Squirrel 2.1 benchmark overhead is due to SqPlus: I have not yet made an optimization pass on the code. The ease of use and generality of template binding systems come at a small runtime overhead cost (template binding system overhead varies with implementation).

Squirrel/SqPlus should not leak any memory (I use it in my game; haven't seen any bug reports regarding memory leaks).

 User Rating: 1464   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

I have profiled them sapratelly. Angel call C++->Script increases.

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

AngelScript doesn't have any memory leaks that I'm aware of. But it does a lot of allocating/deallocating of memory which causes memory fragmentation, which in turn may very well be what is increasing the execution time with repeated execution.

I'll work to minimize the memory fragmentation in future versions.



__________________________________________________________
AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game


 User Rating: 1702   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Quote:
Original post by WitchLord
AngelScript doesn't have any memory leaks that I'm aware of. But it does a lot of allocating/deallocating of memory which causes memory fragmentation, which in turn may very well be what is increasing the execution time with repeated execution.

I'll work to minimize the memory fragmentation in future versions.


Squirrel+SqPlus also allocate and free memory (especially for instances). You might look at his benchmark and see if the code properly releases references when using AngelScript. I had a bug in SqPlus that did not properly restore the Squirrel stack that had behavior similar to what Marius described with AngelScript.



 User Rating: 1464   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Indeed. I took a look at the benchmark code, and it seems that the asIScriptContext is never released after it is used. This would indeed eat a lot of memory and slow down the execution.

Fixing this should improve the overall performance measured in this benchmark. But my guess is that squirrel will still win with about 100 times the performance for calling script functions.

The difference is currently very large between the best and the worse performance for AngelScript:

10000 calls from C++ 2 script    S = 8089       A = 722240      0.01

10000 calls from C++ 2 script    S = 7890       A = 5718446     0.00


All 10 executions should be approximately the same.

Another thing that can be improved is to reuse the context. Instead of creating a new context and then releasing it again for each script function that is called, you could create one context and use that for all the calls. Simply call Prepare() and then Execute() on the context.

Creating multiple contexts is only useful if you want to keep various scripts running in parallel, for example co-routines, or even multiple threads.

__________________________________________________________
AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game


 User Rating: 1702   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

I see the diferences you have between scripts is quite big...
Here is an image taken from latest results I just take on my P4
winnt 512 Mb.



[Edited by - WitchLord on June 4, 2006 9:42:24 AM]

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Your screenshot doesn't show up. I get a Gateway Timeout message when I try to view it in the browser.

Yeah, my computer is getting really old, it isn't very good at running even simple benchmarks any more. It's a Celeron 800MHz with 256MB of memory, running WinXP SP 2. It makes sense that I get a large difference in performance between Squirrel and AngelScript, after all the benchmark is allocating more than 100MB of memory for AngelScript without releasing it. (10*10000 calls, where each call allocates a context with 1KB stack space)

Regards,
Andreas

PS. I edited your post because it was missing an apostrophe in the image tag, making the page show up all wrong.

__________________________________________________________
AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game


 User Rating: 1702   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

I ran some test on C-smile with tiscript.lib and tiscript.dll
(no source code for these libs)
Here are the results:
Because there is no method of holding on to a script
function by a handler or somekind of ID the call has to be explicit
done by function name. When calling a function from C to script
and viceversa the call time is proportional with the function name length.

The call of 'main'() C->c-smile function is ~47943 microseconds/10000 calls
and the call of 'mainmainmainmainmain'() is ~80943 microseconds/10000 calls

A call c-amile->c-smile of a void function receivineg an int is ~7609 microsecs/10000 calls

tested:
Dell OPTIPLEX GX260
P4 2.00 Ghz
1Gb Ram
compiler VC6, release mode non-optimized.

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Are you going to update the benchmark program with the suggestions I made?

I just made a little test on my own (while optimizing the new version of AngelScript) and saw that reusing the context for each call, instead of creating a new context, gives a performance increase of about 10-15 times. You should get the same performance increase in the current version of AngelScript and it would be great if you could update your benchmark program to reflect that.

Especially since it may show that AngelScript is actually faster than Squirrel even with this trivial test of calling a void function. I'm basing this assumption on the first execution from your screenshot. Of course, it may just be wishful thinking.

Regards,
Andreas

__________________________________________________________
AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game


 User Rating: 1702   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link

Ok I've added tisscript based on c-smile from
http://www.terrainformatica.com/tiscript/ in my benchmarks.

I run the 10 loop test for all scripts couple of tousand times.

I monitored with perfmon virtual memory and page faults.
There were no increasing page faults so means none of
the script engines fragmented memory and there were no memory
leaks. For tiscript the calls Script -> Script and Script->Host
took forever so I reduced the call loop to 100 times instead
10.000 times to be able to draw the excell graph.
Here is the chart (next post):

[Edited by - Marius on June 8, 2006 4:07:16 PM]

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link



As a conclusion:

If you want to automate your game making complex
calculus in game and just dispatch events from script
you can pick Squerill. If you want to do most of the
game in script the calculus performed by Angel really rocks
and I think is a good choice.
On tiscript based on c-smile I'll wait to grow up.
:)

PS dont forget. Times were measured for 10000 calls.
For one call just divide the time by 10.000.

test scripts: (angel script)
int Main(int k, bstr s){
    int h=k;
    return h*100;
}
int CallScript(int v){ 
    int k;
    for(k=0;k<v;k++)
        Main(k,"abcdefghijklmnopqstuvxyz");
    return v;
}
int CallHost(CTest o, int n){
    int i;
    for (i = 0; i < n; i++) {
        o.Calla(i,"asdfghjkqwertyuiopasdfghjkqwertyuiopas");
    }
    return i;
}
int Calculus(int v){
    int i;
    float a;
    float b;
    float c;
    float d = 1.0;
    for(i=0; i<v;i++)
    {
        a  = i;
        b  = 10000.0/a;
        c  = a*a;
        d += (b*c);
    }
    return i;
}




test script: ( nut script)
function Main(k,s){
    local h=k;
    return h*100;
}
function CallScript(v){
    local l = 0;
    for (local i = 0; i < v; i+=1) 
        Main(i,"abcdefghijklmnopqstuvxyz");
    return l;
}
function CallHost(o,n){
    for (local i = 0; i < n; i+=1){
        o.Call(i,"asdfghjkqwertyuiopasdfghjkqwertyuiopas");
    }
    return 0;
}
function Calculus(v){
    local i;
    local a;
    local b;
    local c;
    local d = 1.0;
    for(i=0; i<v;i+=1){
        a  = i;
        b  = 10000.0/a;
        c  = a*a;
        d += (b*c);
    }
    return i;
}




and tiscript
function Main(k, k){
    var h=k;
    return h*100;
}
function CallScript(v){
    var l = 0;
    for (var i = 0; i < 10; i++) 
        Main(i,"abcdefghijklmnopqstuvxyz");
    return l;
}

function CallHost(){
    for (var i = 0; i < 10; i+=1){
        alert("asdfaksdjfhalksdhfkajsdhf");
    }
    return 0;
}

function Calculus(v){
    var a;
    var b;
    var c;
    var d = 1.0;
    for(var i=0; i<10000; i++){
        a  = i;
        b  = 10000.0/a;
        c  = a*a;
        d += (b*c);
    }
    return 0;
}




[Edited by - Marius on June 8, 2006 4:08:48 PM]

 User Rating: 865   |  Rate This User  Send Private MessageView Profile Report this Post to a Moderator | Link
Page:   1 2 »»
All times are ET (US)

Post Reply
 Last Thread Next Thread 
Forum Rules:
You may not post new threads
You may post replies
You may not edit your posts
You may not use HTML in your posts
Jump To:
Administrative Options: