|
||||||||||||||||||
Add Forum to Favorites | Send Topic To a Friend | View Forum FAQ | Track this topic Page: 1 2 »» |
Last Thread Next Thread ![]() |
| Benchmarks |
|
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
| P4 2.4 Ghz, 1GRam, win2k sp3, microsft c++ vc7 compiler 2003 edition, nonoptimized code/release build, multithreaded libraries scrip libs statically linked. I ran some tests comparing SQUIRREL2_1_0 and angelscript_2.4 In a 100 runs I get folowing results. Time is in microseconds. ---------------------------------------------------------squ--------angel----ratio ---------------------------------------------------------time in microseconds Calling void function from c/c++ to script 10000times 3894 90453 003 Calculating first 30 fibonacii nums in script 1527859 398040 3.99 Computing 100000 float math divisions/multiplications 47549 31603 1.58 Script calls another script void function 100000 times 47347 17613 2.81 Integer math storing the results into 1000 int array 429811 528918 0.85 10000 calls to C/C++ function passing an integer 10056 1604 6.24 10000 calls to C/C++ function passing a 32 chars string 9891 8484 1.17 -------------------------------------------------------------------------------- [Edited by - Marius on May 26, 2006 12:00:07 PM] |
||||
|
||||
![]() Anonymous Poster |
||||
|
||||
| Thanks for this, it's very interesting. I have been looking for a good comparison between AS and SQ as I am going to need a scripting interpreter in a performance-intense application very soon. |
||||
|
||||
![]() DvDmanDT Member since: 12/10/2004 From: Ostersund, Sweden |
||||
|
|
||||
| What the!? 9ms to call a funcion in AS (from C/C++ that is)? Am I the only one who thinks that sounds kinda much? Can you please post that benchmarking code? I have to see if the same thing happends with later versions as well.. |
||||
|
||||
![]() Deyja Member since: 3/5/2002 From: Stafford, VA, United States |
||||
|
|
||||
| No, that's 9 microseconds. 1000 microseconds = 1 millisecond. Each call took 9 microseconds, or 0.009 milliseconds. And, no, it's not a lot. The overhead of asIScriptContext::prepare is well known. It's doing a lot of work. And, as you can see, AS trumped it in most of the other tests - most noticably the calls back into C++. |
||||
|
||||
![]() DvDmanDT Member since: 12/10/2004 From: Ostersund, Sweden |
||||
|
|
||||
| Yes, sorry, my bad.. I was absolutely sure it was measured in milliseconds.. While I do understand it takes time to prepare and things, 9ms seemed extreme.. 9µs however isn't very much at all.. Again, thanks for pointing this out, I'll read more closely before I make a statement next time (rating-- on self :p).. |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
| Here is the whole project: vc7(2003). Ia a bit messy, (...) math float tests have room of improvement. I was interested in call times between c->script script->script and script->c Project can be imported in code::blocks NP. I zipped the whole structure dir with libs and headers. The exe is gen in /_bin All libs are generated in /_lib If you step in debuger make sure you set the exeutable path to ../_bin couse the dumb vc7 holds the absolute path. BM_.rar [Edited by - Marius on May 26, 2006 3:10:26 PM] |
||||
|
||||
![]() Anonymous Poster |
||||
|
||||
| Yes, this is pretty interesting-however it might be more representative and also more informative if there was some sort of wider comparison. For example, we could not only extend the current test suite but also the scripting languages supported, Lua, Perl, Python and Ruby would seem like ideal candidates for such a comparison-what do you think? This would certainly show those areas in AngelScript and Squirrel that are superior or inferior compared to the mainstream scripting languages. Just an idea :-) |
||||
|
||||
![]() Anonymous Poster |
||||
|
||||
| Pretty much a year ago, we had this thread: http://www.gamedev.net/community/forums/topic.asp?topic_id=311655&forum_id=41&gforum_id=0 |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
Quote: Didn't see any benchmarks there... |
||||
|
||||
![]() WitchLord Moderator - AngelCode Member since: 3/27/2000 From: Sao Paulo, Brazil |
||||
|
|
||||
Cool! ![]() This really shows me where I have to spend some time optimizing the code for AngelScript. :) I'll also take a look at that benchmark code you have, to see if you're doing everything the optimal way. ;) Regards, Andreas __________________________________________________________ AngelCode.com - game development and more - Reference DB - game developer references AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game |
||||
|
||||
![]() CTar Member since: 3/20/2005 From: Farum, Denmark |
||||
|
|
||||
| Interresting comparison, I would wish someone would also test a couple of other scripting languages which "compete" which AS. I don't know much about the techniques in either AS or Squirrel, but one interresting thing to note is that the void function takes much longer in AS, so one must assume that AS does quite a lot more than Squirrel, which makes the other tests unfair. For instance AS might generate some internal byte-code, while Squirrel just store the functions in a string and doesn't parse it until it's actually run. Of course it could also be a possibility that AS just parse the function everytime it's called and Squirrel actually notes that the function have been parsed once, I don't think this is what is happening though since "Script calls another script void function 100000 times" also should be slow in AS then. Quote: Might be a little off-topic, but the 2003 edition is called VC 7.1, VC 7.0 is the 2002 edition, which one do you use? |
||||
|
||||
![]() John Schultz Member since: 3/21/2005 From: Beverly Hills, CA, United States |
||||
|
|
||||
| For simple benchmarks, it is known that Lua can be faster than Squirrel. Here are Alberto's remarks regarding benchmarks and real-world applications (such as large MMO with several megabyte of data living in the VM, etc.): From: http://www.squirrel-lang.org/forums/827/ShowPost.aspx Quote: Keep in mind many games use Python, which is very slow compared to Lua/Sq/As, however it's still fast enough for real-time games. As for Squirrel vs. AngelScript, one would need to first determine how easy/complete the integration with C++ to script is. SqPlus (for Squirrel) does everything I need, quickly and easily. Perhaps someone with an unbiased opinion could review each of the scripting langauge binding systems and ease of use (Lua, AS, Sq, Python, C#, etc.). From the benchmarks presented for AS vs. Sq, the AS C++ to script calls are so slow that it would appear something is wrong (23 times slower?). 3 of the tests are close, with 3 showing AS 3-6x faster. A real game/app would need to be tested with script-to-C++ and C++-to-script calls to see if there is any significant difference between the two languages. For systems with lots of classes, inheritance, and many megs of resource data in script, Alberto states that Sq is 3x faster than Lua* (for his MMO project). More tests/benchmarks would be cool to see. *Estimate based on other comments vs. Lua. For those looking for ultimate scripting speed, look no further than C#. It compiles very quickly to code that is not much slower than native C++. Using Mono or dotGNU, game related C# code should run anywere (not clear about PSP or other memory limited systems). For something like Alberto's MMO, Squirrel may still work better due to his customized memory allocator and Squirrel's design to keep memory related GC burst free. |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
Quote: 1) Thank you Andreas. I hope this 'ugly bench startup' can be improved to became a real bechmark. I also started to profile lua (as seen in the code), but I did not have time (I did it it at my job (cheating) in respiro times) to finalize it. 2) I use VC7.1 3) .. is interesting calling script to script taking less time than script to c/c++ and vicevera. I hope I did not make any mistakes in the code. You guys can doublecheck and any lua expert can finalize the lua part. |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
| I am personally happy with either of them. As any language you find advantages and disadvantages on both sides. I hope the test are close to be accurate and I am waiting authors reply if I did anything wrong (function caching, template wrappers etc.) Thx.... Here are latest results with a more realistinc calls passing 2 params into/and from script (time in microseconds). What is interesting is that over time one of the libs (or both) eats virtual memory and execution times increases exponentially. TEST SQUIRREL2_1_0 angelscript_2.6.0 ---------------------------------------------------------------- 1K calls C++ -> Script S = 2357 A = 5716 1K calls Script -> Script S = 843 A = 1104 1K calls Script -> C++ S = 1686 A = 1357 and the scripts //angel int Main(int k, bstr s) { int h=k; return h*100; } int CallScript(int v) { int k; for(k=0;k<v;k++) Main(k,"abcdefghijklmnopqstuvxyz"); return v; } int CallHost(CTest o, int n) { int i; for (i = 0; i < n; i++) { o.Calla(i,"asdfghjkqwertyuiopasdfghjkqwertyuiopas"); } return i; } //squer function Main(k,s) { local h=k; return h*100; } function CallScript(v) { local l = 0; for (local i = 0; i < v; i+=1) Main(i,"abcdefghijklmnopqstuvxyz"); return l; } function CallHost(o,n) { for (local i = 0; i < n; i+=1) { o.Call(i,"asdfghjkqwertyuiopasdfghjkqwertyuiopas"); } return 0; } //cpp angel int r = s._pengine->RegisterObjectType("CTest", sizeof(CTest), asOBJ_CLASS); r = s._pengine->RegisterObjectMethod("CTest", "int Calla(int i, bstr s)", asMETHOD(CTest,Calla),asCALL_THISCALL); asBSTR pstr = asBStrAlloc(64); strcpy((char*)pstr,"asdfasdfasdfas2458 jhflkajsdhflkjasdhflkjadfasdf"); BOOL b = s.Load("test.ang"); int Main = s.GetFunction("test.ang","int Main(int k, bstr s)"); int CallScript = s.GetFunction("test.ang","int CallScript(int v)"); int CallHost = s.GetFunction("test.ang","int CallHost(CTest o, int n)"); FFunctor fMain(Main); FFunctor fCallScript(CallScript); FFunctor fCallHost(CallHost); DECLARE_TIME(); for(int i=0;i<1000;i++) { fMain(i,&pstr); } GETTIME(); ... //cpp sqer CTest t; SquirrelVM::Init(); SQClassDef<CTest>("CTest").func(&CTest::Call,"Call"); SquirrelObject main = SquirrelVM::CompileScript(_SC("test.nut")); SquirrelVM::RunScript(main); SquirrelFunction<int> fMain("Main"); SquirrelFunction<int> fCallScript("CallScript"); SquirrelFunction<int> fCallHost("CallHost"); const char str[]="asdfasdfasdfas2458 jhflkajsdhflkjasdhflkjadfasdf"; DECLARE_TIME(); for(int i=0;i<1000;i++) { fMain(i, str); } [Edited by - Marius on May 28, 2006 11:08:34 PM] |
||||
|
||||
![]() John Schultz Member since: 3/21/2005 From: Beverly Hills, CA, United States |
||||
|
|
||||
You can reduce SqPlus overhead slightly by changing:#define sq_argassert(arg,_index_) if (!Match(TypeWrapper<P##arg>(),v,_index_)) return sq_throwerror(v,_T("Incorrect function argument")) to #define sq_argassert(arg,_index_) Part of your Squirrel 2.1 benchmark overhead is due to SqPlus: I have not yet made an optimization pass on the code. The ease of use and generality of template binding systems come at a small runtime overhead cost (template binding system overhead varies with implementation). Squirrel/SqPlus should not leak any memory (I use it in my game; haven't seen any bug reports regarding memory leaks). |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
| I have profiled them sapratelly. Angel call C++->Script increases. |
||||
|
||||
![]() WitchLord Moderator - AngelCode Member since: 3/27/2000 From: Sao Paulo, Brazil |
||||
|
|
||||
| AngelScript doesn't have any memory leaks that I'm aware of. But it does a lot of allocating/deallocating of memory which causes memory fragmentation, which in turn may very well be what is increasing the execution time with repeated execution. I'll work to minimize the memory fragmentation in future versions. __________________________________________________________ AngelCode.com - game development and more - Reference DB - game developer references AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game |
||||
|
||||
![]() John Schultz Member since: 3/21/2005 From: Beverly Hills, CA, United States |
||||
|
|
||||
Quote: Squirrel+SqPlus also allocate and free memory (especially for instances). You might look at his benchmark and see if the code properly releases references when using AngelScript. I had a bug in SqPlus that did not properly restore the Squirrel stack that had behavior similar to what Marius described with AngelScript. |
||||
|
||||
![]() WitchLord Moderator - AngelCode Member since: 3/27/2000 From: Sao Paulo, Brazil |
||||
|
|
||||
| Indeed. I took a look at the benchmark code, and it seems that the asIScriptContext is never released after it is used. This would indeed eat a lot of memory and slow down the execution. Fixing this should improve the overall performance measured in this benchmark. But my guess is that squirrel will still win with about 100 times the performance for calling script functions. The difference is currently very large between the best and the worse performance for AngelScript: 10000 calls from C++ 2 script S = 8089 A = 722240 0.01 10000 calls from C++ 2 script S = 7890 A = 5718446 0.00 All 10 executions should be approximately the same. Another thing that can be improved is to reuse the context. Instead of creating a new context and then releasing it again for each script function that is called, you could create one context and use that for all the calls. Simply call Prepare() and then Execute() on the context. Creating multiple contexts is only useful if you want to keep various scripts running in parallel, for example co-routines, or even multiple threads. __________________________________________________________ AngelCode.com - game development and more - Reference DB - game developer references AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
| I see the diferences you have between scripts is quite big... Here is an image taken from latest results I just take on my P4 winnt 512 Mb. ![]() [Edited by - WitchLord on June 4, 2006 9:42:24 AM] |
||||
|
||||
![]() WitchLord Moderator - AngelCode Member since: 3/27/2000 From: Sao Paulo, Brazil |
||||
|
|
||||
| Your screenshot doesn't show up. I get a Gateway Timeout message when I try to view it in the browser. Yeah, my computer is getting really old, it isn't very good at running even simple benchmarks any more. It's a Celeron 800MHz with 256MB of memory, running WinXP SP 2. It makes sense that I get a large difference in performance between Squirrel and AngelScript, after all the benchmark is allocating more than 100MB of memory for AngelScript without releasing it. (10*10000 calls, where each call allocates a context with 1KB stack space)Regards, Andreas PS. I edited your post because it was missing an apostrophe in the image tag, making the page show up all wrong. __________________________________________________________ AngelCode.com - game development and more - Reference DB - game developer references AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
| I ran some test on C-smile with tiscript.lib and tiscript.dll (no source code for these libs) Here are the results: Because there is no method of holding on to a script function by a handler or somekind of ID the call has to be explicit done by function name. When calling a function from C to script and viceversa the call time is proportional with the function name length. The call of 'main'() C->c-smile function is ~47943 microseconds/10000 calls and the call of 'mainmainmainmainmain'() is ~80943 microseconds/10000 calls A call c-amile->c-smile of a void function receivineg an int is ~7609 microsecs/10000 calls tested: Dell OPTIPLEX GX260 P4 2.00 Ghz 1Gb Ram compiler VC6, release mode non-optimized. |
||||
|
||||
![]() WitchLord Moderator - AngelCode Member since: 3/27/2000 From: Sao Paulo, Brazil |
||||
|
|
||||
| Are you going to update the benchmark program with the suggestions I made? I just made a little test on my own (while optimizing the new version of AngelScript) and saw that reusing the context for each call, instead of creating a new context, gives a performance increase of about 10-15 times. You should get the same performance increase in the current version of AngelScript and it would be great if you could update your benchmark program to reflect that. Especially since it may show that AngelScript is actually faster than Squirrel even with this trivial test of calling a void function. I'm basing this assumption on the first execution from your screenshot. Of course, it may just be wishful thinking.Regards, Andreas __________________________________________________________ AngelCode.com - game development and more - Reference DB - game developer references AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
| Ok I've added tisscript based on c-smile from http://www.terrainformatica.com/tiscript/ in my benchmarks. I run the 10 loop test for all scripts couple of tousand times. I monitored with perfmon virtual memory and page faults. There were no increasing page faults so means none of the script engines fragmented memory and there were no memory leaks. For tiscript the calls Script -> Script and Script->Host took forever so I reduced the call loop to 100 times instead 10.000 times to be able to draw the excell graph. Here is the chart (next post): [Edited by - Marius on June 8, 2006 4:07:16 PM] |
||||
|
||||
![]() Marius Member since: 10/3/2001 From: Canada |
||||
|
|
||||
![]() As a conclusion: If you want to automate your game making complex calculus in game and just dispatch events from script you can pick Squerill. If you want to do most of the game in script the calculus performed by Angel really rocks and I think is a good choice. On tiscript based on c-smile I'll wait to grow up. :) PS dont forget. Times were measured for 10000 calls. For one call just divide the time by 10.000. test scripts: (angel script) int Main(int k, bstr s){ int h=k; return h*100; } int CallScript(int v){ int k; for(k=0;k<v;k++) Main(k,"abcdefghijklmnopqstuvxyz"); return v; } int CallHost(CTest o, int n){ int i; for (i = 0; i < n; i++) { o.Calla(i,"asdfghjkqwertyuiopasdfghjkqwertyuiopas"); } return i; } int Calculus(int v){ int i; float a; float b; float c; float d = 1.0; for(i=0; i<v;i++) { a = i; b = 10000.0/a; c = a*a; d += (b*c); } return i; } test script: ( nut script)
function Main(k,s){
local h=k;
return h*100;
}
function CallScript(v){
local l = 0;
for (local i = 0; i < v; i+=1)
Main(i,"abcdefghijklmnopqstuvxyz");
return l;
}
function CallHost(o,n){
for (local i = 0; i < n; i+=1){
o.Call(i,"asdfghjkqwertyuiopasdfghjkqwertyuiopas");
}
return 0;
}
function Calculus(v){
local i;
local a;
local b;
local c;
local d = 1.0;
for(i=0; i<v;i+=1){
a = i;
b = 10000.0/a;
c = a*a;
d += (b*c);
}
return i;
}
and tiscript
function Main(k, k){
var h=k;
return h*100;
}
function CallScript(v){
var l = 0;
for (var i = 0; i < 10; i++)
Main(i,"abcdefghijklmnopqstuvxyz");
return l;
}
function CallHost(){
for (var i = 0; i < 10; i+=1){
alert("asdfaksdjfhalksdhfkajsdhf");
}
return 0;
}
function Calculus(v){
var a;
var b;
var c;
var d = 1.0;
for(var i=0; i<10000; i++){
a = i;
b = 10000.0/a;
c = a*a;
d += (b*c);
}
return 0;
}
[Edited by - Marius on June 8, 2006 4:08:48 PM] |
||||
|
||||
|
Page: 1 2 »» All times are ET (US) ![]() |
Last Thread Next Thread ![]() |
|