Crash on 64bit *nix

Started by
7 comments, last by _Vicious_ 12 years, 10 months ago
Hello!

We've been experiencing occasional crashes related to Angelscript, which seem to be related to some invalid pointer arithmetic in AS. Here's a sample stacktrace:
]quit
==== G_Shutdown ====
*** glibc detected *** ./warsow.x86_64: munmap_chunk(): invalid pointer: 0x0000000027994310 ***
======= Backtrace: =========
/lib/libc.so.6(+0x7342a)[0x7f00ddc4642a]
./libs/angelwrap_x86_64.so(_ZN10asCContext9UnprepareEv+0x10f)[0x7f00dc2d36c5]
./libs/angelwrap_x86_64.so(_ZN10asCContext12DetachEngineEv+0x47)[0x7f00dc2d3009]
./libs/angelwrap_x86_64.so(_ZN10asCContextD1Ev+0x2b)[0x7f00dc2d2e05]
./libs/angelwrap_x86_64.so(_ZN10asCContext7ReleaseEv+0x46)[0x7f00dc2d2fa2]
./libs/angelwrap_x86_64.so(qasReleaseContext+0xac)[0x7f00dc276a39]
./libs/angelwrap_x86_64.so(qasReleaseScriptEngine+0x77)[0x7f00dc276b88]
/home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so(G_asShutdownGametypeScript+0x3d)[0x7f00d1d24ad9]
/home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so(G_Shutdown+0x2d)[0x7f00d1d0d811]
./warsow.x86_64[0x453c0c]
./warsow.x86_64[0x44ef2f]
./warsow.x86_64[0x46401c]
./warsow.x86_64[0x462eb7]
./warsow.x86_64[0x462ec7]
./warsow.x86_64[0x41c084]
./warsow.x86_64[0x41a6b6]
./warsow.x86_64[0x46b234]
./warsow.x86_64[0x467305]
./warsow.x86_64[0x40ef9f]
./warsow.x86_64[0x4771b8]
/lib/libc.so.6(__libc_start_main+0xed)[0x7f00ddbf3f6d]
./warsow.x86_64[0x4053b9]
======= Memory map: ========
00400000-0052c000 r-xp 00000000 08:01 6949399 /home/hc/code/wsw/svn/data/warsow/warsow.x86_64
0072b000-00733000 rw-p 0012b000 08:01 6949399 /home/hc/code/wsw/svn/data/warsow/warsow.x86_64
00733000-014bb000 rw-p 00000000 00:00 0
017ba000-2817b000 rw-p 00000000 00:00 0 [heap]
40183000-401fa000 rw-p 00000000 00:00 0
40948000-4094a000 r-xs 00000000 08:13 292261 /tmp/glhdiq0E (deleted)
7f00c4000000-7f00c4159000 rw-p 00000000 00:00 0
7f00c4159000-7f00c8000000 ---p 00000000 00:00 0
7f00caab3000-7f00ceab4000 rw-s 00000000 00:0f 32433 /dev/shm/pulse-shm-3002665678
7f00ceab4000-7f00ceab9000 r-xp 00000000 08:13 486764 /lib/libnss_dns-2.13.so
7f00ceab9000-7f00cecb8000 ---p 00005000 08:13 486764 /lib/libnss_dns-2.13.so
7f00cecb8000-7f00cecb9000 r--p 00004000 08:13 486764 /lib/libnss_dns-2.13.so
7f00cecb9000-7f00cecba000 rw-p 00005000 08:13 486764 /lib/libnss_dns-2.13.so
7f00cecba000-7f00cecbb000 ---p 00000000 00:00 0
7f00cecbb000-7f00cf4bb000 rw-p 00000000 00:00 0
7f00d12d3000-7f00d1cb4000 rw-p 00000000 00:00 0
7f00d1cb4000-7f00d1d91000 r-xp 00000000 08:01 79043 /home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so
7f00d1d91000-7f00d1f90000 ---p 000dd000 08:01 79043 /home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so
7f00d1f90000-7f00d1f9d000 rw-p 000dc000 08:01 79043 /home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so
7f00d1f9d000-7f00d34bc000 rw-p 00000000 00:00 0
7f00d34bc000-7f00d34c0000 r-xp 00000000 08:13 489787 /lib/libattr.so.1.1.0
7f00d34c0000-7f00d36bf000 ---p 00004000 08:13 489787 /lib/libattr.so.1.1.0
7f00d36bf000-7f00d36c0000 rw-p 00003000 08:13 489787 /lib/libattr.so.1.1.0
7f00d36c0000-7f00d36d3000 r-xp 00000000 08:13 486750 /lib/libresolv-2.13.so
7f00d36d3000-7f00d38d3000 ---p 00013000 08:13 486750 /lib/libresolv-2.13.so
7f00d38d3000-7f00d38d4000 r--p 00013000 08:13 486750 /lib/libresolv-2.13.so
7f00d38d4000-7f00d38d5000 rw-p 00014000 08:13 486750 /lib/libresolv-2.13.so
7f00d38d5000-7f00d38d7000 rw-p 00000000 00:00 0
7f00d38d7000-7f00d3b8a000 r-xp 00000000 08:13 440664 /usr/lib/libvorbisenc.so.2.0.8
7f00d3b8a000-7f00d3d89000 ---p 002b3000 08:13 440664 /usr/lib/libvorbisenc.so.2.0.8
7f00d3d89000-7f00d3da5000 rw-p 002b2000 08:13 440664 /usr/lib/libvorbisenc.so.2.0.8
7f00d3da5000-7f00d3def000 r-xp 00000000 08:13 440929 /usr/lib/libFLAC.so.8.2.0
7f00d3def000-7f00d3fef000 ---p 0004a000 08:13 440929 /usr/lib/libFLAC.so.8.2.0
7f00d3fef000-7f00d3ff1000 rw-p 0004a000 08:13 440929 /usr/lib/libFLAC.so.8.2.0
7f00d3ff1000-7f00d4006000 r-xp 00000000 08:13 486740 /lib/libnsl-2.13.so
7f00d4006000-7f00d4205000 ---p 00015000 08:13 486740 /lib/libnsl-2.13.so
7f00d4205000-7f00d4206000 r--p 00014000 08:13 486740 /lib/libnsl-2.13.so
7f00d4206000-7f00d4207000 rw-p 00015000 08:13 486740 /lib/libnsl-2.13.so
7f00d4207000-7f00d4209000 rw-p 00000000 00:00 0
7f00d4209000-7f00d4218000 r-xp 00000000 08:13 439306 /usr/lib/libXi.so.6.1.0
7f00d4218000-7f00d4417000 ---p 0000f000 08:13 439306 /usr/lib/libXi.so.6.1.0
7f00d4417000-7f00d4418000 rw-p 0000e000 08:13 439306 /usr/lib/libXi.so.6.1.0
7f00d4418000-7f00d441b000 r-xp 00000000 08:13 489077 /lib/libuuid.so.1.3.0
7f00d441b000-7f00d461b000 ---p 00003000 08:13 489077 /lib/libuuid.so.1.3.0
7f00d461b000-7f00d461c000 rw-p 00003000 08:13 489077 /lib/libuuid.so.1.3.0
7f00d461c000-7f00d4620000 r-xp 00000000 08:13 486845 /lib/libcap.so.2.21
7f00d4620000-7f00d481f000 ---p 00004000 08:13 486845 /lib/libcap.so.2.21
7f00d481f000-7f00d4820000 rw-p 00003000 08:13 486845 /lib/libcap.so.2.21
7f00d4820000-7f00d4862000 r-xp 00000000 08:13 456598 /usr/lib/libdbus-1.so.3.5.5
7f00d4862000-7f00d4a62000 ---p 00042000 08:13 456598 /usr/lib/libdbus-1.so.3.5.5
7f00d4a62000-7f00d4a63000 r--p 00042000 08:13 456598 /usr/lib/libdbus-1.so.3.5.5
7f00d4a63000-7f00d4a64000 rw-p 00043000 08:13 456598 /usr/lib/libdbus-1.so.3.5.5
7f00d4a64000-7f00d4a69000 r-xp 00000000 08:13 440919 /usr/lib/libasyncns.so.0.3.1
7f00d4a69000-7f00d4c68000 ---p 00005000 08:13 440919 /usr/lib/libasyncns.so.0.3.1
7f00d4c68000-7f00d4c69000 rw-p 00004000 08:13 440919 /usr/lib/libasyncns.so.0.3.1
7f00d4c69000-7f00d4cc9000 r-xp 00000000 08:13 440934 /usr/lib/libsndfile.so.1.0.24
7f00d4cc9000-7f00d4ec9000 ---p 00060000 08:13 440934 /usr/lib/libsndfile.so.1.0.24
7f00d4ec9000-7f00d4ecc000 rw-p 00060000 08:13 440934 /usr/lib/libsndfile.so.1.0.24
7f00d4ecc000-7f00d4ed0000 rw-p 00000000 00:00 0
7f00d4ed0000-7f00d4ed8000 r-xp 00000000 08:13 461557 /usr/lib/libwrap.so.0.7.6
7f00d4ed8000-7f00d50d7000 ---p 00008000 08:13 461557 /usr/lib/libwrap.so.0.7.6
7f00d50d7000-7f00d50d9000 rw-p 00007000 08:13 461557 /usr/lib/libwrap.so.0.7.6==== G_Shutdown ====
*** glibc detected *** ./warsow.x86_64: munmap_chunk(): invalid pointer: 0x00000000043ca350 ***
======= Backtrace: =========
/lib/libc.so.6(+0x7342a)[0x7f00ddc4642a]
./libs/angelwrap_x86_64.so(_ZN10asCContext9UnprepareEv+0x10f)[0x7f00dc2d36c5]
./libs/angelwrap_x86_64.so(_ZN10asCContext12DetachEngineEv+0x47)[0x7f00dc2d3009]
./libs/angelwrap_x86_64.so(_ZN10asCContextD1Ev+0x2b)[0x7f00dc2d2e05]
./libs/angelwrap_x86_64.so(_ZN10asCContext7ReleaseEv+0x46)[0x7f00dc2d2fa2]
./libs/angelwrap_x86_64.so(qasReleaseContext+0xac)[0x7f00dc276a39]
./libs/angelwrap_x86_64.so(qasReleaseScriptEngine+0x77)[0x7f00dc276b88]
/home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so(G_asShutdownGametypeScript+0x3d)[0x7f00d1d24ad9]
/home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so(G_Shutdown+0x2d)[0x7f00d1d0d811]
./warsow.x86_64[0x453c0c]
./warsow.x86_64[0x44ef2f]
./warsow.x86_64[0x44915d]
./warsow.x86_64[0x40d91b]
./warsow.x86_64[0x4769cb]
/lib/libc.so.6(+0x34780)[0x7f00ddc07780]
/lib/libc.so.6(gsignal+0x35)[0x7f00ddc07705]
/lib/libc.so.6(abort+0x17b)[0x7f00ddc08b7b]
/lib/libc.so.6(+0x6d5a3)[0x7f00ddc405a3]
/lib/libc.so.6(+0x7342a)[0x7f00ddc4642a]
./libs/angelwrap_x86_64.so(_ZN10asCContext9UnprepareEv+0x10f)[0x7f00dc2d36c5]
./libs/angelwrap_x86_64.so(_ZN10asCContext12DetachEngineEv+0x47)[0x7f00dc2d3009]
./libs/angelwrap_x86_64.so(_ZN10asCContextD1Ev+0x2b)[0x7f00dc2d2e05]
./libs/angelwrap_x86_64.so(_ZN10asCContext7ReleaseEv+0x46)[0x7f00dc2d2fa2]
./libs/angelwrap_x86_64.so(qasReleaseContext+0xac)[0x7f00dc276a39]
./libs/angelwrap_x86_64.so(qasReleaseScriptEngine+0x77)[0x7f00dc276b88]
/home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so(G_asShutdownGametypeScript+0x3d)[0x7f00d1d24ad9]
/home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so(G_Shutdown+0x2d)[0x7f00d1d0d811]
./warsow.x86_64[0x453c0c]
./warsow.x86_64[0x44ef2f]
./warsow.x86_64[0x46401c]
./warsow.x86_64[0x462eb7]
./warsow.x86_64[0x462ec7]
./warsow.x86_64[0x41c084]
./warsow.x86_64[0x41a6b6]
./warsow.x86_64[0x46b234]
./warsow.x86_64[0x467305]
./warsow.x86_64[0x40ef9f]
./warsow.x86_64[0x4771b8]
/lib/libc.so.6(__libc_start_main+0xed)[0x7f00ddbf3f6d]
./warsow.x86_64[0x4053b9]
======= Memory map: ========
00400000-0052c000 r-xp 00000000 08:01 6949399 /home/hc/code/wsw/svn/data/warsow/warsow.x86_64
0072b000-00733000 rw-p 0012b000 08:01 6949399 /home/hc/code/wsw/svn/data/warsow/warsow.x86_64
00733000-014bb000 rw-p 00000000 00:00 0
017ba000-2817b000 rw-p 00000000 00:00 0 [heap]
40183000-401fa000 rw-p 00000000 00:00 0
40948000-4094a000 r-xs 00000000 08:13 292261 /tmp/glhdiq0E (deleted)
7f00c4000000-7f00c4159000 rw-p 00000000 00:00 0
7f00c4159000-7f00c8000000 ---p 00000000 00:00 0
7f00caab3000-7f00ceab4000 rw-s 00000000 00:0f 32433 /dev/shm/pulse-shm-3002665678
7f00ceab4000-7f00ceab9000 r-xp 00000000 08:13 486764 /lib/libnss_dns-2.13.so
7f00ceab9000-7f00cecb8000 ---p 00005000 08:13 486764 /lib/libnss_dns-2.13.so
7f00cecb8000-7f00cecb9000 r--p 00004000 08:13 486764 /lib/libnss_dns-2.13.so
7f00cecb9000-7f00cecba000 rw-p 00005000 08:13 486764 /lib/libnss_dns-2.13.so
7f00cecba000-7f00cecbb000 ---p 00000000 00:00 0
7f00cecbb000-7f00cf4bb000 rw-p 00000000 00:00 0
7f00d12d3000-7f00d1cb4000 rw-p 00000000 00:00 0
7f00d1cb4000-7f00d1d91000 r-xp 00000000 08:01 79043 /home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so
7f00d1d91000-7f00d1f90000 ---p 000dd000 08:01 79043 /home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so
7f00d1f90000-7f00d1f9d000 rw-p 000dc000 08:01 79043 /home/hc/.warsow-0.7/basewsw/tempmodules656/game_x86_64.so
7f00d1f9d000-7f00d34bc000 rw-p 00000000 00:00 0
7f00d34bc000-7f00d34c0000 r-xp 00000000 08:13 489787 /lib/libattr.so.1.1.0
7f00d34c0000-7f00d36bf000 ---p 00004000 08:13 489787 /lib/libattr.so.1.1.0
7f00d36bf000-7f00d36c0000 rw-p 00003000 08:13 489787 /lib/libattr.so.1.1.0
7f00d36c0000-7f00d36d3000 r-xp 00000000 08:13 486750 /lib/libresolv-2.13.so
7f00d36d3000-7f00d38d3000 ---p 00013000 08:13 486750 /lib/libresolv-2.13.so
7f00d38d3000-7f00d38d4000 r--p 00013000 08:13 486750 /lib/libresolv-2.13.so
7f00d38d4000-7f00d38d5000 rw-p 00014000 08:13 486750 /lib/libresolv-2.13.so
7f00d38d5000-7f00d38d7000 rw-p 00000000 00:00 0
7f00d38d7000-7f00d3b8a000 r-xp 00000000 08:13 440664 /usr/lib/libvorbisenc.so.2.0.8
7f00d3b8a000-7f00d3d89000 ---p 002b3000 08:13 440664 /usr/lib/libvorbisenc.so.2.0.8
7f00d3d89000-7f00d3da5000 rw-p 002b2000 08:13 440664 /usr/lib/libvorbisenc.so.2.0.8
7f00d3da5000-7f00d3def000 r-xp 00000000 08:13 440929 /usr/lib/libFLAC.so.8.2.0
7f00d3def000-7f00d3fef000 ---p 0004a000 08:13 440929 /usr/lib/libFLAC.so.8.2.0
7f00d3fef000-7f00d3ff1000 rw-p 0004a000 08:13 440929 /usr/lib/libFLAC.so.8.2.0
7f00d3ff1000-7f00d4006000 r-xp 00000000 08:13 486740 /lib/libnsl-2.13.so
7f00d4006000-7f00d4205000 ---p 00015000 08:13 486740 /lib/libnsl-2.13.so
7f00d4205000-7f00d4206000 r--p 00014000 08:13 486740 /lib/libnsl-2.13.so
7f00d4206000-7f00d4207000 rw-p 00015000 08:13 486740 /lib/libnsl-2.13.so
7f00d4207000-7f00d4209000 rw-p 00000000 00:00 0
7f00d4209000-7f00d4218000 r-xp 00000000 08:13 439306 /usr/lib/libXi.so.6.1.0
7f00d4218000-7f00d4417000 ---p 0000f000 08:13 439306 /usr/lib/libXi.so.6.1.0
7f00d4417000-7f00d4418000 rw-p 0000e000 08:13 439306 /usr/lib/libXi.so.6.1.0
7f00d4418000-7f00d441b000 r-xp 00000000 08:13 489077 /lib/libuuid.so.1.3.0
7f00d441b000-7f00d461b000 ---p 00003000 08:13 489077 /lib/libuuid.so.1.3.0
7f00d461b000-7f00d461c000 rw-p 00003000 08:13 489077 /lib/libuuid.so.1.3.0
7f00d461c000-7f00d4620000 r-xp 00000000 08:13 486845 /lib/libcap.so.2.21
7f00d4620000-7f00d481f000 ---p 00004000 08:13 486845 /lib/libcap.so.2.21
7f00d481f000-7f00d4820000 rw-p 00003000 08:13 486845 /lib/libcap.so.2.21
7f00d4820000-7f00d4862000 r-xp 00000000 08:13 456598 /usr/li


Note the pointer address '0x0000000027994310' which looks suspiciously like a 64bit pointer truncated to 32bits. The only relevant bit in Unprepare which could lead to such crashes in my opinion is the asDELETEARRAY loop, here my analysis actually stops :) Could you please take a look at what's actually going on there?
Advertisement
This type of error is usually caused by some type of memory invasion that ends up corrupting the heap. The root cause of the error can be anywhere, and may not even be related to the actual object that is being deleted.

I suggest you run your application with Valgrind. It is an excellent open source tool for Linux that is used to find these kind of bad behaviours in the code. droz has set up an automatic build bot for AngelScript that runs Valgrind on the regression test suite for every check-in I do, and it has been very helpful in cleaning up some odd memory leaks and invasions for me in the past.

Please let me know if Valgrind finds something.

Other than that it would be necessary to find some steps that reliably reproduces the problem so that it can be debugged. I don't exclude a bug in AngelScript, but without being able to reproduce it myself it will be nearly impossible to find.

AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game


This type of error is usually caused by some type of memory invasion that ends up corrupting the heap. The root cause of the error can be anywhere, and may not even be related to the actual object that is being deleted.

I suggest you run your application with Valgrind. It is an excellent open source tool for Linux that is used to find these kind of bad behaviours in the code. droz has set up an automatic build bot for AngelScript that runs Valgrind on the regression test suite for every check-in I do, and it has been very helpful in cleaning up some odd memory leaks and invasions for me in the past.

Please let me know if Valgrind finds something.

Other than that it would be necessary to find some steps that reliably reproduces the problem so that it can be debugged. I don't exclude a bug in AngelScript, but without being able to reproduce it myself it will be nearly impossible to find.


I ran valgrind with the 64-Bit build of test_feature and everything is good:
[source]
---------
MEMORY STATISTICS
number of allocations : 377520
max allocated memory at any one time : 247788
max number of simultaneous allocations: 2531
total amount of allocated memory : 49504160
medium size of allocations : 131
--------------------------------------------
All of the tests passed with success.

==6141==
==6141== HEAP SUMMARY:
==6141== in use at exit: 0 bytes in 0 blocks
==6141== total heap usage: 1,137,495 allocs, 1,137,495 frees, 86,148,674 bytes allocated
==6141==
==6141== All heap blocks were freed -- no leaks are possible
==6141==
[/source]
Thanks, Jeremy. It's good to know there are at least no known problems in AngelScript, even on 64bit. But I still can't rule out a possible problem that is only encountered in some scenario produced by Vicious code.

AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game

We'll keep running valgrind on our codebase to try and possibly catch the bug, it's not 100% reproducable on each run unfortunately..
I don't know if this may be related, but I *think* there may be an issue with asIScriptContext bleeding an object's reference count if you Prepare(), SetObject() and then Unprepare() or Release() the context without Execute()ing it - because Unprepare() releases the object but its reference count is only incremented if you Execute() (I guess?). And if this happens repeatedly it can cause a crash in Unprepare(). I'm just getting started with AngelCode so I may be wrong though - I haven't been able to track through what is happening fully yet. This is on VS2010, x86 compile. At any rate I was forgetting to execute when I found it which is probably an uncommon occurrence.
Hi immortius,

you're right. I reviewed the code and this can indeed cause a crash, as it would release the object without the application knowing it. I'll add some extra code to avoid this from happening.

thanks,
Andreas

AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game

Immortius,

I've fixed the problem with Unprepare releasing the object if Execute hadn't been called yet. The changes are done in revision 903.



_Vicious_

Any luck with reproducing the crash with Valgrind?

AngelCode.com - game development and more - Reference DB - game developer references
AngelScript - free scripting library - BMFont - free bitmap font generator - Tower - free puzzle game

Nope, none so far :(

This topic is closed to new replies.

Advertisement