• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
Zylann

How to limit memory used by AngelScript ?

9 posts in this topic

Hi,

I'm currently writing a script sanbox with AngelScript, but I don't understand how to limit the memory used by the scripts.
I've looked at asSetGlobalMemoryFunctions, but I'm getting stuck on writing the free function :

[CODE]size_t g_allocatedMem = 0; // memory currently allocated. Must be 0 at program end.
size_t g_nbAllocations = 0; // ++ on alloc, -- on dealloc. Must be 0 at program end.

void * customAlloc(size_t amount)
{
g_nbAllocations++;
// Check memory used
g_allocatedMem += amount;
if(g_allocatedMem > MAX)
{
// Error...
}
return malloc(amount);
}

void customFree(void * ptr)
{
g_nbAllocations--;
g_allocatedMem -= ... // How to know freed data size?
free(ptr);
}[/CODE]
How can I do this?

PS: I also started by counting the number of calls to customAlloc and customFree, but g_nbAllocations is non-zero at the end (690). Edited by Zylann
0

Share this post


Link to post
Share on other sites
In order to know how much is deallocated you'll need to keep track of how much was allocated in one way or another.

If you don't care much about portability I'm pretty sure the malloc() function stores this in the allocated block of memory, by examining the bytes just before the returned pointer you'll probably find the size there. This will of course depend on the implementation that the compiler vendor provides. But if you value portability you should do it yourself.

The easiest is to simply allocate a few extra bytes with each allocation and store the size of the allocated block in those bytes. Another option is to keep an associative map with the pointer is the key and the size is the value.

In the test_feature project that I use to test the library I've implemented a memory manager like this:

[code]

//#define TRACK_LOCATIONS
//#define TRACK_SIZES

static int numAllocs = 0;
static int numFrees = 0;
static size_t currentMemAlloc = 0;
static size_t maxMemAlloc = 0;
static int maxNumAllocsSameTime = 0;
static asQWORD sumAllocSize = 0;

static map<void*,size_t> memSize;
static map<void*,int> memCount;

#ifdef TRACK_SIZES
static map<size_t,int> meanSize;
#endif

#ifdef TRACK_LOCATIONS
struct loc
{
const char *file;
int line;

bool operator <(const loc &other) const
{
if( file < other.file ) return true;
if( file == other.file && line < other.line ) return true;
return false;
}
};

static map<loc, int> locCount;
#endif

void *MyAllocWithStats(size_t size, const char *file, int line)
{
// Avoid compiler warning when variables aren't used
UNUSED_VAR(line);
UNUSED_VAR(file);

// Allocate the memory
void *ptr = malloc(size);

// Count number of allocations made
numAllocs++;

// Count total amount of memory allocated
sumAllocSize += size;

// Update currently allocated memory
currentMemAlloc += size;
if( currentMemAlloc > maxMemAlloc ) maxMemAlloc = currentMemAlloc;

// Remember the size of the memory allocated at this pointer
memSize.insert(map<void*,size_t>::value_type(ptr,size));

// Remember the currently allocated memory blocks, with the allocation number so that we can debug later
memCount.insert(map<void*,int>::value_type(ptr,numAllocs));

// Determine the maximum number of allocations at the same time
if( numAllocs - numFrees > maxNumAllocsSameTime )
maxNumAllocsSameTime = numAllocs - numFrees;

#ifdef TRACK_SIZES
// Determine the mean size of the memory allocations
map<size_t,int>::iterator i = meanSize.find(size);
if( i != meanSize.end() )
i->second++;
else
meanSize.insert(map<size_t,int>::value_type(size,1));
#endif

#ifdef TRACK_LOCATIONS
// Count the number of allocations for each location in the library
loc l = {file, line};
map<loc, int>::iterator i2 = locCount.find(l);
if( i2 != locCount.end() )
i2->second++;
else
locCount.insert(map<loc,int>::value_type(l,1));
#endif

return ptr;
}

void MyFreeWithStats(void *address)
{
// Count the number of deallocations made
numFrees++;

// Remove the memory block from the list of allocated blocks
map<void*,size_t>::iterator i = memSize.find(address);
if( i != memSize.end() )
{
// Decrease the current amount of allocated memory
currentMemAlloc -= i->second;
memSize.erase(i);
}
else
assert(false);

// Verify which memory we are currently removing so we know we did the allocation, and where it was allocated
map<void*,int>::iterator i2 = memCount.find(address);
if( i2 != memCount.end() )
{
// int numAlloc = i2->second;
memCount.erase(i2);
}
else
assert(false);

// Free the actual memory
free(address);
}

void InstallMemoryManager()
{
#ifdef TRACK_LOCATIONS
assert( strstr(asGetLibraryOptions(), " AS_DEBUG ") );
#endif

asSetGlobalMemoryFunctions((asALLOCFUNC_t)MyAllocWithStats, MyFreeWithStats);
}

void PrintAllocIndices()
{
map<void*,int>::iterator i = memCount.begin();
while( i != memCount.end() )
{
printf("%d\n", i->second);
i++;
}
}

void RemoveMemoryManager()
{
asThreadCleanup();

PrintAllocIndices();

// assert( numAllocs == numFrees );
// assert( currentMemAlloc == 0 );

printf("---------\n");
printf("MEMORY STATISTICS\n");
printf("number of allocations : %d\n", numAllocs); // 125744
printf("max allocated memory at any one time : %d\n", (int)maxMemAlloc); // 121042
printf("max number of simultaneous allocations: %d\n", maxNumAllocsSameTime); // 2134
printf("total amount of allocated memory : %d\n", (int)sumAllocSize); // 10106765
printf("medium size of allocations : %d\n", (int)sumAllocSize/numAllocs);

#ifdef TRACK_SIZES
// Find the mean size of allocations
map<size_t,int>::iterator i = meanSize.begin();
int n = 0;
int meanAllocSize = 0;
while( i != meanSize.end() )
{
if( n + i->second > numAllocs / 2 )
{
meanAllocSize = (int)i->first;
break;
}

n += i->second;
i++;
}
printf("mean size of allocations : %d\n", meanAllocSize);
printf("smallest allocation size : %d\n", meanSize.begin()->first);
printf("largest allocation size : %d\n", meanSize.rbegin()->first);
printf("number of different allocation sizes : %d\n", meanSize.size());

// Print allocation sizes
i = meanSize.begin();
while( i != meanSize.end() )
{
if( i->second >= 1000 )
printf("alloc size %d: %d\n", i->first, i->second);
i++;
}
#endif

#ifdef TRACK_LOCATIONS
// Print allocation counts per location
map<loc,int>::iterator i2 = locCount.begin();
while( i2 != locCount.end() )
{
const char *file = i2->first.file;
int line = i2->first.line;
int count = i2->second;
printf("%s (%d): %d\n", file, line, count);
i2++;
}
#endif

asResetGlobalMemoryFunctions();
}

int GetNumAllocs()
{
return numAllocs;
}
[/code]


I'm not aware of any memory leaks in the library. Do you release all objects before you check the final g_nbAllocations? Edited by Andreas Jonsson
1

Share this post


Link to post
Share on other sites
Thanks for your quick answer.
Actually, I forgot to call cleanup functions, now everything is ok about memory leaks.

I think that adding extra-data to memory blocks will be better for portability.
I had also imagined to first allocate a big memory block the size of the memory limit (not so much),
and then using it to store script's data, taking care not run out of it... but I don't know how to manage this.

In my project, every user will be able to send scripts to a server, that will run them (each in separated sandboxes),
so I have to secure the script manager a lot to avoid spamming.
Also, are the scriptarray or scriptstdstring addons using the same alloc functions? Or do I have to modify them?

I will examine your code, it will be very useful for the moment [img]http://public.gamedev.net//public/style_emoticons/default/smile.png[/img] Edited by Zylann
0

Share this post


Link to post
Share on other sites
The add-ons are not using the memory allocation functions that the engine do. If you want them to allocate memory through the same routines you'll have to modify them yourself.
0

Share this post


Link to post
Share on other sites
Hi,

I've successfully implemented the custom alloc and free functions, but now I have a new problem :

When I reach the memory limit (actually just before), how do I inform the program to stop the sandbox without crash?
Because I'm currently returning 0 (out of memory) in memAlloc, but then my program gets a SIGSEV and closes immediately.

Should I throw an exception instead?

My code :
[CODE]
void * Sandbox::memAlloc1(size_t size)
{
// Check memory used
m_state.currentMemUse += size;
if(m_state.currentMemUse >= m_state.memLimit)
{
return 0; // out of memory
}

if(m_state.currentMemUse > m_state.memUsePeak)
m_state.memUsePeak = m_state.currentMemUse;

// Allocate memory + hidden field size
void * ptr = malloc(size + sizeof(hidden_field_t));
if(ptr == 0)
return 0;

// Setting hidden size field
hidden_field_t * hptr = (hidden_field_t*)ptr;
*hptr = size;
ptr += sizeof(hidden_field_t); // ptr is then shifted

m_state.nbAllocs++;

if(m_statsEnabled)
m_stats.push_back(m_state);

return ptr;
}

void Sandbox::memFree1(void * ptr)
{
// Get hidden size field
ptr -= sizeof(hidden_field_t); // Unshift
hidden_field_t * hptr = (hidden_field_t*)ptr;

m_state.currentMemUse -= *hptr;

// Free memory
free(ptr);
m_state.nbFrees++;

if(m_statsEnabled)
m_stats.push_back(m_state);
}[/CODE] Edited by Zylann
0

Share this post


Link to post
Share on other sites
Can you show me where the SIGSEV is occurring? i.e. the callstack?

Returning 0 is the right way as I'm not using exceptions at all in AngelScript, but I must confess that AngelScript probably doesn't handle out-of-memory situations correctly in all locations. I'll have to fix these as they are found out.
0

Share this post


Link to post
Share on other sites
On a "Hello World" script, the memory use peak is 63221.
If I set the limit to 63000, SIGSEV occurs with the following call stack :

[CODE]#0 0x80a9efe asCMap<unsigned int, asCThreadLocalData*>::Insert(this=0x81a04e8, key=@0xbffff444: 3083679488, value=@0xbffff448: 0x0) (../../source/as_map.h:159)
#1 0x80a9800 asCThreadManager::SetLocalData(this=0x81a04e4, threadId=3083679488, tld=0x0) (../../source/as_thread.cpp:204)
#2 0x80a98fa asCThreadManager::GetLocalData() (../../source/as_thread.cpp:232)
#3 0x807a069 asPushActiveContext(ctx=0x81b8f6c) (../../source/as_context.cpp:148)
#4 0x807c035 asCContext::Execute(this=0x81b8f6c) (../../source/as_context.cpp:1003)
#5 0x80799d7 program(argc=1, argv=0xbffff8a4) (/home/marc/PROJETS/INFO/CPP/Botcraft/main.cpp:153)
#6 0x8079cd3 main(argc=1, argv=0xbffff8a4) (/home/marc/PROJETS/INFO/CPP/Botcraft/main.cpp:205)[/CODE]

I added some stuff to the script (one class with 2 attributes and 2 methods) and set the limit to 65000.
SIGSEV occured again as the new limit has been reached :

[CODE]#0 0x80c66d9 cByteInstruction::AddAfter(this=0x81ba124, nextCode=0x0) (../../source/as_bytecode.cpp:2474)
#1 0x80c3cee asCByteCode::AddInstruction(this=0xbfffe64c) (../../source/as_bytecode.cpp:1308)
#2 0x80c5810 asCByteCode::InstrSHORT_DW(this=0xbfffe64c, bc=asBC_ADDSi, a=20, b=134217741) (../../source/as_bytecode.cpp:2214)
#3 0x80e388d asCCompiler::CompileVariableAccess(this=0xbffff0ec, name=..., scope=..., ctx=0xbfffe64c, errNode=0x81b949c, isOptional=false, noFunction=false, objType=0x0) (../../source/as_compiler.cpp:6493)
#4 0x80e4ee4 asCCompiler::CompileExpressionValue(this=0xbffff0ec, node=0x81b949c, ctx=0xbfffe64c) (../../source/as_compiler.cpp:6806)
#5 0x80e2e4d asCCompiler::CompileExpressionTerm(this=0xbffff0ec, node=0x81b944c, ctx=0xbfffeb9c) (../../source/as_compiler.cpp:6348)
#6 0x80e2c06 asCCompiler::CompilePostFixExpression(this=0xbffff0ec, postfix=0xbfffe898, ctx=0xbfffeb9c) (../../source/as_compiler.cpp:6321)
#7 0x80e2a15 asCCompiler::CompileExpression(this=0xbffff0ec, expr=0x81b93fc, ctx=0xbfffeb9c) (../../source/as_compiler.cpp:6286)
#8 0x80e26f9 asCCompiler::CompileCondition(this=0xbffff0ec, expr=0x81b93ac, ctx=0xbfffeb9c) (../../source/as_compiler.cpp:6236)
#9 0x80e175c asCCompiler::CompileAssignment(this=0xbffff0ec, expr=0x81b935c, ctx=0xbfffecdc) (../../source/as_compiler.cpp:6035)
#10 0x80d5a9d asCCompiler::CompileExpressionStatement(this=0xbffff0ec, enode=0x81b930c, bc=0xbfffedf4) (../../source/as_compiler.cpp:3124)
#11 0x80d2b23 asCCompiler::CompileStatement(this=0xbffff0ec, statement=0x81b930c, hasReturn=0xbfffef76, bc=0xbfffedf4) (../../source/as_compiler.cpp:2367)
#12 0x80cbf94 asCCompiler::CompileStatementBlock(this=0xbffff0ec, block=0x81a0b84, ownVariableScope=false, hasReturn=0xbfffef76, bc=0xbfffef78) (../../source/as_compiler.cpp:907)
#13 0x80ca5fe asCCompiler::CompileFunction(this=0xbffff0ec, builder=0x81b7a1c, script=0x81b7adc, signature=0x0, func=0x81ad2fc, outFunc=0x81b7dd4) (../../source/as_compiler.cpp:508)
#14 0x80ae503 asCBuilder::CompileFunctions(this=0x81b7a1c) (../../source/as_builder.cpp:635)
#15 0x80ac5ff asCBuilder::Build(this=0x81b7a1c) (../../source/as_builder.cpp:194)
#16 0x80fe533 asCModule::Build(this=0x81b74cc) (../../source/as_module.cpp:205)
#17 0x805d7fe CScriptBuilder::Build(this=0xbffff558) (/home/marc/PROJETS/INFO/CPP/Botcraft/angelscript/addons/scriptbuilder/scriptbuilder.cpp:417)
#18 0x805c001 CScriptBuilder::BuildModule(this=0xbffff558) (/home/marc/PROJETS/INFO/CPP/Botcraft/angelscript/addons/scriptbuilder/scriptbuilder.cpp:85)
#19 0x80798db program(argc=1, argv=0xbffff8a4) (/home/marc/PROJETS/INFO/CPP/Botcraft/main.cpp:127)
#20 0x8079cd3 main(argc=1, argv=0xbffff8a4) (/home/marc/PROJETS/INFO/CPP/Botcraft/main.cpp:205)[/CODE] Edited by Zylann
0

Share this post


Link to post
Share on other sites
Thanks I'll have a look at this and see how to best handle the out-of-memory condition. I'll also go through the code and look for as many problems as possible.
0

Share this post


Link to post
Share on other sites
I have added my custom memory functions to scriptarray, seems to work fine.
But when I try to create a too large buffer, I get a SIGSEV too.

I realized that if we run out of sandbox's memory, AngelScript wouldn't have space to allocate for errors handling.
So I've added a condition in memFree : if memLimit is reached, memFree will return 0 only once, and will not check the limit after (because the program using the sandbox is assumed to have stopped).

I recompiled and retried my test :
When the big buffer is created, the context exception is set to "Out of memory", the script stops.
Then the console prints "An exception 'Out of memory' occurred. Please correct the code and try again."
But when the engine is released, I get SIGSEV anyway :

[CODE]#0 0xb7f56dca std::string::resize(unsigned int, char) () (/usr/lib/i386-linux-gnu/libstdc++.so.6:??)
#1 0xb7f56e9b std::string::resize(unsigned int) () (/usr/lib/i386-linux-gnu/libstdc++.so.6:??)
#2 0x806fc44 StringResize(l=135985784, str=...) (/home/marc/PROJETS/INFO/CPP/Botcraft/angelscript/addons/scriptstdstring/scriptstdstring.cpp:242)
#3 0x809a26d asCScriptEngine::CallObjectMethodRetInt(this=0x819f034, obj=0x81afa78, func=41) (../../source/as_scriptengine.cpp:3254)
#4 0x80fbce9 asCGarbageCollector::DestroyOldGarbage(this=0x819f784) (../../source/as_gc.cpp:436)
#5 0x80fb2a6 asCGarbageCollector::GarbageCollect(this=0x819f784, flags=1) (../../source/as_gc.cpp:137)
#6 0x809a8f8 asCScriptEngine::GarbageCollect(this=0x819f034, flags=1) (../../source/as_scriptengine.cpp:3472)
#7 0x808dd33 asCScriptEngine::~asCScriptEngine(this=0x819f034, __in_chrg=<optimized out>) (../../source/as_scriptengine.cpp:531)
#8 0x808ece3 asCScriptEngine::Release(this=0x819f034) (../../source/as_scriptengine.cpp:684)
#9 0x8079abb program(argc=1, argv=0xbffff8a4) (/home/marc/PROJETS/INFO/CPP/Botcraft/main.cpp:167)
#10 0x8079d47 main(argc=1, argv=0xbffff8a4) (/home/marc/PROJETS/INFO/CPP/Botcraft/main.cpp:205)[/CODE]

I hope this will help :)
0

Share this post


Link to post
Share on other sites
I've made several improvements to the code for handling out-of-memory situations. You can get those improvements from the SVN, specifically revisions 1318 and 1319.

Out of memory situations are extremely difficult to test though so I really can't guarantee that there won't be any more SIGSEV problems, or memory leaks due to lack of clean-up due to 'out-of-memory'. I'll have to fix these problems as they are detected.

Let me know if you find any further problems.



I do suggest you avoid setting a hard limit for the memory consumption. It is a very severe condition and as you already noted, it will make it difficult to get proper error messages, and doing a proper cleanup of the script that uses too much memory. Instead of using the hard limit and returning 0 on the allocation routines, I suggest you monitor how much memory is used, and if the memory usage exceeds a threshold, you abort the script execution (with asIScriptContext::Abort), and then do the normal cleanup as if shutting down the script.
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0