Sign in to follow this  
irreversible

Cleanly reloading a Lua script

Recommended Posts

I'm loading my scripts from disk to a memory buffer, parsing them using my own preprocessor and then passing the buffer off to worker threads to be compiled. What I want to achieve is script reloading at runtime, but I'm finding it difficult to work out what I need to do to make this clean and error-free.

 

As far as I understand:

 

1) as step one I need to (recursively) "unrequire" require'd modules in the script. This is not really an issue as I can get this info from my preprocessing step. In particular, this link seems to outline what accomplishes that.

1.1) sources (which I seem to have lost) say that state within modules will be maintained for dependencies loaded via require and in particular that unloading these in the wrong order will crash the program.  In particular, this thread suggests that doing so is unsafe no matter what. As a workaround, am I right in assuming that the safest thing would be to manually perform any dependency resolution, essentially processing all scripts into stand-alone modules, with zero Lua-bound dependencies?

2) next I need to invalidate the script itself to force it be released. Does Lua's GC kick in when a chunk with the same name is (re)loaded or do I need to take additional steps to ensure the previous object gets released? That is, LuaL_loadbuffer takes a name argument. Can I expect that to be matched by Lua when reloading a script with the same name?

3) how do C-side calls like luaopen_math correlate with require's inside modules? Eg do I need to unrequire or is there any harm in unrequiring modules that have been luaopene'd globally?

Edited by irreversible

Share this post


Link to post
Share on other sites

Whenever any lua file is updated, I simply un-require absolutely all my files and load them all again from scratch.

 

Lots of the recommended ways of writing Lua modules/classes don't seem to work very well with reloading....

I ended up just using a pattern of recycling existing objects when reloading code:

MyClass = MyClass or {} -- if reloading, use existing table, otherwise make new table. Also note not a local variable...
 
function MyClass:blah() -- note when reloading, this stomps the previous value of MyClass.blah with this new function
  print('foo')
end

Share this post


Link to post
Share on other sites

The thing I ended up spending a better part of my day on was trying to figure out how to unload modules that are loaded directly via lua_loadbuffer() (eg do not end up in the globals table as separate entities). For now I settled on auto-generating a hot-loader after my resource manager finishes loading that essentially spits out a proxy module, which sets up paths, require's all modules in the resource manager and also piles them into a single unload() call, which recusively also signals all require'd modules. This basically reduces the whole thing to unrequiring everything and creating the entire user-side LUA environment from scratch upon reload. As for recycling - I'm leaning towards purging the LUA state and reloading the modules that haven't changed from a binary proxy, which I'm storing anyway.

Share this post


Link to post
Share on other sites

This basically reduces the whole thing to unrequiring everything and creating the entire user-side LUA environment from scratch upon reload. As for recycling - I'm leaning towards purging the LUA state and reloading the modules that haven't changed from a binary proxy, which I'm storing anyway.

What about preserving game-state across the reload?

Share this post


Link to post
Share on other sites

 

This basically reduces the whole thing to unrequiring everything and creating the entire user-side LUA environment from scratch upon reload. As for recycling - I'm leaning towards purging the LUA state and reloading the modules that haven't changed from a binary proxy, which I'm storing anyway.

What about preserving game-state across the reload?

 

 

 

Well, presently I don't have any game state on the LUA server, so essentially scripts are just a bunch of atomic callbacks.

 

However, if maintaining state did become a necessity, I would probably store it in a separate module that can be excluded from the purge. I might be shortsighted, but I don't really see how partial state preservation would necessarily keep things simpler - you'd have to repopulate whatever state you're keeping in unrequired modules anyway and since script hotloading is mostly a debug/development feature, it doesn't seem like overkill to just send it all again. Also, if a module was reloaded that contained locally derived state, then that would be lost anyway when the module was unrequired. Or am I getting this wrong?

Share this post


Link to post
Share on other sites

Well, presently I don't have any game state on the LUA server

That makes things easier :)

Also, if a module was reloaded that contained locally derived state, then that would be lost anyway when the module was unrequired. Or am I getting this wrong?

Removing a module from the package.loaded table doesn't destroy any objects that have been created by that code.
If my C++ code is holding onto a handle to a Lua table, that table will continue to exist no matter what happens to the code that spawned it.

e.g. if my C++ code calls MyModule.CreateWidget, which returns a Lua table -- the C++ code holding onto that Lua table is unaware of Lua code reloading.

Also, as I recycle my "module" tables, and C++ links to those tables remain valid after a code reload, so I don't need to notify any of the C++ systems.
 
On that note, if your C++(etc) code has grabbed any handles to Lua structures whatsoever, these will all have to be re-acquired after your great purge.
 
Also, there's a billion ways to create "modules" in Lua, not everyone does that the same way; modules more of a coding convention / guideline than a language feature. Which module pattern are you using?

Edited by Hodgman

Share this post


Link to post
Share on other sites

Removing a module from the package.loaded table doesn't destroy any objects that have been created by that code.

If my C++ code is holding onto a handle to a Lua table, that table will continue to exist no matter what happens to the code that spawned it.

e.g. if my C++ code calls MyModule.CreateWidget, which returns a Lua table -- the C++ code holding onto that Lua table is unaware of Lua code reloading.

Also, as I recycle my "module" tables, and C++ links to those tables remain valid after a code reload, so I don't need to notify any of the C++ systems.

 

Interesting... :)

 

On that note, if your C++(etc) code has grabbed any handles to Lua structures whatsoever, these will all have to be re-acquired after your great purge.
 
Also, there's a billion ways to create "modules" in Lua, not everyone does that the same way; modules more of a coding convention / guideline than a language feature. Which module pattern are you using?

 

To answer your question first: I'm not really sure what to consider a module in this case (or even what you mean by a module pattern). I mean, I'm somewhat old school, so a module to me is just a file that contains code, which in LUA's case is little more than a fairly abstract logical namespace that may or may not have a name. My C++ side creates a script object for each file, which also contains information for stuff like writing to and loading from binary, file watching etc. Plus, I limit realoding files from disk to the first thread that tries to execute code in an externally modified file. Which strongly enforces the idea that modules are files.

 

Unless you meant something else with your question, that is.

 

In general I want to minimize the shenanigans that are or can be performed in LUA, so my ultimate goal is to contain all code management to my own preprocessor. Meaning that I don't want LUA to indirectly require any modules whatsoever, because a) required modules are loaded from disk, which is directly at odds with both a custom preprocessor step and maintaining some 3-10 odd LUA instances, even if the reads get cached and so forth and b) I like control :). The problem here is still that I'm not sure how to unload a "module"/file that's been passed to LUA directly using luaL_loadbuffer(). Having to resort to something silly like the following would be fairly annoying:

 

1) generate a hotloader that require's my preprocessed file and pass it off to LUA

2) (re)load all changed script files, parse them and recurseively merge require'd entities into a single code file

3) write this preprocessed file to disk

4) compile the hotloader

5) when a reload occurs, unrequire the code file/module and goto step 2

 

In terms of state management - if it's preserved, then your way of recycling it makes a lot of sense. That being said, I'd still keep data in a separate module, kind of the way the PE format forces segregation between the code and data segments. BTW - I'm assuming the ffi segment isn't affected by an unrequire?

 

As a sidenote - as far as syntax goes, I'm not really a fan of LUA's. I hate the comment style and I dislike the lack of braces and forced statement termination, which hinder clarity and bloat code with unnecessary line breaks. Luckily all this is something a simple preprocessing step can fix :).

Share this post


Link to post
Share on other sites

To answer your question first: I'm not really sure what to consider a module in this case (or even what you mean by a module pattern). I mean, I'm somewhat old school, so a module to me is just a file that contains code, which in LUA's case is little more than a fairly abstract logical namespace that may or may not have a name.

Ok, cool. People in the Lua community use that word to mean a bunch of different things. There was an attempt to standardize what a "lua module" was at one point, but it didn't work out :lol:
If you're interested, check out all these conflicting ideas on how you can create "modules" in lua:
http://lua-users.org/wiki/ModulesTutorial
http://lua-users.org/wiki/LuaModuleFunctionCritiqued
http://lua-users.org/wiki/ModuleDefinition
http://lua-users.org/wiki/AlternativeModuleDefinitions
http://hisham.hm/2014/01/02/how-to-write-lua-modules-in-a-post-module-world/
https://blog.separateconcerns.com/2014-01-03-lua-module-policy.html


The problem here is still that I'm not sure how to unload a "module"/file
Do you even have to ever unload it? Can  you just overwrite it by loading the new code on top of it?

If you overwrite global variables with new values, or if your C++ code requires handles to new Lua tables, then the old code will get garbage collected eventually.

Edited by Hodgman

Share this post


Link to post
Share on other sites

[...]LUA[...]LUA[...]LUA[...]


Since Hodgman seems to have the core issue well in hand allow me to deal with my own personal demon as well. To quote Wikipedia:

Lua (/?lu??/ loo-?, from Portuguese: lua [?lu.(w)?] meaning moon; explicitly not "LUA"[1] because it is not an acronym)

Emphasis by bolding mine. Thank you for your cooperation.

Share this post


Link to post
Share on other sites

 

To answer your question first: I'm not really sure what to consider a module in this case (or even what you mean by a module pattern). I mean, I'm somewhat old school, so a module to me is just a file that contains code, which in LUA's case is little more than a fairly abstract logical namespace that may or may not have a name.

Ok, cool. People in the Lua community use that word to mean a bunch of different things. There was an attempt to standardize what a "lua module" was at one point, but it didn't work out :lol:
If you're interested, check out all these conflicting ideas on how you can create "modules" in lua:
http://lua-users.org/wiki/ModulesTutorial
http://lua-users.org/wiki/LuaModuleFunctionCritiqued
http://lua-users.org/wiki/ModuleDefinition
http://lua-users.org/wiki/AlternativeModuleDefinitions
http://hisham.hm/2014/01/02/how-to-write-lua-modules-in-a-post-module-world/
https://blog.separateconcerns.com/2014-01-03-lua-module-policy.html

 

 

Hm - what is your approach, may I inquire?

 

I don't have any formal Looa (here you go, Bitmaster) background, so I'm pretty much making this up as I go along and the concept of modules seems a bit dim which ever way you look at it. Once you accept that compiled code is just placed in a single global namespace, which is divided into functions and typeless closures (which are really just scopes), it feels unintuituve to try and assign a label to just one of these abstractions. Since I'm bent on getting rid of implicitly managing require'd files anyway, the concept of modules becomes increasingly moot. I'm perfectly happy with having files that belong to a namespace and internally creating objects, which I'll just call tables or classes :).

 

 

Do you even have to ever unload it? Can  you just overwrite it by loading the new code on top of it?

If you overwrite global variables with new values, or if your C++ code requires handles to new Lua tables, then the old code will get garbage collected eventually.

 

I decided to double-check this before expressing how sure I was that I already checked it and, lo and behold - it does work. Turns out I was using one my own APIs wrong and the file I was reloading was taken from a cache  -_- .

 

 

 

[...]LUA[...]LUA[...]LUA[...]


Since Hodgman seems to have the core issue well in hand allow me to deal with my own personal demon as well. To quote Wikipedia:

Lua (/?lu??/ loo-?, from Portuguese: lua [?lu.(w)?] meaning moon; explicitly not "LUA"[1] because it is not an acronym)

Emphasis by bolding mine. Thank you for your cooperation.

 

 

Say it ain't so!  :o

Edited by irreversible

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this