Jump to content

  • Log In with Google      Sign In   
  • Create Account

Not dead...

Basic Lua tokenising is go...

Posted by , in Lua 16 January 2011 - - - - - - · 367 views

Over a few weekends leading up until Xmas and the last couple since then I have been playing around with boost::spirit and taking a quick look at ANTLR in order to setup some code to parse Lua and generate an AST.

Spirit looked promising, the ability to pretty much dump the Lua BNF into it was nice right up until I ran into Left Recursion and ended up in stack overflow land. I then went on a hunt for an existing example but that failed to compile, used all manner of boost::fusion magic and was generally a pain to work with.

I had a look at ANTLR last weekend and while dumping out a C++ parser using their GUI tool was easy enough the C++ docs are... lacking.. it seems and I couldn't make any headway when it came to using it.

This afternoon I decided to bite the bullet and just start doing it 'by hand'. Fortunately the Lua BNF isn't that complicated with a low number of 'keywords' to deal with and a syntax which shouldn't be too hard to build into a sane AST from a token stream.

I'm not doing things completely by hand; the token extraction is being handled by boost::tokeniser with a custom written skipper which dumps space and semi-colons, keeps the rest of the punctuation required by Lua and, importantly, it aware of floating point/double numbers so that it can correctly spit out a dot as a token when it makes sense.

Currently it doesn't deal with/hasn't been tested with octal or escaped characters and comments would probably cause things to explode, however I'll deal with them in the skipper at some point.

Given the following Lua;

foo = 42; bar = 43.4; rage = {} rage:fu() rage.omg = "wtf?"

The following token stream is pushed out;

<foo (28)> <= (18)> <42 (30)>
<bar (28)> <= (18)> <43.4 (29)>
<rage (28)> <= (18)> <{ (20)> <} (21)>
<rage (28)> <: (26)> <fu (28)> <( (24)> <) (25)>
<rage (28)> <. (27)> <omg (28)> <= (18)> <"wtf?" (31)>

Where the number is the token id found

There is a slight issue right now, such as when given this code;

foo = 42; bar <= 43.4; rage = {} rage:fu() rage.omg = "wtf?"

The token stream created is;

<foo (28)> <= (18)> <42 (30)>
<bar (28)> << (26)> <= (18)> <43.4 (29)>
<rage (28)> <= (18)> <{ (20)> <} (21)>
<rage (28)> <: (26)> <fu (28)> <( (24)> <) (25)>
<rage (28)> <. (27)> <omg (28)> <= (18)> <"wtf?" (31)>

Notice that it create two tokens for the '<=' sequence; this will probably need to be solved in the skipper as well.

So, once that is solved the next step will be the AST generation.. fun times...




From small ideas....

Posted by , in LLVM, Lua 12 January 2011 - - - - - - · 300 views
LLVM, Lua
Some time ago now I had an idea for a game which spawned from reading about a perticular graphics technique in GPU Gems 3.
I decided early on with the game idea that I wanted to target DX11 hardware and multi-core CPUs only; mostly because I was under no illusion about a fast release time and partly because I decided I wanted things to look nice thus a DX11 card was pretty much a must.

This then lead to my deciding to build the game, more or less from the ground up, by myself. This was never going to be an 'engine' thing, more a set of code required to make the game I could see in my head happen. After some playing around I decided that, in order to support scaling across cores, I would use Intel's TBB as my 'core' around which the game would be built. This is easy enough to do it's just a matter of getting into the 'task based' mindset and being mindful of your data flow in order to get the most from the cpu.

However, while large chunks of the game would be done in C++ (because as much as I like C# sometimes I just love to hack on C++) various pieces of logic could be 'soft coded' in script form with no problem. Well, apart from one; concurrency.

My scripting language of choice has been, for some time now, Lua; I like the lightness of the language, the speed of the language and the synxtax of the language. While I can work with Python and others something about Lua drags me back. The problem is Lua has some thread safety issues even when executing different contexts on different threads.

The first problem I came across was the function recursion count; basically Lua only lets you go so deep, but as the inc/dec on this wasn't protected.. well.. bang!

I "fixed" that by shifting the count out to the thread state from the global state, all seemed good... right up until the first GC cycle at which point... bang!

At this point I admitted defeat on the Lua front and took a look at some other languages however between them I couldn't find something I liked. GameMonkey script came close however no dice.

About that point I went mad and decided 'sod this, I'll do it myself...' and with that I've decided to build my own LLVM backend'd system using Lua-like syntax and features.

The aim is, once up and running, to extend it with a few game specific features (and probably a few stolen from other languages such as GMScript and Squirrel) and generally make it awesome.

No time scale is planned on this front, mostly because work is a bit mad right now but this is where I've ended up after a coming up with a game idea..





Recent Entries

Recent Comments