Jump to content

  • Log In with Google      Sign In   
  • Create Account

Not dead...

Basic Lua tokenising is go...

Posted by , in Lua 16 January 2011 - - - - - - · 387 views

Over a few weekends leading up until Xmas and the last couple since then I have been playing around with boost::spirit and taking a quick look at ANTLR in order to setup some code to parse Lua and generate an AST.

Spirit looked promising, the ability to pretty much dump the Lua BNF into it was nice right up until I ran into Left Recursion and ended up in stack overflow land. I then went on a hunt for an existing example but that failed to compile, used all manner of boost::fusion magic and was generally a pain to work with.

I had a look at ANTLR last weekend and while dumping out a C++ parser using their GUI tool was easy enough the C++ docs are... lacking.. it seems and I couldn't make any headway when it came to using it.

This afternoon I decided to bite the bullet and just start doing it 'by hand'. Fortunately the Lua BNF isn't that complicated with a low number of 'keywords' to deal with and a syntax which shouldn't be too hard to build into a sane AST from a token stream.

I'm not doing things completely by hand; the token extraction is being handled by boost::tokeniser with a custom written skipper which dumps space and semi-colons, keeps the rest of the punctuation required by Lua and, importantly, it aware of floating point/double numbers so that it can correctly spit out a dot as a token when it makes sense.

Currently it doesn't deal with/hasn't been tested with octal or escaped characters and comments would probably cause things to explode, however I'll deal with them in the skipper at some point.

Given the following Lua;

foo = 42; bar = 43.4; rage = {} rage:fu() rage.omg = "wtf?"

The following token stream is pushed out;

<foo (28)> <= (18)> <42 (30)>
<bar (28)> <= (18)> <43.4 (29)>
<rage (28)> <= (18)> <{ (20)> <} (21)>
<rage (28)> <: (26)> <fu (28)> <( (24)> <) (25)>
<rage (28)> <. (27)> <omg (28)> <= (18)> <"wtf?" (31)>

Where the number is the token id found

There is a slight issue right now, such as when given this code;

foo = 42; bar <= 43.4; rage = {} rage:fu() rage.omg = "wtf?"

The token stream created is;

<foo (28)> <= (18)> <42 (30)>
<bar (28)> << (26)> <= (18)> <43.4 (29)>
<rage (28)> <= (18)> <{ (20)> <} (21)>
<rage (28)> <: (26)> <fu (28)> <( (24)> <) (25)>
<rage (28)> <. (27)> <omg (28)> <= (18)> <"wtf?" (31)>

Notice that it create two tokens for the '<=' sequence; this will probably need to be solved in the skipper as well.

So, once that is solved the next step will be the AST generation.. fun times...

On HD5870 and Memory

Posted by , in OpenCL, DX11, AMD 15 January 2011 - - - - - - · 339 views
DX11, OpenCL, AMD, HD5870
While gearing up to work on parser/AST generator as mentioned in my previous entry I decided to watch a couple of Webinars from AMD talking about OpenCL (because while I'm DX11 focused I do like OpenCL as a concept); the first of which was talking about the HD5870 design.

One of the more intresting things to come out of it was some details on the 'global data store' (GDS), wihch while only given an overview had an intresting nugget of information in it which would have been easy to skip over.

While not directly exposed in DXCompute or OpenCL the GDS does come into play with DXCompute's 'appendbuffers' (and whatever the OpenCL version of this same construct is) as that is where the data is written to thus allowing the GPU to accelerate the process.

What this means in real terms is that if you compute shader needs to store memory which everyone in dispatch needs to get to for some reason then you could use these append buffers with only a small hit (25 cycles on the HD5 series) as long as the data will fit into the memory block. Granted, you would still need to place barriers into your shader/OpenCL code to ensure that everyone is done writing but it might allow for faster data sharing in some situations.

I don't know if NV does anything simular however, maybe I'll check that out later..

Right, back to the Webinars...

From small ideas....

Posted by , in LLVM, Lua 12 January 2011 - - - - - - · 313 views
Some time ago now I had an idea for a game which spawned from reading about a perticular graphics technique in GPU Gems 3.
I decided early on with the game idea that I wanted to target DX11 hardware and multi-core CPUs only; mostly because I was under no illusion about a fast release time and partly because I decided I wanted things to look nice thus a DX11 card was pretty much a must.

This then lead to my deciding to build the game, more or less from the ground up, by myself. This was never going to be an 'engine' thing, more a set of code required to make the game I could see in my head happen. After some playing around I decided that, in order to support scaling across cores, I would use Intel's TBB as my 'core' around which the game would be built. This is easy enough to do it's just a matter of getting into the 'task based' mindset and being mindful of your data flow in order to get the most from the cpu.

However, while large chunks of the game would be done in C++ (because as much as I like C# sometimes I just love to hack on C++) various pieces of logic could be 'soft coded' in script form with no problem. Well, apart from one; concurrency.

My scripting language of choice has been, for some time now, Lua; I like the lightness of the language, the speed of the language and the synxtax of the language. While I can work with Python and others something about Lua drags me back. The problem is Lua has some thread safety issues even when executing different contexts on different threads.

The first problem I came across was the function recursion count; basically Lua only lets you go so deep, but as the inc/dec on this wasn't protected.. well.. bang!

I "fixed" that by shifting the count out to the thread state from the global state, all seemed good... right up until the first GC cycle at which point... bang!

At this point I admitted defeat on the Lua front and took a look at some other languages however between them I couldn't find something I liked. GameMonkey script came close however no dice.

About that point I went mad and decided 'sod this, I'll do it myself...' and with that I've decided to build my own LLVM backend'd system using Lua-like syntax and features.

The aim is, once up and running, to extend it with a few game specific features (and probably a few stolen from other languages such as GMScript and Squirrel) and generally make it awesome.

No time scale is planned on this front, mostly because work is a bit mad right now but this is where I've ended up after a coming up with a game idea..

Really, not dead...

Posted by , 12 October 2010 - - - - - - · 233 views

So, I'm really not dead... which at least doesn't make this journal title a lie [grin]

It's been quite a busy few months for me, getting used to work, getting used to living in more than one room and getting used to being 100% single again (which I shant bang on about).

Work wise things are going well, nicely entrenched in the rendering team at Codies and enjoying it even with the hard work. In fact compared to the last place I was at I've been more likely to stay late here which shows just how much I like it.

I've also managed to be the most unlucky guy on the team as my first commit of work took 3 weeks to get from my branch to main due to it constantly failing out on the publish system which tests the builds. The real kicker was it never failed in anything todo with my changes which were mostly shader and data related!

After alot of trying, searching and being unable to reproduce it on my branch the problem was finally tracked down to an assert handler which wasn't thread safe and my changes, which added extra data, were just enough to throw out the thread timing to cause it to hit the problem.

Talk about unlucky [sad]
Still, got it fixed, got it all published and got my work in [grin]

We are now in the final push to alpha, which was high lighted by my 54h week last week (normal week is just shy of 38h) which works out at 2 days over time. This week is looking to be much easier for me as I (currently) have no alpha critial work so I'm going to dial back the over time I think if only because its left me too burnt out to work on my own stuff at home.

I do have plans to do my own stuff at home, I want to do a space type shooter right now (although I have an RTS idea knocking about in my head as well), which will be purely DX11 and has come about thanks to listening to "Club Foot" by Kasabian too many times and an article in GPU Gems 3 about fluids on the GPU and using them for particle simulation.

I've so far only done some reading and found a free test model, I'm hoping this weekend I won't be too burnt out to do some work on it; if nothing else I need to kick a deferred renderer into life.

But yeah, not dead and still dreaming...

Codemasters : Week 1

Posted by , 07 August 2010 - - - - - - · 238 views

So, the move last weekend went off without a hitch; well aside from the window getting a crack in it about 30mins after leaving Brighton but that at least gave me something to watch as it made slow progress down the screen while the satnav seemed todo its best to make us avoid the M25.

Last weekend was mostly spent watching TV, watching films and watch Babylon5 DVDs... in fact, the B5 DVDs would be a continuing theme during the week due to lack of internets and lack of desk to setup my PC properly [grin]

On the monday I dragged myself out of bed at 8am, got on a bus some 40mins later and arrived at Codemasters pretty close to 9am for my first day.

The openning section of said day was taken up with the normal admin type tasks before the 5 of us got taken to our respective studios/teams.

As previously mentioned I was joining the Action Studio, however at that point I had no idea what I would be doing, although given my curent background I assumed it would be some form of general coding.

So, I was slightly surprised and delighted when I discovered that I'd be working in the rendering team for the next Operation Flashpoint game [grin]

The rest of the week (apart from Thursday where I sat at home and waited for a desk) was spent installing things, getting code from Perforce and then working on a project to learn how their rendering system works.

By the end of the week I'd got something 'new' rendering on screen and understood how I'd gotten there so I was happy with the progress so far [smile]

The team themselves are a good bunch of people; helpful, friendly and a laugh which is alway good, so I think I'll enjoy my time working with them.

The week was finished off by me joining various people down the pub for a few(!) drinks before stagging to a bus stop and home.

All in all a good week and a good move [grin]

I'll just finish with a belated Thank You to everyone who wished me luck/well during the last few entries; much appricated guys [smile]

Recent Entries

Recent Comments