• entries
743
1924
• views
582218

Vague rambling with the occasional tangible result

## Further developments

Been working hard on Om and added a few new features.

Figured out a way to have implicit [font='courier new']this[/font] in functions, as long as they are defined inside the object of which they are methods. Name lookup rules similar to C++, so a parameter or a local variable is considered first, then an object member, then variables in the outer scope.

var x = "foo";var o ={ f = function { out x; // outputs "bar" if(true) { var x = "fum"; out x; // outputs "fum"; } }; g = function(x) { out x; // outputs whatever the parameter is }; x = "bar";};out x; // outputs "foo"This was easier than I expected to implement. I added a [font='courier new']containers[/font] stack to the compiler [font='courier new']Context[/font] object which is passed around to all the compilation methods, and when compiling [font='courier new']ObjectNodes[/font] and [font='courier new']FunctionNodes[/font], it just pushes the node onto this [font='courier new']containers[/font] stack, then pops it off at the end.

When the [font='courier new']SymbolNode[/font] does its lookup, after it has checked the local symbol table, it just checks to see if the node second-to-top on the container stack is an [font='courier new']ObjectNode[/font] and, if so, uses the [font='courier new']NodeQuery[/font] system to ask it if it has a member with the same name as the symbol.

If so, it just emits the code to push the current [font='courier new']this[/font] pointer onto the stack, and calls the [font='courier new']OpCode::Type::GetMb[/font] or [font='courier new']OpCode::Type::SetMb[/font], essentially emitting exactly the same code that an explicit [font='courier new']ThisNode[/font] would.

I realised shortly afterwards that I then needed some syntax to explicitly refer to the outer scope in an object method, so came up with prefixing a symbol with dots, the number of dots representing how many scopes to jump up to find the symbol. Probably easier explained with an example:

var x = function(a, b){ return a + b; };var o ={ f = function { return "flibble"; };};var f = function{ var x = "bar"; var o = { f = function { out x; // outputs "paul" via property below out .x; // outputs "bar" out ..x(1, 3); // outputs "4" }; n = "paul"; x = property { get { return n; }; set(a) { n = a; }; }; }; o.f(); out .o.f(); // outputs "flibble"};f();This is all compile-time only stuff. I just added a "dots" member to the [font='courier new']SymbolNode[/font] and the compiler just handles a primary expression starting with a dot like:

NodePtr dot(Context &c, bool get){ uint dots = 1; c.scanner.next(get); while(c.scanner.type() == Scanner::TokenType::Dot) { ++dots; c.scanner.next(true); } c.scanner.match(Scanner::TokenType::Id, false); NodePtr n(new SymbolNode(c, c.scanner.token(), dots)); return c.next(n, true);}When [font='courier new']SymbolNode[/font] is generated now, if it has a non-zero dots value, it ignores the local symbol table and the object member lookup and passes its dots value to the method that searches in enclosing scopes, using this to start the searching at the correct level.

All seems to work well and was simple anough to implement.

Have also now got subscript string access working on object members, including the new properties.

var o ={ name = "foo"; func = function { return "bar"; }; prop = property { get { return name; }; set(a) { name = a; }; };};out o["name"]; // outputs "foo"var s = "fun";out o[s + "c"](); // outputs "bar";o["prop"] = "flibble"; // calls the property setterObviously this is a bit less efficient but allows for dynamic name lookup as in the second example above.

Actually, the first and third examples are optimised back to a dot access, since the strings are constant. This was pretty trivial now all the optimisation infrastructure is in place:

void SubscriptNode::optimise(Context &c, Node *parent){ target->optimise(c, this); expr->optimise(c, this); Variant v = expr->query(NodeQuery::AsVariant); if(v.type() == Om::Type::String) { parent->replace(this, new DotNode(c, target, v.toString())); }}Essentially, the node asks if its expression inside the subscripts is a constant of string type and, if so, asks its parent to replace it with a [font='courier new']DotNode[/font]. The generated code is then exactly the same as if you used a dot expression.

out o["fish"];// becomesout o.fish;Because the optimisation system deals with addition of string constants at compile time, even this is equivalent:

out o["f" + "i" + "s" + "h"];// becomesout o.fish;Which is rather cool, but probably not that useful in the real world. Effortless to implement though so may as well do it.

The subscript stuff also takes account of the prototype system of course as indeed do properties.

var b ={ p = property { get { return "foo"; }; };};var o ={ prototype = b;};out o["p"]; // outputs "foo" via b's propertyWhich brings us to an interesting point about properties in a prototype-based-inheritance environment that occured to me the other day.

For those not familiar with languages like Javascript, the prototype system is a nice, simple way to have object inheritance in a dynamic, duck-typed language. The object's [font='courier new']prototype[/font] member points to another object, and when you read a property, the system searches first in the object itself, then up through the prototype chain, looking for the property in each "parent" object.

However, when you write a property, if the object does not have its own copy of the property, a new one is created (the same as if no prototypes are involved) and from then on, the object has its own copy.

Example (in Om, not Javascript :)):
var b ={ name = "base";};var o ={ prototype = b;};out o.name; // outputs "base"o.name = "foo"; // "name" added to o as a new propertyout o.name; // outputs "foo"Now, this is all fine and dandy, but consider the following:

var b ={ name = property { get { return n; }; set(a) { if(a == "") { out "error: name must not be blank"; return; } n = a; }; }; n = "foo";};So we have a base class that uses the properties to "enforce" that the name is not empty. Now:

var o ={ prototype = base;};out o.name; // outputs "foo"o.name = ""; // adds new "name" to oout o.name; // outputs ""Should this be the case? It is an interesting question. I can see arguments both way.

I've just been experimenting with QScript (ECMAScript compliant scripting) and it seems in their implementation, the prototype's setter is called when you assign to the "derived" object instance. Presumably to replace it in a child, you have to use [font='courier new']Object.defineProperty()[/font] or delete the member first and assign the new one.

I decided to go the same way. So when writing, we have to look up the prototype chain and check to see if a base object has a property before we can just add the new value to the child object.

var b ={ n = "foo"; name = property { get { return n; }; set(x) { out "setter ", n, " == ", x; n = x; }; };};var x ={ prototype = b;};var o ={ prototype = x;};out o.name; // calls b name gettero.name = "bar"; // calls b name setterout o.name; // calls b name gettervar s = "name"; // to supress the optimisationout o[s]; // calls b name gettero[s] = "bar"; // calls b name setterout o[s]; // calls b name getterCan't imagine the runtime cost is relevant. The implementation now looks like this:

TypedValue &Imp::findProperty(Machine &machine, uint object, uint id){ auto &e = machine.state.entity(object); if(e.properties.find(id) == e.properties.end()) { uint p = nextObject(e.properties); while(p != invalid_uint) { auto &m = machine.state.entity(p).properties; auto i = m.find(id); if(i != m.end() && i->value.type() == Om::Type::Property) { return m[id]; } p = nextObject(m); } machine.state.tc.inc(id); e.trefs.push_back(id); } return e.properties[id];}So one extra lookup for the majority of cases where the object has no prototype, and a quick run up the chain for those that do.

All working from the C++ side to, via [font='courier new']Om::Value::property()[/font] and [font='courier new']Om::Value::setProperty()[/font]. Sharing a bit of new code between these methods and the virtual machine has actually simplified the code a bit now too.

What this does imply is we now need a way, both in the C++ API and the script, to actually explicitly overwrite a property and add it to the local object.

So the C++ API [font='courier new']Om::Value::setProperty()[/font] now takes an [font='courier new']Om::Value::UpdateType[/font] that defaults to [font='courier new']Om::Value::UpdateType::Normal[/font], but you can pass [font='courier new']Om::Value::UpdateType::Overwrite[/font] to explicitly ignore properties either in the local class or in a prototype parent.

In the script, I decided on adding a new assignment operator: .= (pronounced dot-assign)

var b = { x = property { get { return "getter; "}; set(x) { out "setter"; }; };};var o ={ prototype = b;};o.x = "foo"; // calls b's settero.x .= "bar"; // adds a new x member to oThe new operator works in all the relevant contexts i.e. with the subscript access, and when referencing a member in an object method using the new implicit this e.g.

var o = { n = property { get { return "getter"; }; set(x) { out "setter"; }; }; f = function { n = "foo"; // calls the setter n .= "foo"; // replaces the n property with a string member };};Final little point. I've just implemented for loops in the new version, by copying from the old version and modifying the code to the new system and its a great example of why using exceptions to handle compiler and run-time errors has massively simplified this code base.

This was the function that compiled a for loop in the old version:

bool forStatement(Context &c, BlockNode *block, bool get){ TRACE; bool any = true; if(!c.sc.match(c, Scanner::TokenType::LeftParen, get)) return false; NodePtr init = optional(c, conditionalExpr, { Scanner::TokenType::SemiColon }, any, true); if(any && !init) return false; if(!c.sc.match(c, Scanner::TokenType::SemiColon, false)) return false; ForNode *f = new ForNode(c); block->nodes.push_back(f); if(init) f->init = init.release(); NodePtr cond = optional(c, expr, { Scanner::TokenType::SemiColon }, any, true); if(any && !cond) return false; if(!c.sc.match(c, Scanner::TokenType::SemiColon, false)) return false; if(cond) f->cond = cond.release(); NodePtr post = optional(c, expr, { Scanner::TokenType::RightParen }, any, true); if(any && !post) return false; if(!c.sc.match(c, Scanner::TokenType::RightParen, false)) return false; if(post) f->post = post.release(); NodePtr b = statementBlock(c, true); if(!b) return false; f->block = b.release(); return true;}In the new version, this becomes:

void forStatement(Context &c, BlockNode *block, bool get){ c.scanner.match(Scanner::TokenType::LeftParen, get); NodePtr init = optional(c, conditionalExpr, { Scanner::TokenType::SemiColon }, true); ForNode *f = new ForNode(c); block->nodes.push_back(f); f->init = init; f->cond = optional(c, expr, { Scanner::TokenType::SemiColon }, true); f->post = optional(c, expr, { Scanner::TokenType::RightParen }, true); f->block = statementBlock(c, true);}I think that is enough for one entry anyway. Thanks for stopping by.

## Implementing Properties

I seem to have become obsessed with Om, my scripting language, again and have done nothing whatsoever on my game since last entry, just been busy with Om. I justify this on the basis that a) I'll need a scripting language for my game and b) this is hobby stuff so I can do whatever the hell I like :)

I've been fairly disciplined about implememnting the boring stuff. Most of it is easy enough as I can copy the code from the previous version of Om, simplifiying it a bit since the new version is a lot cleaner, and I have the fun of working on the optimisation now in the new version as I add each feature, since optimisation wasn't included in the last one.

But I reached a good point to implement properties in this incarnation, which I've never done in a scripting language before. Has been enormous fun and not nearly as difficult as I thought.

By properties, I mean object members that call a function when read from or written to as values. This is complicated slightly by the fact that functions in Om can either be script functions or C++ functions wrapped up in an Om::Value.

I went through a number of different approaches before I settled on what I have now. I've added [font='courier new']Om::Type::Property[/font] as a new type to the system, which is another reference type that is implemented by a [font='courier new']PropertyEntity[/font]. The [font='courier new']PropertyEntity[/font] stores [font='courier new']TypedValues[/font] for the get and set methods, which can be either an [font='courier new']Om::Type::Function[/font], and external C++ function or [font='courier new']Om::Type::Null[/font] (making the property read or write only, or both).

First up, let's look at the script syntax for defining properties. There are two versions. The first simply allows you to assign values to the getter and setter, so you can use anything to initialise them:

var f = function(a){ this.n = a;};var o ={ n = "Foo"; name = property { get = function { return this.n; }; set = f; }; };out o.name; // prints "Foo";o.name = "Bar"; // calls f("Bar")out o.name; // prings "Bar";You can assign anything to the getter and setter, but you will get errors if you then reference the getter or setter in an expression if they are not set to the correct type i.e. a function taking no arguments for the getter and a function taking one argument for the setter.

As a bit of syntax sugar, you can also define the methods inline:

var o ={ n = "Foo"; name = property { get { return this.n; }; set(a) { this.n = a; } }};out o.name; // prints "Foo";o.name = "Bar"; // calls f("Bar")out o.name; // prings "Bar";Essentially the same as above, just expressed a bit more cleanly, and we can enforce the correct function types here at compile-time.

From the C++ side, [font='courier new']Om::Engine[/font] has a new [font='courier new']makeProperty()[/font] method that you can use to implement properties using external functions. You can use the same function for getting and setting if you wish, by checking the incoming argument count.

script.txt:
return function(x){ var o = { n = "Foo"; name = x; }; out o.name; // prints "Foo"; o.name = "Bar"; // calls f("Bar") out o.name; // prings "Bar";};C++:
Om::Value test(Om::Engine &engine, Om::Context &ctx){ if(ctx.argumentCount()) { ctx.thisObject().setProperty("n", ctx.argument(0)); } return thisObject().property("n");}void f(Om::Engine &engine){ Om::Value f = engine.evaluate("script.txt", Om::Engine::EvaluateType::File); Om::Value p = engine.makeProperty(engine.makeFunction(test)); engine.call(f, Om::Value(), { p });}Similarly you can construct an [font='courier new']Om::Type::Object[/font] externally by calling [font='courier new']Om::Engine::makeObject()[/font] then using [font='courier new']Om::Value::setProperty()[/font] to set a property up as an [font='courier new']Om::Type::Property[/font].

So now we'll take a look at compilation and execution of this stuff. We'll define a slightly simpler script to make this a bit less verbose.

var o ={ n = property { get { out "getter"; }; set(a) { out "setter ", a; } };};out o.n;o.n = 10;First this is compiled by the system into the following abstract syntax tree:

function() block var o object member n property function() block trace string "getter" trace eol return function(a) block trace string "setter " trace symbol a trace eol return trace dot n symbol o trace eol expr assign dot n symbol o int 10 returnThe optimisation pass is run next, but there is nothing in this particular script that gets optimised.

Next, the bytecode for the virtual machine is generated from the tree. Here is the standard dump that we use constantly to check how things are looking internally. Each function is generated as a separate [font='courier new']FunctionEntity[/font] with its own program memory.

I'll annotate each line of the bytecode with some explanation. Comments are added here, not part of the dump.

Entity 0 flags[0] refs[0] vrefs[0] (function) params: 0 // function takes no parameters 0: MkEnt object // create an ObjectEntity, push a reference to it onto the stack, increment its reference count 6: MkEnt property // create a PropertyEntity, push a reference to it onto the stack, increment its reference count 12: Push function 1 // push a reference to the function with ID 1 (the getter) onto the stack, increment its reference count 18: AddPm 0 // pop the value from the top of the stack and set it as the stack-top PropertyEntity's get value 23: Push function 2 // push a reference to the function with ID 1 (the getter) onto the stack, increment its reference count 29: AddPm 1 // pop the value from the top of the stack and set it as the stack-top PropertyEntity's set value 34: AddCh 3 // pop the value from the top of the stack (the reference to PropertyEntity) and set it as a member of the top-of-stack (ObjectEntity) 39: GetLc 2 // grab a value from further down the stack and push it on top (ObjectEntity) 44: GetMb 3 // treat top of stack as ObjectEntity and get its member that has the name that resolves to 3 in the TextCache ("n") 49: Out // pop the top of stack, print it and decrement its reference count, destroying it if zero 50: OutNl // print a newline 51: Push int 10 // push an integer with the value of 10 on the stack 57: GetLc 2 // grab a value from further down the stack and push it on top (ObjectEntity) 62: PutMb 3 // treat top of stack as ObjectEntity and set its member that has the name that resolves to 3 in the TextCache ("n") 67: Pop // pop the top of the stack and decrement its reference count, destroying it if zero 68: Push null null // push a null onto the stack 74: PutLc 0 // put the value further down the stack (the return value slot in this case) 79: PopN 3 // pop three items from the stack, decrement each reference count (if applicable) and destroy if zero for each 84: Ret // returnEntity 1 flags[0] refs[1] vrefs[0] (function) params: 0 0: MkEnt string "getter" 6: Out 7: OutNl 8: Push null null 14: PutLc 0 19: PopN 2 24: RetEntity 2 flags[0] refs[1] vrefs[0] (function) params: 1 0: MkEnt string "setter " 6: Out 7: GetLc 2 12: Out 13: OutNl 14: Push null null 20: PutLc 0 25: PopN 3 30: RetTextCache:0: "prototype" (refs:1) [DS]1: "getter" (refs:1)2: "setter " (refs:1)3: "n" (refs:3)All the magic for properties is essentially handled inside the [font='courier new']OpCode::Type::GetMb[/font] and [font='courier new']OpCode::Type::SetMb[/font] instructions. These call [font='courier new']Machine::mb()[/font], passing the [font='courier new']TextCache[/font] id of the member (resolved at compile-time) and a flag to indicate if we are reading ([font='courier new']GetMb[/font]) or writing ([font='courier new']PutMb[/font]).

Here's [font='courier new']mb()[/font] in all its glory, although I appreciate it probably doesn't mean a lot to the casual reader.

void Machine::mb(uint id, AccessType type){ TypedValue o = vs.pop_back(); TypedValueGuard guard(state, { o }); if(o.type() == Om::Type::Object) { if(type == AccessType::Read) { uint object = o.toUint(); while(object != invalid_uint) { auto &m = state.entity(object).properties; auto i = m.find(id); if(i != m.end()) { vs.push_back(i->value); inc(state, vs.back()); if(vs.back().type() == Om::Type::Property) { TypedValue p = vs.pop_back(); TypedValue get = state.entity(p.toUint()).get; if(!get.valid()) { throw Exception("cannot read property", 0); } vs.push_back(TypedValue()); setupStack(state, vs, o, { }); vs.push_back(get); inc(state, vs.back()); call(0); dec(state, p); } return; } object = invalid_uint; i = m.find(DefinedStrings::Prototype); if(i != m.end() && i->value.type() == Om::Type::Object) { object = i->value.toUint(); } } vs.push_back(TypedValue()); return; } else { auto &e = state.entity(o.toUint()); if(e.properties.find(id) == e.properties.end()) { state.tc.inc(id); e.trefs.push_back(id); } TypedValue &m = e.properties[id]; if(m.type() == Om::Type::Property) { TypedValue set = state.entity(m.toUint()).set; if(!set.valid()) { throw Exception("cannot write to property", 0); } execute(set, o, { vs.back() }); } else { dec(state, m); m = vs.back(); inc(state, m); } } return; } throw Exception("dot applied to invalid type", 0);}Of relevance to the new property system are the checks for [font='courier new']type() == Om::Type::Property[/font] in the two blocks there.

If we are reading and it turns out we have read a property, we check it has a get method and, if so, set up the stack then call the [font='courier new']Call()[/font] method, passing in a zero parameter count. This saves us having to make a recursive call here and uses the same method that [font='courier new']OpCode::Type::Call[/font] does when calling a normal function.

Unfortunately, we can't do this when writing because we have to do the write "immediately", so when writing, we have to make a recusive call to [font='courier new']Machine::execute()[/font]. This is the entry point to the virtual machine so we are already running inside a call to this method, but everything is stack-based so a recursive call is fine.

We pass a reference to the object as the "this" value, and the value on the top of the stack as the parameter, which will be the value we assigned to the property in the script.

[font='courier new']Machine::execute()[/font]'s first parameter is a [font='courier new']TypedValue[/font] that can either point to an [font='courier new']Om::FunctionEntity[/font] or to a [font='courier new']TypedValue[/font] with a type of [font='courier new']ExternalFunctionType[/font], which is a special type not exposed by the API but just used internally. In this case, the data of the [font='courier new']TypedValue[/font] is a pointer to an [font='courier new']Om::Function[/font], which is defined as:

namespace Om{typedef Value(*Function)(Engine&, Context&);}This is what the [font='courier new']test()[/font] method in the C++ back up a ways implements.

[font='courier new']Om::Context[/font] is new to this version of Om as well. Previously I was passing an [font='courier new']Om::ValueList[/font] as the parameters to an external function, but the nice thing about [font='courier new']Om::Context[/font] is it internally stores a reference to the virtual machine's stack, and only converts the [font='courier new']TypedValue[/font] to an [font='courier new']Om::Value[/font] when you call the [font='courier new']Om::Context::argument(int index)[/font] method, so it is a bit more efficient.

So next up I need to figure out how to make [font='courier new']Om::Value::property()[/font] and [font='courier new']Om::Value::setProperty()[/font] call the property methods if the value is already initialised with an [font='courier new']Om::Type::Property[/font]. The two uses of the term "property" are a bit confused here. Need to think all this through carefully before I do anything.

Anyway, that's that. Thanks as always for stopping by.

## Optimising my scripting language

Decided I needed a break from graphics programming so decided to revisit Om, my scripting language. For those who didn't read the entries about this a few months ago, I came heavily unstuck in my last iteration of this project, since I had decided to implement deterministic destruction in a dynamically typed, JavaScript-like language and it didn't occur to me until it was pretty much all written that the problem of cyclic references basically makes this impossible.

E.g. if you have an object a that has a reference to object b, then you assign a reference in object b to a:

var a ={};var b = { ref = a;};a.ref = b;Then no amount of reference counting and clever trickery in the world is going to end up freeing those objects.

So, I have restarted Om from scratch and made a few decisions at the very start this time that, so far, are leading to a far more stable-feeling codebase.

• The language will use a stop-the-world mark-and-sweep garbage collector (for now)
• Objects will not offer user-defined destructors - we'll find other solutions to RAII if we need it
• This one might not go down well, but I decided to use exceptions to report compiler and runtime errors
Now I appreciate that last one might have people closing my journal in disgust, since compiler errors and runtime errors are certainly not "exceptional", but I have to say that, so far, the entire codebase feels so much more stable and easy to manage than it did before, that I'm willing to live with the hate :).

I have the garbage collection working now in a very simple way. I did some profiling and it can collect 100,000 entities in less than a tenth of a second, so I'm confident a stop-the-world approach will be fast enough. Haven't tested it yet with really deep trees of object-references though so we'll see.

I've also taken the time, while we are in the early stages, to tackle optimisation so that it is already in place and I can improve it as we go along. I'll go into a bit of detail about how the optimisation system works today, since I've rambled on about the language itself in previous entries and this is all new.

So I guess the minimum you need to know is that the compilation side of this project is based on building an abstract syntax tree based on a [font='courier new']Node[/font] base class:

enum class NodeQuery : int;class Node{public: explicit Node(Context &c); virtual ~Node(); void generate(Context &c); void generateAssign(Context &c); virtual void generateImp(Context &c) = 0; virtual void generateAssignImp(Context &c); virtual void optimise(Context &c, Node *parent) = 0; virtual void replace(Node *before, Node *after) = 0; virtual void remove(Node *node); virtual Variant query(NodeQuery query) const; virtual void output(std::ostream &os, int i) const = 0;};typedef scoped_ptr NodePtr;typedef ptr_vector NodeList;The new methods relevant to optimisation are [font='courier new']optimise()[/font], [font='courier new']replace()[/font], [font='courier new']remove()[/font] and [font='courier new']query()[/font].

[font='courier new']optimise()[/font] is a visitor method that each subclass calls recursively on any nodes it owns, then does any optimisation that it can do on itself.

Often this results in the node needing to replace itself with a new node - for example when a [font='courier new']BinaryNode[/font] detects that both of its child nodes are constants, it can do its operation on the children at compile time then replace itself with a new [font='courier new']LiteralNode[/font] representing the result.

So [font='courier new']optimise()[/font] passes a reference to the parent node, and [font='courier new']replace()[/font] and [font='courier new']remove()[/font] are implemented in each node so that a child node can request itself be replaced with a new node it supplies.

For example:

namespace{Variant compute(State &state, OpCode::Type type, const Variant &a, const Variant &b){ switch(type) { case OpCode::Type::Add: return mathOp(state, a, b); case OpCode::Type::Sub: return mathOp(state, a, b); case OpCode::Type::Mul: return mathOp(state, a, b); case OpCode::Type::Div: return mathOp(state, a, b); default: return Variant(); }}}void BinaryNode::optimise(Context &c, Node *parent){ left->optimise(c, this); right->optimise(c, this); Variant a = left->query(NodeQuery::AsVariant); Variant b = right->query(NodeQuery::AsVariant); if(a.valid() && b.valid()) { parent->replace(this, LiteralNode::fromVariant(c, compute(c.state, type, a, b))); }}void BinaryNode::replace(Node *before, Node *after){ if(left == before) left = after; if(right == before) right = after;}Here we see the use of the [font='courier new']query()[/font] method. [font='courier new']query()[/font] returns a [font='courier new']Variant[/font] which is a simple internal type capable of storing an [font='courier new']int[/font], [font='courier new']float[/font], [font='courier new']bool[/font] and [font='courier new']std::string[/font]. It also has an invalid state. Nodes implement [font='courier new']query()[/font] and respond in different ways depending on the query type.

The only nodes that respond to [font='courier new']NodeQuery::AsVariant[/font] are the literal nodes. For example:

Variant IntNode::query(NodeQuery query) const{ return query == NodeQuery::AsVariant ? Variant(value) : Variant();}Variant StringNode::query(NodeQuery query) const{ return query == NodeQuery::AsVariant ? Variant(value) : Variant();}So [font='courier new']BinaryNode[/font] is effectively asking "Are both my child nodes literals?" If so, it uses [font='courier new']compute()[/font] to calculate the expression at compile-time, returned as a [font='courier new']Variant[/font] again, then [font='courier new']LiteralNode::fromVariant()[/font] to create a new literal node representing the result, then finally calls its parent's [font='courier new']replace()[/font] method to replace itself with the new node.

Obviously we have to be cautious using replace and make sure we don't touch any of the members of the class in question after this call, since it will delete the node.

Another example is the [font='courier new']WhileNode[/font]. This can use [font='courier new']NodeQuery::AsVariant[/font] to check if its expression is constant, and if so, either remove itself entirely if the expression is false or remember to not bother generating its expression if the expression is constant and true:

void WhileNode::generateImp(Context &c){ uint top = c.pm().position(); ByteStreamPatch end; c.pushScope(); c.pushBreaks(c.locals().total()); if(!exprVar.valid()) { expr->generate(c); c.pm() << OpCode::Type::JmpF << end; } uint exprTotal = c.locals().scopeTotal(); block->generate(c); c.pop(c.locals().scopeTotal()); c.pm() << OpCode::Type::Jmp << top; if(!exprVar.valid()) { end = c.pm().position(); } c.pop(exprTotal); c.patchBreaks(c.pm().position()); c.popScope(); c.popBreaks();}void WhileNode::optimise(Context &c, Node *parent){ expr->optimise(c, this); block->optimise(c, this); exprVar = expr->query(NodeQuery::AsVariant); if(exprVar.valid() && !exprVar.toBool()) parent->remove(this);}Here [font='courier new']exprVar[/font] is a member variable of [font='courier new']WhileNode[/font] which is defaulted to invalid. It is then set in the [font='courier new']optimise()[/font] method and, if constant and false, we ask the parent to remove the [font='courier new']WhileNode[/font] altogether. If it is constant and true, we use this state to modify the code generation phase slightly to not bother generating the expression - just create an infinite loop instead.

Another minor optimisation is trimming out unreachable code. [font='courier new']ReturnNode[/font] and [font='courier new']BreakNode[/font] both respond with a [font='courier new']Variant(true)[/font] to the [font='courier new']NodeQuery::TerminatesBlock[/font], which [font='courier new']BlockNode[/font] uses to check for unreachable code:

void BlockNode::optimise(Context &c, Node *parent){ for(auto i = nodes.begin(); i != nodes.end(); ++i) { i->optimise(c, this); if(i->query(NodeQuery::TerminatesBlock).toBool()) { nodes.erase(i + 1, nodes.end()); return; } }}Since every scope creates its own [font='courier new']BlockNode[/font], it is demonstrably true that any nodes following a [font='courier new']ReturnNode[/font] or [font='courier new']BreakNode[/font] are unreachable, and can thus be deleted from the tree before we start generating bytecode.

So that is about the current state of play regarding optimisation.

Thanks, as always, for stopping by.

tl;dr: health much improved, wrapped [font='courier new']QtScript[/font] in a command line application with a DLL plugin system.

When I was messing around getting normal mapping working recently, I decided to use the alpha channel of the normal map as a specular value to allow for per-pixel specularity in my game. I decided to write a little Qt app that would allow me to load the normal map, then load a red-black image and import the red values into the alpha channel of the base image and save it.

I started this as a GUI application but it felt a bit heavy so I figured out how to set up a Qt console application and started writing a command-line app to do the task. It occurred to me that I could generalise this a bit and make the app extendable to support various image manipulations I might require in the future.

Then I started thinking, why don't I write a command line wrapper around the depreciated but still working [font='courier new']QtScript[/font]? I could expose a [font='courier new']QImage[/font] to the script easily enough, and implement [font='courier new']getPixel()[/font] and [font='courier new']setPixel()[/font] methods and so on, and loading and saving methods.

So once this was working, it then occurred to me - why make this specific to image processing? Why not have a generic wrapper around [font='courier new']QtScript[/font] that I can invoke on any script file, and gradually create a library of classes I expose to the script? Why not indeed.

The final bit of insanity was the decision to write a compleletly agnostic wrapper around [font='courier new']QtScript[/font] that loads DLLs that add classes to the script. And that is what we now have.

[font='courier new']Qsc.exe[/font] is the product of the [font='courier new']QscDriver[/font] project. It is a pretty trivial program:

int main(int argc, char *argv[]){ QCoreApplication app(argc, argv); if(app.arguments().count() < 2) { std::cout << "QSC Copyright(C)2017 Aardvajk Software Ltd\n"; std::cout << "USAGE: QSC script [arguments...]\n"; return 0; } QScriptEngine *engine = new QScriptEngine(&app); QStringList args; for(int i = 2; i < app.arguments().count(); ++i) args.push_back(app.arguments()); engine->globalObject().setProperty("args", qScriptValueFromSequence(engine, args)); QString bin = QCoreApplication::applicationDirPath(); foreach(const QString &plugin, QDir(bin).entryList(QStringList() << "*.dll" << "*.qsc", QDir::Files)) { if(QFileInfo(plugin).suffix() == "dll" && plugin != "QscApi.dll") { if(!loadDllPlugin(bin, plugin, engine)) { continue; } } else if(QFileInfo(plugin).suffix() == "qsc") { if(!loadQscPlugin(bin, plugin, engine)) { continue; } } } QFile file(app.arguments()[1]); if(!file.open(QIODevice::ReadOnly | QIODevice::Text)) { std::cerr << "QSC: Unable to open - " << app.arguments()[1].toStdString() << "\n"; return -1; } QScriptValue result = engine->evaluate(QString::fromUtf8(file.readAll())); if(result.isError()) { std::cerr << result.toString().toStdString() << "\n"; return -1; }}Bascially it just initailises a [font='courier new']QScriptEngine[/font], exposes an array of strings called [font='courier new']args[/font] to the script containing any command line parameters passed to [font='courier new']Qsc.exe[/font] (after the script file itself).

Then it gets the directory that [font='courier new']Qsc.exe[/font] is in and searches that directory, loading any DLLs it finds and calling a [font='courier new']QscInit(QScriptEngine*)[/font] method on them.

bool loadDllPlugin(const QString &bin, const QString &plugin, QScriptEngine *engine){ QString path = bin + QDir::separator() + plugin; HMODULE module = LoadLibraryA(path.toStdString().c_str()); if(module == NULL) { std::cerr << "Warning: unable to load plugin - " << plugin.toStdString() << "\n"; return false; } typedef void(*InitFunc)(QScriptEngine*); InitFunc func = reinterpret_cast(GetProcAddress(module, "QscInit")); if(func == NULL) { std::cerr << "Warning: unable to locate QscInit in plugin - " << plugin.toStdString() << "\n"; return false; } func(engine); return true;}Each DLL then just implements this call to add whatever classes it wants to the engine's [font='courier new']globalObject()[/font].

extern "C" __declspec(dllexport) void QscInit(QScriptEngine *engine){ engine->globalObject().setProperty("Color", engine->newFunction(colorConstructor)); engine->globalObject().setProperty("Image", engine->newFunction(imageConstructor));}It also searches for any files in the exe directory with a .QSC extension, which it treats as text files containing script that are just then evaluated by the [font='courier new']QScriptEngine[/font] prior to executing the actual script. This allows us to have a library of script functions and so on automatically loaded as well.

bool loadQscPlugin(const QString &bin, const QString &plugin, QScriptEngine *engine){ QFile file(bin + QDir::separator() + plugin); if(!file.open(QIODevice::ReadOnly | QIODevice::Text)) { std::cerr << "Warning: unable to load plugin - " << plugin.toStdString() << "\n"; return false; } QScriptValue result = engine->evaluate(QString::fromUtf8(file.readAll())); if(result.isError()) { std::cerr << "Warning: error evaluating plugin - " << plugin.toStdString() << "\n"; std::cerr << result.toString().toStdString() << "\n"; return false; } return true;}To aid the creation of the DLLs, there is a [font='courier new']QscApi[/font] project. This yields [font='courier new']QscApi.h[/font], [font='courier new']libQscApi.a[/font] and [font='courier new']QscApi.dll[/font] which is statically linked to each DLL and provides some helper methods.

#include namespace Qsc{template T objectData(const QScriptValue &object){ return qvariant_cast(object.data().toVariant());}template T objectData(QScriptContext *ctx){ return objectData(ctx->thisObject());}template T scriptCast(const QScriptValue &value){ return T(); }template<> int scriptCast(const QScriptValue &value){ return value.toInteger(); }template<> float scriptCast(const QScriptValue &value){ return value.toNumber(); }template<> QString scriptCast(const QScriptValue &value){ return value.toString(); }template QScriptValue getset(QScriptContext *ctx, QScriptEngine *engine, Get get, Set set){ T t = qvariant_cast(ctx->thisObject().data().toVariant()); if(ctx->argumentCount() == 1) { (t.*set)(scriptCast(ctx->argument(0))); ctx->thisObject().setData(engine->newVariant(t)); } return QScriptValue((t.*get)());}int checkArguments(QScriptContext *ctx, const QString &method, const QStringList &codes, QList &args);}Mental though some of this may look, it makes implementing the script classes a lot more concise, especially when combined with our wonderful new lambdas.

The [font='courier new']getet[/font] method is slightly interesting. It takes as its template arguments the type of the object represented by the script value, the type of the property of the object we are getting/setting, and types that end up being pointer-to-member-function pointers that are then invoked on the object.
The [font='courier new']scriptCast[/font] template and its specialisations then allow us to generically convert the argument from the [font='courier new']QScriptContext[/font] (a [font='courier new']QScriptValue)[/font] to the correct property type based on the template.
For example, here we set a property on a [font='courier new']Color[/font] object that contains a [font='courier new']QColor[/font] in its [font='courier new']QVariant[/font] data, using [font='courier new']int QColor::red()[/font] and [font='courier new']void QColor::setRed(int value)[/font].

void f(){ object.setProperty("red", engine->newFunction([](QScriptContext *c, QScriptEngine *e){ return Qsc::getset(c, e, &QColor::red, &QColor::setRed); } ), QScriptValue::PropertyGetter | QScriptValue::PropertySetter);}The ImageClass DLL implements both an Image and a Color class.

QScriptValue makeColor(QScriptContext *ctx, QScriptEngine *engine, QColor color){ QScriptValue object = engine->newObject(); object.setData(engine->newVariant(color)); object.setProperty("red", engine->newFunction([](QScriptContext *c, QScriptEngine *e){ return Qsc::getset(c, e, &QColor::red, &QColor::setRed); } ), QScriptValue::PropertyGetter | QScriptValue::PropertySetter); object.setProperty("green", engine->newFunction([](QScriptContext *c, QScriptEngine *e){ return Qsc::getset(c, e, &QColor::green, &QColor::setGreen); } ), QScriptValue::PropertyGetter | QScriptValue::PropertySetter); object.setProperty("blue", engine->newFunction([](QScriptContext *c, QScriptEngine *e){ return Qsc::getset(c, e, &QColor::blue, &QColor::setBlue); } ), QScriptValue::PropertyGetter | QScriptValue::PropertySetter); object.setProperty("alpha", engine->newFunction([](QScriptContext *c, QScriptEngine *e){ return Qsc::getset(c, e, &QColor::alpha, &QColor::setAlpha); } ), QScriptValue::PropertyGetter | QScriptValue::PropertySetter); return object;}QScriptValue colorConstructor(QScriptContext *ctx, QScriptEngine *engine){ QList ps; switch(Qsc::checkArguments(ctx, "Color", { "", "nnn", "nnnn" }, ps)) { case 0: return makeColor(ctx, engine, Qt::black); case 1: return makeColor(ctx, engine, QColor(ps[0].toInteger(), ps[1].toInteger(), ps[2].toInteger())); case 2: return makeColor(ctx, engine, QColor(ps[0].toInteger(), ps[1].toInteger(), ps[2].toInteger(), ps[3].toInteger())); } return QScriptValue();}QScriptValue imageConstructor(QScriptContext *ctx, QScriptEngine *engine){ QScriptValue object = engine->newObject(); QImage image; QList ps; switch(Qsc::checkArguments(ctx, "Image", { "nn", "s" }, ps)) { case 0: { image = QImage(ps[0].toInteger(), ps[1].toInteger(), QImage::Format_ARGB32); image.fill(Qt::black); } break; case 1: { if(!image.load(ps[0].toString())) { ctx->throwError(QString("Unable to open Image - %1").arg(ps[0].toString())); return QScriptValue(); } } break; default: return QScriptValue(); } object.setData(engine->newVariant(image)); object.setProperty("width", engine->newFunction([](QScriptContext *ctx, QScriptEngine*){ return QScriptValue(Qsc::objectData(ctx).width()); }), QScriptValue::PropertyGetter); object.setProperty("height", engine->newFunction([](QScriptContext *ctx, QScriptEngine*){ return QScriptValue(Qsc::objectData(ctx).height()); }), QScriptValue::PropertyGetter); object.setProperty("pixel", engine->newFunction(imagePixel)); object.setProperty("setPixel", engine->newFunction(imageSetPixel)); object.setProperty("save", engine->newFunction(imageSave)); return object;}[font='courier new']Qsc::checkarguments[/font] returns the index of the matching character code string (n = number, s = string, o = object etc) or -1 if none match, in which case it calls [font='courier new']QScriptContext::throwError()[/font] with an appropriate message.

We can take advanatage of Qt's [font='courier new']QVariant[/font] system to store the [font='courier new']QImage[/font] in the [font='courier new']QScriptValue[/font] here. [font='courier new']QImage[/font], like most of Qt's heavy classes, is copy-on-write so the amount of copying going on isn't as bad as it looks.
So finally, loading a red-black image into the alpha channel of a base image can be implemented as a [font='courier new']QtScript[/font] function:

var source = Image(args[0]);var alpha = Image(args[1]);for(var y = 0; y < source.height; ++y){ for(var x = 0; x < source.width; ++x) { var s = source.pixel(x, y); var c = alpha.pixel(x, y); s.alpha = c.red; source.setPixel(x, y, s); }}source.save(args[2]);Or something similar. Or this could be defined as a script function in a .QSC file in the exe directory and we can then just write a script that calls the function.

I've also added a [font='courier new']TextFile[/font] class (via another DLL of course) that loads a text file into a script array, and provides a [font='courier new']save()[/font] method as well. This is quite simple so will list this one in full.

#include "C:/Projects/Qsc/QscApi/QscApi/QscApi.h"#include QScriptValue textFileSave(QScriptContext *ctx, QScriptEngine *engine){ QScriptValue object = ctx->thisObject(); QList ps; switch(Qsc::checkArguments(ctx, "save", { "s" }, ps)) { case 0: break; default: return QScriptValue(); } QFile file(ps[0].toString()); if(!file.open(QIODevice::WriteOnly | QIODevice::Text)) { ctx->throwError(QString("Unable to create - %1").arg(ps[0].toString())); return QScriptValue(); } int length = object.property("length").toInteger(); for(int i = 0; i < length; ++i) { file.write(object.property(i).toString().toUtf8()); if(i < length - 1) { file.write("\n"); } } return QScriptValue();}QScriptValue textFileConstructor(QScriptContext *ctx, QScriptEngine *engine){ QScriptValue object = engine->newArray(); QList ps; switch(Qsc::checkArguments(ctx, "TextFile", { "", "s" }, ps)) { case 0: break; case 1: { QFile file(ps[0].toString()); if(!file.open(QIODevice::ReadOnly | QIODevice::Text)) { ctx->throwError(QString("Unable to open - %1").arg(ps[0].toString())); return QScriptValue(); } QString content = QString::fromUtf8(file.readAll()); int line = 0; foreach(const QString &text, content.split('\n', QString::KeepEmptyParts)) { object.setProperty(line++, text); } } } object.setProperty("save", engine->newFunction(textFileSave)); return object;}extern "C" __declspec(dllexport) void QscInit(QScriptEngine *engine){ engine->globalObject().setProperty("TextFile", engine->newFunction(textFileConstructor));}So I can now use [font='courier new']Qsc[/font] to manipulate text files in pretty much any way I want - something that, particularly for work, I often found myself setting up and writing temporary C++ projects which is always a pain and total overkill.

Few more bits to do - need to be able to invoke a function from the command line to avoid having to write a script if I just want to call one of my library methods on some arguments for example, and I should probably set up a system to provide more control over which DLLs and QSCs are loaded on startup.

So there you go. A scripting language I already know and am comfortable with, thoroughly extensible and should save me a great deal of time in the future.
We'll get back to the game next update. Thanks for stopping by.

## Various

tl:dr; collapsed in the street, spend a night in hospital. Transparent rendering works. Character controller works. Animation works. Have a first person arm-rig sort of working.

I'll start with a personal note - collapsed in the street last Wednesday and discovered I couldn't get up. A couple of very nice passers-by stopped and called me an ambulance and waited with me till it arrived - really restores your faith in humanity.

Ambulance scraped me up and took me to hospital where they decided I was severly anemic and transfused four units of blood through me overnight. Had an endoscopy the next day and, like an idiot, thought I'd get out of hospital quicker if I didn't have the sedation. Swallowed the camera okay - thought that would be the worst bit - but once it got as far down as my bowel I freaked right out and they ended up having to sedate me halfway through. I would not recommend anyone ever have an endoscopy without sedation - felt like I was being assimilated by the Borg :)

Anyway, they let me out and I'm feeling a lot better. Need to eat a bit better but my body should be able to maintain the levels now okay. All good fun. Smashed my hand and knees up a bit and a bit achy from where I fell but otherwise seem to have fully recovered and am back to work already.

I decided to wimp out and make my game first-person perspective. It is just so much easier and increases the chance I might actually get a game out of this project.

Have made some slight architectural changes to the [font='courier new']Scene[/font] manager in my game. I have set it up now so that there are multiple [font='courier new']RenderGroups[/font] - currently [font='courier new']RenderGroup::Main[/font] and [font='courier new']RenderGroup::Transparents[/font]. I've added a [font='courier new']Scene::prepare(const SceneParams&)[/font] method that can then sort the [font='courier new']RenderGroup::Transparents[/font] into back to front order based on the current view and projection matrices so that they render correctly. Nothing very exciting really.

The player character is now a proper kinematic character controller with the camera sat inside a capsule, and collides and slides with the level geometry correctly.

I imported my skeleton animation system into my [font='courier new']Gx[/font] library and have set up first-person rig using a nice low-poly arm model I found on the internet. The camera is actually attached to a head bone in the rig, so when I animate the rig, the camera should move with the head nicely.

I'm struggling to make decent animation using my model editor though. I need to spend a bit more time on this figuring out what the problems are and improving the editor's facilities.

So [font='courier new']GxAnimation[/font] is structured as follows.

You have a [font='courier new']Gx::Skeleton[/font] class that represents the skeleton, loaded from one of the bespoke binary formats my model editor exports. The basic idea is that you set a [font='courier new']Gx::KeyFrame[/font] on the skeleton, then read back the [font='courier new']Gx::MatrixPalette[/font] to feed into the scene node that then passes this into the rigging vertex shaders.

class Skeleton{public: Skeleton(); Skeleton(const Skeleton &other); ~Skeleton(); Skeleton &operator=(const Skeleton &other); Index count() const; SkeletonJoint joint(Index index) const; void setKeyFrame(const KeyFrame &key); Matrix parentMatrix(Index index) const; Vec3 bindPosition(Index index, const Matrix &world) const; Vec3 transformedPosition(Index index, const Matrix &world) const; Matrix placementTransform(Index index, const Matrix &offset, const Matrix &world) const; Index index(const std::string &name) const; std::string name(Index index) const; MatrixPalette palette() const; void loadFromStream(DataInStream &ds); static const unsigned char invalidJoint = 255;private: Matrix buildMatrix(Index index, const PodVector &pose) const; class Rep; Rep *rep;};I had great fun importing this from the old code, when I decided to implement it using PIMPL and forgot to add copy and assignment constructors. That drove me mad for a while.

The [font='courier new']placementTransform()[/font] method is very useful - you can use this to, for example, attach a sword to a hand bone and so on.

The [font='courier new']Gx::KeyFrames[/font] are generated from the [font='courier new']Gx::AnimationController[/font] class, which is able to interpolate between any number of running animation tracks. [font='courier new']Gx::Animations[/font] are loaded into a [font='courier new']Gx::AnimationMap[/font] from the binary export file and you can then set a current animation, or ask for a transition from one animation to another.

class AnimationController{public: AnimationController(); AnimationController(const AnimationController &other); ~AnimationController(); AnimationController &operator=(const AnimationController &other); void setCurrent(AnimationId id); void transitionTo(AnimationId id, bool matchToCurrent); AnimationId currentAnimation() const; void store(); void update(float delta, const AnimationMap &map); KeyFrame keyFrame(float blend, const AnimationMap &map) const; Signal event;private: void registerId(AnimationId id); void checkEvents(AnimationId id, const Animation &a, float from, float to); class Rep; Rep *rep;};So, game-side, it looks a bit like this in the [font='courier new']Pc[/font] class representing the player character:

Gx::ViewTransform Pc::viewTransform(float blend) const{ Gx::Matrix tr = Gx::rotationYMatrix(data.ang.value(blend).x) * Gx::translationMatrix(data.pos.value(blend)); Gx::Vec3 pos = data.anim.skeleton.transformedPosition(data.anim.headBone, tr); Gx::Vec2 ang = data.ang.value(blend); return Gx::ViewTransform(pos, ang);}Gx::Matrix Pc::calculateView(float blend){ data.anim.skeleton.setKeyFrame(data.anim.controller.keyFrame(blend, data.anim.map)); Gx::ViewTransform tr = viewTransform(blend); Gx::Vec3 look, right, up; Gx::computeViewVectors(look, right, up, tr.angle()); return Gx::lookAtMatrix(tr.position(), tr.position() + look, up);}void Pc::update(GameState &state, float delta){ data.pos.store(); data.ang.store(); Gx::Vec3 step(0, 0, 0); // snip, read the input events to update the step for the KCC data.kcc.move(state.physics, step); data.pos.set(data.kcc.position()); data.anim.controller.store(); data.anim.controller.update(delta, data.anim.map);}void Pc::prepareScene(SceneParams ¶ms, float blend){ node->setPalette(data.anim.skeleton.palette()); node->setTransform(Gx::translationMatrix(Gx::Vec3(0, -1.5f, 0)) * Gx::inverseMatrix(params.view));}This is probably all a bit impenetrable to the casual reader. Not sure why I bother posting it really.

Anyway, that's pretty much where we are. Next step is to try and figure out how to make some better animations for the player rig and get some kind of weapon attached to the hand.

## Progress Update

tl;dr: My level editor lets you assign a texture and/or normal map to each face of the level geometry and the game dynamically generates pixel shaders to support the required combinations.

Progress Report:

I've updated the level editor now to support setting an optional diffuse texture and/or normal map on each face of the geometry brushes. These are located on the "Assets" path you set as a level global property.

Each face also has properties for the UV scale and offset, so you have a lot of flexibility here.

When generating the vertex buffer for the solid geometry, the editor now groups all the triangles that share the same textures and keeps track of the offsets into the buffer for each group, so that when we render, we can now draw each group separately, setting the relevant samplers before each call to [font='courier new']DrawPrimitive[/font].

This is achived by using a simple [font='courier new']RenderKey[/font] class that provides [font='courier new']operator==[/font] and [font='courier new']qHash[/font] so it can be used with a [font='courier new']QHash[/font] or [font='courier new']QSet:[/font]

class RenderKey{public: RenderKey(){ } RenderKey(const QString &texture, const QString &normalMap) : texture(texture), normalMap(normalMap) { } bool operator==(const RenderKey &key) const { return texture == key.texture && normalMap == key.normalMap; } QString texture; QString normalMap;};inline uint qHash(const RenderKey &key){ return qHash(key.texture) ^ qHash(key.normalMap);}When generating the preview buffer, we first run over the entities and collect a [font='courier new']QPair[/font] representing an entity index and a face index into a [font='courier new']QHash > >[/font] mapping structure that is then walked to generate the vertices into the buffer so that they are grouped correctly.

When the editor exports the level to the file format supported by the game, it generates a separate mesh per RenderKey. Later we can add more attributes to [font='courier new']RenderKey[/font], such as a manual [font='courier new']Group[/font] attribute that will allow the level desiger to sub-divide the level geometry into geographical groups for the purposes of frustrum culling and so on.

In theory, we can just update [font='courier new']RenderKey[/font] now with any additional attributes and everything else will "just work".

Back to the game and the almost immediate discovery that I have run into the issue of shader permutations - the great unsolved problem in graphics programming according to the internet. I guess everyone stumbles across this eventually.

I want to support both textures and normal maps being option on faces and to even support having a normal map but no texture. My main pixel shader was doing both texture and normal map sampling for everything as was, because everything was using a generic texture and normal map so this was fine. But now, of course, I need versions of the pixel shader to support:

- no texture or normal map
- texture only
- normal map only
- texture and noraml map

And so the fun begins.

I was pleased to discover that it is pretty easy to feed into the HLSL preprocessor with [font='courier new']D3DXCompileShaderFromFile[/font] by passing in an array of [font='courier new']D3DXMACRO[/font] structures. Readers are probably already groaning as they can see the insanity ahead :)

So game side, we now have a [font='courier new']RenderParams[/font] structure that each scene node provides, declaring a [font='courier new']Gx::GraphicsResourceId[/font] (which can be null) for the texture and normal map. An [font='courier new']operator<[/font] is provided so we can use this in a [font='courier new']std::set[/font]. I'll change this to support hashing into a [font='courier new']std::unordered_set[/font] later on.

class RenderParams{public: enum { Texture = 1, NormalMap = 2 }; RenderParams() : features(0) { } Gx::GraphicsResourceId texture; Gx::GraphicsResourceId normalMap; Gx::Index features;};Gx::Index renderFeatures(const RenderParams ¶ms);bool operator==(const RenderParams &a, const RenderParams &b);bool operator!=(const RenderParams &a, const RenderParams &b);bool operator<(const RenderParams &a, const RenderParams &b);[font='courier new']Gx::Index[/font] is just a typedef for [font='courier new']unsigned int[/font]. Once we have set the texture and normalMap IDs in the level loader or wherever, we set up the [font='courier new']features[/font] property as a bitwise combination of [font='courier new']Feature[/font] flags based on the [font='courier new']valid()[/font] states of the resource IDs.

Gx::Index renderFeatures(const RenderParams ¶ms){ Gx::Index features = 0; if(params.texture.valid()) features |= RenderParams::Texture; if(params.normalMap.valid()) features |= RenderParams::NormalMap; return features;}Trivial of course, but thinking ahead for future expansion.

The [font='courier new']Scene[/font] class [font='courier new']render()[/font] method now takes the node's [font='courier new']RenderParams[/font] into account now when deciding when it needs to do a state change, so as long as we sort the scene nodes correctly, we can minimise the number of state changes as we render the scene.

In the level loading code now, as we load the mesh instances from the file, we insert their features value into a [font='courier new']std::set[/font] so we have a list of the possible permutations. This gets fed into [font='courier new']loadShaderSet():[/font]

The features attribute handily maps to an integer between 0 and (at the moment) 3 which we can append to the .dat file name to distinguish each one from the others.

We can also store the IDs of the shaders in an array and we can then just use the features attribute of a given [font='courier new']RenderParams[/font] as an index into the array to select the correct shader when we are setting up the state.

The [font='courier new']DBEUGLOAD[/font] define can be disabled in the distributed version of the game. I had some issues on a friend's PC once that was missing the shader compiler runtime so I prefer to ship the precompiled shader binaries only and just invoke the compiler locally. My shader loading code uses the timestamps of the source .txt and the binary .dat to decide whether to recompile or not.

Obviously when distributing, we have to make sure we ship every possible permutation but the numeric suffix makes this easy to confirm and the game will still only load the shaders it requires for each level.

bool loadShaderSet(Gx::Shader::Type type, const std::string &id, const std::set &permutations, Gx::Graphics &graphics, Gx::GraphicsResourceId *ids){ std::string ts = (type == Gx::Shader::Type::Vertex ? "vertex" : "pixel"); std::string url = id + ts + "shader"; std::string file = "shaders/" + id + ts; for(auto &ps: permutations) { std::string datFile = Gx::stringFormat(file, "_", ps, ".dat");#ifdef DEBUGLOAD Gx::PodVector macros; if(ps & RenderParams::Texture) macros.push_back({ "TEXTURE", "1" }); if(ps & RenderParams::NormalMap) macros.push_back({ "NORMAL", "1" }); macros.push_back({ NULL, NULL }); if(needsCompiling(file + ".txt", datFile)) { if(!compileShader(type, file + ".txt", datFile, macros)) return Gx::GraphicsResourceId(); }#endif Gx::DataInFileStream ds(datFile); if(ds.fail()) return false; Gx::GraphicsResourceId id = makeShader(type, Gx::stringFormat(url, "_", ps), ds.get(), graphics); if(!id) { return false; } ids[ps] = id; } return true;}Here we set up the macros based on the permutations that have been generated when loading the level and we generate a different shader binary for each permutation.

The shader then uses the preprocessor to include/exclude features as required, and we only generate the shaders we actually need for the particular level.

So yet another wheel thoroughly reinvented. I'm not completely sure I've finished this yet in terms of how it is structured, but it should allow me to expand with other optional features if I need them painlessly enough.

In other news, I did an experiment yesterday to see what the performance gains of using a position-only mesh structure for rendering the depth cubemap would be. I currently just use the same [font='courier new']GeneralVertex[/font] structure for all the rendering which includes UV, tangent, diffuse and a bunch of other attributes that aren't required by the depth render.

First up, I generated some timing data based on something I know has a big effect on performance to ensure my timing code was working properly - namely the size of the cube map used to render the depth. The graph below shows the times to render frames 100 to 199 using a cube map with dimensions 512, 1024 and 2048 respectively.

This is pretty much what I would expect so I was confident the timing code was correct.

I then set things up so that, for each mesh, I generated a clone of the mesh containing only the position for each vertex, set up an equivalent vertex declaration and set up the scene node rendering to use the position-only mesh when rendering the depth to the cube map. The graph below shows the results for frames 100 to 199:

So, makes no apparent difference. I guess when the levels get larger, it is possible we will start having the vertex buffers getting paged on and off the GPU, in which case this might become more effective so perhaps it is a bit premature to be testing this while the level geometry is so simple and small. Can always revisit this later on.

## Multiple Lights and Shadows - Video and Code Walkthrough

So finally a video showing the multiple lights and shadows in motion. It's a lot smoother here, maintaining 60 fps effortlessly on my machine. My keyboard response code seems to break down a bit when recording with CamStudio for some reason so the video doesn't look quite so smooth.

I'll break down the rendering stages in this entry.

First a quick tale about why one should really start using [font='courier new']nullptr[/font] over a literal zero when assigning null defaults to pointers.

[font='courier new']Pc[/font] is the name of the class I have representing the player character, which is currently the red capsules you see in the screenshots and videos. It is derived from [font='courier new']Entity[/font] and the entities are stored and owned by the [font='courier new']LevelModel[/font], but I also needed to keep a (non-owning) pointer to one in my [font='courier new']GameMode[/font] to update the light position and so on.

Originally I had one [font='courier new']Pc *pc[/font] pointer as a member of [font='courier new']GameMode[/font], which I initialised to 0 in the [font='courier new']GameMode[/font] constructor:

GameMode::GameMode(Gx::Graphics &graphics, Gx::ApplicationEvents &events) : graphics(graphics), events(events), pc(0){ receiver.connect(events.keyDown, this, &GameMode::keyDown);}When I wanted to start mesisng with multiple lights, I changed [font='courier new']Pc *pc[/font] in [font='courier new']GameMode[/font] header to [font='courier new']Gx::PodVector pc[/font].

Everything compiled and apparently worked okay, then I noticed that my game was crashing on shutdown, and very occasionally if I reloaded the level. After much gnashing of teeth, I discovered if I swapped out the [font='courier new']Gx::PodVector[/font] for [font='courier new']std::vector[/font], the issue went away. But I have been using the code upon which [font='courier new']Gx::PodVector[/font] is based for literally years without any issues of this sort.

Turns out that [font='courier new']GameMode[/font]'s constructor still had [font='courier new']pc(0)[/font] in it...

Now, one of the many constructors that [font='courier new']Gx::PodVector[/font] takes is one that defines the initial size/capacity. This value is also then used as the chunk size when growing:

template class PodVector{public: // snip PodVector(size_type s) : r(s, s) { } // snipprivate: struct rep { rep(size_type r, size_type n) : chunk(r) { front = new T[r]; back = (front + n) - 1; cap = (front + r) - 1; } };};Now, of course, because I used 0 instead of [font='courier new']nullptr[/font] in [font='courier new']GameMode[/font]'s constructor, I was creating a [font='courier new']PodVector[/font] that calls [font='courier new']new T[0][/font]. Whoops. And in great C++ fashion, this was silently working, stomping all over unallocated memory, and not crashing until the program actually shut down.

Had I used [font='courier new']pc(nullptr)[/font] in the [font='courier new']GameMode[/font] constructor, as soon as I turned the pointer into a [font='courier new']PodVector[/font], the compiler would have caught the error and saved me several hours of frustration :)

So the rendering process for this is as follows. Nothing original going on here I'd stress, all just standard techniques.

My system works based on a [font='courier new']Scene[/font] class that maintains (but does not own) a list of pointers to [font='courier new']SceneNodes[/font], the only one of which at the moment is a [font='courier new']StaticMeshNode[/font].

We have [font='courier new']RenderPass[/font] and [font='courier new']RenderType[/font] defined as follows (ignoring [font='courier new']Particle[/font] for now):

enum class RenderPass { Base, Lit, Depth };enum class RenderType { General, Particle, Null };[font='courier new']SceneNode[/font] has the following interface:

class SceneNode{public: SceneNode(); virtual ~SceneNode(); virtual bool pass(RenderPass type) const = 0; virtual RenderType type() const = 0; virtual void render(RenderPass pass, Gx::Graphics &graphics, const SceneParams ¶ms, float blend) const = 0; Gx::Signal destroyed;};[font='courier new']SceneParams[/font] is just a small structure we use to pass the parameters around more easily and will make future expansion easier on the code. Barely worth sharing at the moment:

class SceneParams{public: SceneParams(){ } Gx::Matrix view; Gx::Matrix proj; Gx::Vec3 shadowLightPos; Gx::Vec4 lightColor;};So a node can tell the Scene which passes it should be involved in, which [font='courier new']RenderType[/font] it uses (just [font='courier new']General[/font] used at the moment) and can be told to render itself, which for [font='courier new']StaticMeshNode[/font] is as simple as:

void StaticMeshNode::render(RenderPass pass, Gx::Graphics &graphics, const SceneParams ¶ms, float blend) const{ graphics.device.vertexShader().setMatrix(graphics.device, "world", world); graphics.device.renderTriangleList(graphics.resources.get(mesh));}[font='courier new']Scene[/font] then uses the following loop when you call its render() method with a given [font='courier new']RenderPass:[/font]

void Scene::render(RenderPass pass, Gx::Graphics &graphics, const SceneParams ¶ms, float blend){ RenderType curr = RenderType::Null; for(auto i: nodes) { if(i->pass(pass)) { RenderType type = i->type(); if(curr != type) { endType(graphics); curr = type; beginType(pass, type, graphics, params); } if(curr != RenderType::Null) { i->render(pass, graphics, params, blend); } } } endType(graphics);}[font='courier new']beginType()[/font] and [font='courier new']endType()[/font] just set up the global states, shaders, blend states etc for each render type. It is currently up to the external owner of [font='courier new']Scene[/font] to ensure the [font='courier new']SceneNodes[/font] are ordered in a way that makes this efficient.

So with all this in mind, the process to render multiple lights and shadows is as follows:

1) Disable color writes, do a depth-only render of the scene from the camera's perspective to get the Z buffer set up correctly.

This also has the benefit that subsequent renders to the frame buffer will only run the pixel shader on visible pixels. This uses the simplest possible vertex shader and no pixel shader:

matrix world;matrix viewproj;struct VS_INPUT{ vector position : POSITION;};struct VS_OUTPUT{ vector position : POSITION;};VS_OUTPUT main(VS_INPUT input){ VS_OUTPUT output = (VS_OUTPUT)0; output.position = mul(input.position, mul(world, viewproj)); return output;}2) For each light source:

2a) Get hold of the pre-allocated cube texture (512x512x512 D3DFMT_R32F) and, for each of the six faces, set the face as the render target, set a pre-allocated 512x512 depth/stencil surface, set up the correct 90 degree view matrix and render the scene.

void GameMode::renderShadowCube(const Gx::Vec3 &lightPos, float blend){ Gx::CubeMap &map = graphics.resources.get("shadowcube"); Gx::DepthStencilSurface &depthStencil = graphics.resources.get("shadowcubedepthstencil"); Gx::Vec3 look[6] = { Gx::Vec3(1, 0, 0), Gx::Vec3(-1, 0, 0), Gx::Vec3(0, 1, 0), Gx::Vec3(0, -1, 0), Gx::Vec3(0, 0, 1), Gx::Vec3(0, 0, -1) }; Gx::Vec3 up[6] = { Gx::Vec3(0, 1, 0), Gx::Vec3(0, 1, 0), Gx::Vec3(0, 0, -1), Gx::Vec3(0, 0, 1), Gx::Vec3(0, 1, 0), Gx::Vec3(0, 1, 0) }; SceneParams params; params.proj = Gx::perspectiveMatrix(Gx::Size(map.dimension(), map.dimension()), M_PI / 2, 0.1f, 200); params.shadowLightPos = lightPos; Gx::RenderContext old(graphics.device); for(int i = D3DCUBEMAP_FACE_POSITIVE_X; i <= D3DCUBEMAP_FACE_NEGATIVE_Z; ++i) { params.view = Gx::lookAtMatrix(lightPos, lightPos + look, up); Gx::RenderContext cube(map, static_cast(i), depthStencil); cube.apply(graphics.device); graphics.device.clear(Gx::Color(0, 1, 0, 1), 1); scene.render(RenderPass::Depth, graphics, params, blend); } old.apply(graphics.device);}This uses the following vertex and pixel shaders:

matrix world;matrix viewproj;struct VS_INPUT{ vector position : POSITION;};struct VS_OUTPUT{ vector position : POSITION; vector worldpos : TEXCOORD;};VS_OUTPUT main(VS_INPUT input){ VS_OUTPUT output = (VS_OUTPUT)0; output.position = mul(input.position, mul(world, viewproj)); output.worldpos = mul(input.position, world); return output;}float3 shadowLight;struct PS_INPUT{ vector worldpos : TEXCOORD;};struct PS_OUTPUT{ vector diffuse : COLOR;};PS_OUTPUT main(PS_INPUT input){ PS_OUTPUT output = (PS_OUTPUT)0; float dist = length(((float3)input.worldpos) - shadowLight); output.diffuse = vector(dist / 200, 0, 0, 1); return output;}2b) Render the scene from the camera's perspective, with the cube texture from 2a) set as input to the pixel shader. Enable alphablending and set the blend states to [font='courier new']D3DBLEND_SRCCOLOR[/font], [font='courier new']D3DBLEND_ONE[/font]. This uses the following vertex and pixel shaders - the pixel shader is the most complex in this system.

matrix world;matrix viewproj;struct VS_INPUT{ vector position : POSITION; float3 normal : NORMAL; vector diffuse : COLOR;};struct VS_OUTPUT{ vector position : POSITION; vector diffuse : COLOR; float3 normal : TEXCOORD0; vector worldpos : TEXCOORD1;};VS_OUTPUT main(VS_INPUT input){ VS_OUTPUT output = (VS_OUTPUT)0; output.position = mul(input.position, mul(world, viewproj)); output.normal = mul(input.normal, (float3x3)world); output.diffuse = input.diffuse; output.worldpos = mul(input.position, world); return output;}sampler ds : register(s0);float3 shadowLight;vector lightColor;struct PS_INPUT{ vector diffuse : COLOR; float3 normal : TEXCOORD0; float3 worldpos : TEXCOORD1;};struct PS_OUTPUT{ vector diffuse : COLOR;};float shadowDarkness = 0.5f;float shadowFac(float3 tolight){ float sampled = texCUBE(ds, tolight).r; float actual = (length(tolight) - 0.1f) / 200; if(sampled < actual) return shadowDarkness; return 0;}PS_OUTPUT main(PS_INPUT input){ PS_OUTPUT output = (PS_OUTPUT)0; float3 n = normalize(input.normal); float3 toShadowLight = ((float3)input.worldpos + (n * 0.08f)) - shadowLight; float shadow = (dot(toShadowLight, n) < 0) ? shadowFac(toShadowLight) : shadowDarkness; float s = dot(n, normalize(-toShadowLight)); if(s < 0.3f) s = 0.3f; float d = length(toShadowLight); s *= 1 - saturate(d / 60); output.diffuse = input.diffuse * lightColor * s * (1 - shadow); output.diffuse.a = 1; return output;}Speaks for itself really, entirely standard and not yet optimised. For example, I could store and compare the squared distance rather than the distance easily enough. But I wanted it working in a non-optimised fashion first so I can get some real world frame timing that I can then compare to when I start optimising as I'm doubtful it will make a lot of difference.

The radius of each light (60) is just hardcoded at the moment. Obviously this can be a new input parameter eventually.

So far, my impression is that the main overhead in this system is the fill rate on the texture cubes. This is evidenced by how much faster three lights with 512x512x512 cubes are than with 1024x1024x1024 cubes.

I'm also applying no PCF or similar to the shadows at the moment. Wanted to get some real frame timing before I added this in to get a realistic idea of the cost.

So the main [font='courier new']GameMode::render()[/font] looks like ([font='courier new']Base[/font] is the [font='courier new']RenderPass[/font] for the depth-only render to the frame buffer):

void GameMode::render(float blend){ SceneParams params = sceneParams(blend); model.prepareScene(params, blend); graphics.device.clear(Gx::Color(0, 0, 0, 0), 1.0f); scene.render(RenderPass::Base, graphics, params, blend); for(Gx::Index i = 0; i < pc.size(); ++i) { params.shadowLightPos = pc->position(blend) + Gx::Vec3(0, 2, 0); params.lightColor = Gx::Vec4(1, 1, 1, 1); renderShadowCube(params.shadowLightPos, blend); scene.render(RenderPass::Lit, graphics, params, blend); }}So there you have it. Thanks for stopping by.

This is more of a design ramble than a technical issue that I have today. I was hoping to take the approach of having multiple point lights in my game levels, but just use a single shadow caster, but as you can see from the screenshot, this creates some weird situations where a shadow can be cast directly onto a light source which creates a really incorrect result.

As I see it I have three main options here.

• Live with unrealistic lighting. It isn't that important that a dungeon-crawler presents a realistically lit scene.
• Continue to experiment with multiple shadow casters and try to overcome the problems I had. If I use 512x512x512 texture cubes, I seem to be able to render up to about five or six shadow cubes per frame without sacrificing frame rate on my mid-range hardware, so this is technically possible.
• A new approach of maybe fading shadows out if they are falling on pixels that have been lit by another light source. I haven't really thought this through but might be another option.
I'm interested in canvassing for people's experience with this issue. I cannot seem to find the correct terms to Google to find other people's thoughts on this. Everything I look up is taking me to techniques for doing multiple shadow casting. I cannot be the only person to want to use a single shadow caster in an environment with multiple point lights, so if anyone wants to chip in with their experience and what they tried or ended up doing, I'd be fascinated to hear.

In the meantime, work has me pretty busy so will just keep a thread running at the back of my brain on this and see if I get any sudden inspiration.

I'm really hoping that whatever solution I come up with doesn't mean I have to apply a lot of restrictions to level design to accomodate. Will be a real pain if the only way I can get acceptable results is to apply a set of very rigid rules to how the level layouts work in terms of where lights and shadow casters are allowed. Obviously I can't expect "just anything" to work properly, but be very nice to be fairly free to design levels without having to worry too much about this.

Thanks for stopping by and please do contribute any thoughts you have on the subject.

[EDIT] Woo hoo

Colour me pretty damn happy with that :)

Loads to do to improve but that will be next post.

[EDIT again]

And one more screenshot for tonight... :)

We are maintaining a steady 60 fps at 1280x768 on my mid-range laptop so seems three lights is going to work fine. Four starts to push it unless I drop the cubemaps down to 256x256x256 at which point the shadow edges start looking a bit ropey, but I reckon I can live with three lights active at a time in a dungeon crawler.

The main thing is that lights can be mobile, as I can see a lot of good gameplay coming from the player having to carry a light source around dark areas.

## Some progress (added new vid)

Making some decent progress on the game over the last few days.

First up, I implemented a billboard-based particle system, used for the torch effect in the video above. It uses the following sprite atlas that I stole from the internet somewhere and generates a dynamic vertex buffer each frame. Since I only need a position and a texture coordinate in the vertex structure, I decided to use triangle lists rather than indexed buffers for this case and it performs pretty well.

Next up, I've started work on implementing shadow mapping from an omni-directional point light. This uses the usual method of rendering depth into a texture cube then projecting this onto the scene. Here's one face of the texture cube, false coloured but you get the idea.

I'm actually using D3DFMT_R32F format for the first time. I have previously used D3DFMT_A8R8G8B8 and packed/unpacked the depth into the colour channels but I've noticed I seem to get much better results using a floating point texture format and I believe this is supported pretty much everywhere now.

The video is using a cube map of dimension 512x512 and currently no PCF or filtering of any kind and looks okay. Some quick and dirty experiments suggest I can happily render eight shadow cubes a frame without dropping below 60 fps on my mid-range laptop, so I'm hoping that I can finally crack multiple point-light shadowing and lighting in this game.

The approach I want to take with multiple point lights is to have a consistent shadow colour and a consistent lit colour across the screen i.e. each shadowed pixel should be the same level of darkness and each visible pixel should be as if it has only been lit once.

I'm thinking vaguely about doing a first pass drawing everything as if it is in shadow, then clearing the z-buffer and doing a pass for each light where the pixel shader first tests to see if it is in shadow and discards the pixel if so. I'm not yet sure how to solve the "light once" problem for the pixels that are not in shadow, but plan to experiment to see if this is possible.

Anyway, that's all for now. Not a very technical entry and not like me to post no code, but there is nothing very interesting behind any of this in terms of C++ so we'll leave this here.

Thanks for stopping by.

[EDIT] I was asked by a non-technical friend how the fire effect worked so made the below video, not really aimed at you lot as you already understand billboard particle systems, I'm sure :)

Thought I'd stick it in here for posterity anyway. It's also on my Facebook. You never know, might manage to engage the interest of at least one non-programmer, weird bunch though they are :)

## New Game

Just a very quick entry today since I haven't posted for a while, to introduce my new game. Not that we have much to show yet.

My vision for this project is as follows:

"What do we get if we cross Super Mario 3D with a Rogue-Like?"

The answer - Rogue-Like Platform Game or RPG for short.

Essentially the screenshot shows that we are loading polyhedra from the level file and creating them as physics objects, we are loading a static mesh from the level file and displaying it and we have a very simple character controller working that walks on slopes, falls under gravity and so on.

What I want to do, unlike previous 3D platform projects, is concentrate on the level graphics a bit first rather than the central character, so plan is to just use the debug capsule for the player for a while and instead try to make the levels look nice with textures, lighting and the associated features in the editor which make the levels easy to create.

I've got some per-pixel dynamic lighting stuff working in the level editor ready to import into the game so I thought the first place to start might be to make some particle emitters for torches, using billboard sprites to create a nice fire effect. Not something I've done in 3D before so be nice to be working on something new for a while.

That's really all for now. Probably wasn't really worth the update. Sorry.

[EDIT]

Since this has been featured and shared on Facebook, thought I'd just tart the entry up a bit with my progress on a particle system.

Looks better in motion. This is eventually going to be an emitter for torches that can either be statically placed on walls or carried around by other entities and will tie into the dynamic lighting system.

That is all :)

## I reinvented JSON :)

As always happens when I'm starting a new game, I've started a new level editor. This one, however, is based on my [font='courier new']Gx[/font] and [font='courier new']QGx[/font] frameworks and so I'm hoping to keep it generic enough that it might be the last one I ever have to write (yeah, right) :)

I'm going to need to have human-readable configuration files for the editor and I hate working with XML, especially with the Qt interfaces. I also find it very hard to read. So I've reimplemented an old system I invented years ago as part of the [font='courier new']Gx[/font] framework, called [font='courier new']Gx::Settings[/font].

It is more like JSON than XML. Here's an example of a script:

window{ width = 640; height = 480; maximized = false; controls { title = "This is a literal"; }}levels{ "C:/levels/level1.dat"; "C:/levels/level2.dat"; "C:/levels/level3.dat";}Each entry starts with either an ID (starts with letter or underscore, contains letters, numbers and underscores) or a literal. In the second case, an unnamed node is created with the provided value. In the former case, you have an optional assignment of a literal to the value, then an optional brace enclosed list of entries that become child nodes of the node.

The semicolons are pretty much ignored and are there to aid readability. They are required because two string literals separated only by whitespace are concatenated into one string, so in the "levels" example above, they need to be there.

The code-side interface to this is a recursive [font='courier new']Gx::Settings[/font] object. This is a PIMPL-based class implementing the rule of five:

namespace Gx{class Settings{public: Settings(); Settings(const Settings &s); Settings(Settings &&s); ~Settings(); Settings &operator=(const Settings &s); Settings &operator=(Settings &&s); // snipprivate: class Rep; Rep *rep;};}[font='courier new']Gx::Settings[/font] offers a non-const-only [font='courier new']operator[](const std::string &id)[/font] which either finds or creates a new node. You can also assign a [font='courier new']std::string[/font] to it, which sets its value. There is then a template assign method that allows you to directly assign any supported type.

Similarly there are template [font='courier new']value()[/font] and [font='courier new']canConvert()[/font] methods for retrieving the value directly into a supported type.

It is based upon [font='courier new']Gx::lexical_cast[/font] which uses the standard-library stringstreams for doing conversions, so any type that provides overloads for the stream inseration and extraction operators can be used.

void f(){ Gx::Settings root; Gx::Vec3 v(1, 2, 3); // assume stream operators provided for type root["hello"]["world"] = v; if(root["hello"]["world"].canConvert()) { Gx::Vec3 t = root["hello"]["world"].value(); }}[font='courier new']Gx::Settings[/font] also provides [font='courier new']iterator[/font] and [font='courier new']const_iterator[/font] for traversing its children. From the "levels" example above:

void listLevels(Gx::Settings &root){ for(auto &i: root["levels"]) { std::cout << i.value() << "\n"; }}The child nodes are stored in each parent in the order they are created. Calling the [font='courier new']operator[][/font] with an empty string argument appends an unnamed node to the end of the list.

Finally there is a static method for loading from a stream. The following loads a configuration file and prints it recursively as a tree:

void dump(Gx::Settings &node, int indent){ for(const auto &i: node) { std::cout << i.id() << " = " << i.value() << "\n"; dump(node, indent + 4); }}void f(){ std::ifstream is("data.txt"); if(is.is_open()) { std::string error; Gx::Settings root = Gx::Settings::fromStream(is, error); if(root.empty()) { std::cerr << "Error " << error << "\n"; return; } dump(root, 0); }}This is the entire header for [font='courier new']Gx::Settings[/font] itself. There are also some internal-only files that deal with parsing and constructing the node tree from a file, but they obviously don't form any of the public interface of the library.

namespace Gx{class Settings{public: class iterator { public: iterator &operator++(){ ++index; return *this; } iterator operator++(int){ return iterator(parent, index++); } iterator &operator--(){ --index; return *this; } iterator operator--(int){ return iterator(parent, index--); } iterator operator+(unsigned int s) const { return iterator(parent, index + s); } iterator operator-(unsigned int s) const { return iterator(parent, index - s); } bool operator==(const iterator &o) const { return parent == o.parent && index == o.index; } bool operator!=(const iterator &o) const { return parent != o.parent || index != o.index; } Settings &operator*() const; Settings *operator->() const; private: friend class Settings; iterator(Settings *parent, unsigned int index) : parent(parent), index(index) { } Settings *parent; unsigned int index; }; class const_iterator { public: const_iterator &operator++(){ ++index; return *this; } const_iterator operator++(int){ return const_iterator(parent, index++); } const_iterator &operator--(){ --index; return *this; } const_iterator operator--(int){ return const_iterator(parent, index--); } const_iterator operator+(unsigned int s) const { return const_iterator(parent, index + s); } const_iterator operator-(unsigned int s) const { return const_iterator(parent, index - s); } bool operator==(const const_iterator &o) const { return parent == o.parent && index == o.index; } bool operator!=(const const_iterator &o) const { return parent != o.parent || index != o.index; } const Settings &operator*() const; const Settings *operator->() const; private: friend class Settings; const_iterator(const Settings *parent, unsigned int index) : parent(parent), index(index) { } const Settings *parent; unsigned int index; }; Settings(); Settings(const Settings &s); Settings(Settings &&s); ~Settings(); Settings &operator=(const Settings &s); Settings &operator=(Settings &&s); Settings &operator[](const std::string &id); bool contains(const std::string &id) const; bool empty() const; Settings &operator=(const std::string &value); Settings &operator=(const char *value); template Settings &operator=(const T &t){ return (*this) = lexical_cast(t); } template T value() const { return lexical_cast(val()); } template bool canConvert() const { return can_lexical_cast(val()); } iterator begin(); iterator end(); const_iterator begin() const; const_iterator end() const; std::string id() const; static Settings fromStream(std::istream &is, std::string &error);private: std::string val() const; class Rep; Rep *rep;};}So we have both easily-readable and editable text file sources for defining arbitrary hierachical data and a fairly nice and neat interface for manipulating this data in the host application and a simple way to extend it to support custom types by just requiring stream insertion and extraction operators to convert a custom type to and from a string.

And so [font='courier new']Gx[/font] progresses. Time to get back into the editor now :)

Thanks for stopping by.

## Gx::Camera wafflings

Time for the next gripping installment of the development of my [font='courier new']Gx[/font] game framework library. The latest addition I'll waffle about today is the [font='courier new']Gx::Camera[/font] class from [font='courier new']GxMaths[/font].

The requirement I have for the camera is that it can either be controlled directly by the user, for debugging and development purposes and also run in a sort of tracking mode, where you set a target position and rotation each frame and the camera smoothly moves towards this target.

As a result, it is necessary for me to store the rotation of the camera as a quaternion, so I can easily do smooth interpolations. However, I find quaternions hard to visualise, so I also want to be able to talk to the camera in terms of a 2D vector of angles.

With all this in mind, the place to start is (most of) the public interface to [font='courier new']Gx::Camera[/font].

namespace Gx{class Camera{public: enum class Mode { Manual, Tracking }; Camera(); Camera(const ViewTransform &transform); Camera(const Vec3 &position, const Vec2 &rotation); void store(); void update(float delta); void update(float trackingSpeed, float delta); ViewTransform transform() const; ViewTransform transform(float blend) const; void setTransform(const ViewTransform &transform); void setTarget(const ViewTransform &transform); Matrix viewMatrix(float blend) const; Vec3 position(float blend) const; Mode mode() const;};So what is a [font='courier new']Gx::ViewTransform[/font] when it is at home? It is a simple enough class that aggregates a position and a quaternion representation of a rotation, but also provides an interface to express the rotation in terms of two angles.

namespace Gx{class ViewTransform{public: ViewTransform(); ViewTransform(const Vec3 &position, const Quaternion &rotation); ViewTransform(const Vec3 &position, const Vec2 &angle); Vec3 position() const; void setPosition(const Vec3 &value); Quaternion rotation() const; void setRotation(const Quaternion &value); Vec2 angle() const; void setAngle(const Vec2 &value); void flatVectors(Vec3 &look, Vec3 &right) const;private: Vec3 pos; Quaternion rot;};ViewTransform interpolate(const ViewTransform &a, const ViewTransform &b, float t);}All pretty self-explanatory. Internally we just use a method from [font='courier new']Gx::Quaternion[/font]'s home file to change the angle into a quaternion, for example:

void Gx::ViewTransform::setAngle(const Vec2 &value){ rot = axisRotationQuaternion(Vec3(value.y, value.x, 0));}This just translates into a call to [font='courier new']D3DXQuaternionRotationYawPitchRoll[/font] internally.

The next question you may be asking (and I appreciate you are probably not but let's pretend you are) is what the [font='courier new']blend[/font] parameter is all about in the various getter methods on [font='courier new']Gx::Camera[/font].

My game loops are always based on the Gaffer fix-your-timestep approach, where we render as fast as possible but call the update loop with a fixed timestep (normally 1 / 60 but can be set to anything). To keep everything looking smooth in such a system, when you render, the game loop produces a [font='courier new']blend[/font] value between 0 and 1, which is a kind of representation of the accumulated error between the actual framerate and the physics step.

The results are quite striking in terms of how much smoother everything looks, especially when you have a lower framerate or a lot of variation in how long the render takes from frame to frame.

To support this, I have a simple but widely-used [font='courier new']Gx::BlendValue[/font] class, which takes care of storing and interpolating between a value and its previous state.

namespace Gx{template class BlendValue{public: BlendValue(const T &t = T()) : v(t), o(v) { } template BlendValue(Args... args) : v(args...), o(v) { } void store(){ o = v; } void set(const T &t){ v = t; } void add(const T &t){ v += t; } T value() const { return v; } T value(float blend) const { return interpolate(o, v, blend); }private: T v; T o;};typedef BlendValue BlendFloat;typedef BlendValue BlendVec2;typedef BlendValue BlendVec3;typedef BlendValue BlendQuaternion;typedef BlendValue BlendViewTransform;}This can be instantiated with any type for which a free [font='courier new']interpolate[/font] method is defined. You call [font='courier new']store()[/font] somewhere at the start of the frame, then use [font='courier new']set()[/font] or [font='courier new']add()[/font] to modify the value. Then when you are rendering, you can just call the version of [font='courier new']value()[/font] which takes the [font='courier new']blend[/font] parameter to get the interpolated value for the current frame.

Looking at a few of the [font='courier new']Gx::Camera[/font]'s methods with this in mind:

void Gx::Camera::store(){ frame.store();}void Gx::Camera::update(float delta){ update(8, delta);}void Gx::Camera::update(float trackingSpeed, float delta){ if(m == Mode::Tracking) { t += delta * trackingSpeed; frame.set(interpolate(cur, tar, t)); if(t >= 1) { cur = tar; frame.set(cur); } } else { frame.set(cur); }}Gx::Matrix Gx::Camera::viewMatrix(float blend) const{ Vec3 look(0, 0, 1); Vec3 up(0, 1, 0); ViewTransform tr = frame.value(blend); Quaternion r = tr.rotation(); look = transformNormal(look, r); up = transformNormal(up, r); Vec3 p = tr.position(); return lookAtMatrix(p, p + look, up);}This ultimately gives us a camera that works within the game loop system we are using.

I wanted to have a fairly easy way to control the camera with user-input as well from the game project using the library but wanted to decouple the camera from the input events system I have. [font='courier new']Gx::ApplicationEvents[/font] is a class that takes care of informing the application about input events.

class ApplicationEvents{public: ApplicationEvents(); bool isKeyDown(int key) const; Vec2 rawMouseDelta() const; Signal activated; Signal keyDown; Signal keyUp; Signal graphicsDeviceReset;};We won't bother going into details of how all this is implemented. Suffice to say it has a [font='courier new']wndProc[/font] at its heart and uses [font='courier new']WM_INPUT[/font] messages to get raw mouse information.

So [font='courier new']Gx::Camera[/font] has a chid class - [font='courier new']Gx::Camera::Step[/font] and a [font='courier new']step()[/font] method:

namespace Gx{class Camera{public: class Step { public: Step(); bool forward; bool backward; bool left; bool right; bool rise; bool fall; float speed; Vec2 rawMouseDelta; }; void step(const Step &step, float delta);};This is nice and easy to use from a host project as long as we have access to the [font='courier new']Gx::ApplicationEvents[/font] instance, which is a member of [font='courier new']Gx::Application[/font]. I just bind references to this when I need it in other classes. So in my game application, we just call this method each frame:

void GameMode::updateCamera(float delta){ camera.store(); Gx::Camera::Step step; step.forward = events.isKeyDown('W'); step.backward = events.isKeyDown('S'); step.left = events.isKeyDown('A'); step.right = events.isKeyDown('D'); step.rise = events.isKeyDown('R'); step.fall = events.isKeyDown('F'); step.speed = 10; step.rawMouseDelta = events.rawMouseDelta(); camera.step(step, delta);}Once we get to the point where we have a central character running around the level, we can instead run the camera in its tracking mode, where we just work out the point above and behind the player's head, work out the angle that points the camera in the appropriate direction, then call the [font='courier new']setTarget()[/font] method each frame.

We then just have to call the [font='courier new']update()[/font] method of the camera each frame and the camera will tracking smoothly in a third-person style.

So, that is the [font='courier new']Gx::Camera[/font] class.

On a personal note, I'm going to scrap my car tonight and do without transport for a while. Seems one of my front suspension springs broke yesterday. It really was the suspension spring that broke the camel's back as far as throwing any more money at that rusty old heap goes. It has passed the end of its useful life and I'd be a fool to throw another penny at it.

It has cost me a fortune to keep it on the road and MOT-ed over the last couple of years. Enough is enough.

I walked from home to where I work (I technically work as a C++ developer from home, but can't stand working at home so work from a local Wetherspoons pub every day, taking advantage of the free coffee refills and wi-fi) carrying my laptop bag this morning. Took about 45 minutes, which is bearable.

Not having the car will be a pain in some ways, but it also kind of feels like I'm taking a massive weight off my shoulders. Even without the constant repair bills, I reckon fuel, insurance, tax and car-parking are costing me between GBP200 and GBP300 a month, so when the walking is bothering me, I can just think about my bank balance to cheer myself up. Quite like walking anyway. It's good for my sanity.

Thanks for stopping by.

## Building a bridge (to Qt)

Having made a good start on [font='courier new']Gx[/font] and having set up a game project that links to the [font='courier new']Gx[/font] headers and lib (in-place in the [font='courier new']Gx[/font] project for ease of update), I realised that a new game will need a new level editor, which for me means using the [font='courier new']Gx[/font] library from within a Qt project.

This has the most impact on the [font='courier new']Gx::GraphicsDevice[/font], which has now undergone some changes to support.

In a game project, [font='courier new']Gx[/font] just creates one graphics window. However, when using [font='courier new']Gx::GrahicsDevice[/font] in a Qt project for a level editor, we commonly want to be able to create any number of graphics views, on the fly, that can be hosted inside widgets.

So in addition to Gx, I have now made a start on a small library called [font='courier new']QGx[/font], which is designed to be a bridge between [font='courier new']Gx[/font] and Qt.

[font='courier new']QGx[/font] currently just contains [font='courier new']QGx::GraphicsDevice[/font], [font='courier new']QGx::GraphicsWidget[/font] and [font='courier new']QGx::Graphics[/font].

In order to facilitate this, [font='courier new']Gx::GraphicsDevice[/font] has been split inside the [font='courier new']Gx[/font] library into [font='courier new']Gx::AbstractGraphicsDevice[/font] and [font='courier new']Gx::GraphicsDevice[/font]. This is not for use in a polymorphic sense - an application will never use [font='courier new']Gx::AbstractGraphicsDevice[/font]. It is just a way to have the [font='courier new']Gx::GraphicsDevice[/font] and [font='courier new']QGx::GraphicsDevice[/font] share most of the methods.

So this is the public interface to [font='courier new']Gx::AbstractGraphicsDevice:[/font]

class AbstractGraphicsDevice{public: AbstractGraphicsDevice(); virtual ~AbstractGraphicsDevice(); void clear(Color color, float z); void setVertexDeclaration(const VertexDeclaration &declaration); void setVertexDeclaration(); void setVertexShader(VertexShader &shader); void setVertexShader(); void setPixelShader(PixelShader &shader); void setPixelShader(); VertexShader &vertexShader(); PixelShader &pixelShader(); void setTexture(Index stage, const Texture &texture); void setTexture(Index stage, const CubeMap &texture); void setTexture(Index stage); void renderTriangleList(const VertexBuffer &buffer); void renderLineList(const VertexBuffer &buffer); void renderTriangleList(const VertexBuffer &buffer, const IndexBuffer &indices); bool isLost() const; bool isReadyToReset() const;};Extended thus by [font='courier new']Gx::GraphicsDevice:[/font]

class GraphicsDevice : public AbstractGraphicsDevice{public: GraphicsDevice(); ~GraphicsDevice(); bool acquire(HWND hw, const DisplaySettings &settings); bool reset(const DisplaySettings &settings); bool reset(); void release(); void begin(); void end(); DisplaySettings settings() const;};This gives us the same [font='courier new']Gx::GraphicsDevice[/font] as we had before.

But now, in [font='courier new']QGx[/font] library, the public interface to [font='courier new']QGx::GraphicsDevice[/font] is as follows:

class GraphicsDevice : public Gx::AbstractGraphicsDevice{public: GraphicsDevice(); ~GraphicsDevice(); bool initialise(); void registerWidget(GraphicsWidget *widget); void unregisterWidget(GraphicsWidget *widget); bool reset(); void begin(GraphicsWidget *widget); void end(GraphicsWidget *widget); bool needsResetting() const; bool isReadyToReset() const; void scheduleReset();};A completely different approach, where you initialise once, then you add one or more [font='courier new']QGx::GraphicsWidget[/font] instances, and you also now specify which [font='courier new']QGx::GraphicsWidget[/font] you are targetting when you [font='courier new']begin()[/font] and [font='courier new']end()[/font].

There is some slight hackery in [font='courier new']QGx::GraphicsDevice[/font] that means we create a dummy device when there are no [font='courier new']QGx::GraphicsWidgets[/font] registered, inside the initialise call. This dummy device is destroyed and removed as soon as we add a [font='courier new']QGx::GraphicsWidget[/font], but it means we have a valid device before we create any [font='courier new']QGx::GraphicsWidgets[/font], so we are thus able to create global resources before we create the widgets. Not ideal, but most of the [font='courier new']Gx::GraphicsResources[/font] map to Direct3D types that need to be passed a valid device when they are created so I can live with it.

[font='courier new']QGx::GraphicsWidget[/font] is designed to be a base class that you derive your graphics views from. You then just implement the [font='courier new']paintEvent()[/font] method on your derived class, and use the accessor [font='courier new']QGx::Graphics &graphics()[/font] to access the [font='courier new']QGx::Graphics[/font] instance, which is an aggregate of [font='courier new']QGx::GraphicsDevice[/font] and a [font='courier new']Gx::GraphicsResources[/font] object (which is a typedef for [font='courier new']Gx::ResourceMap[/font]), all inherited from the [font='courier new']Gx[/font] library.

So you can now use all of the other [font='courier new']Gx::GraphicsResource[/font] classes in the Qt project and any new ones added are available automatically on a rebuild. It is only the set-up and view management methods that are different.

[font='courier new']QGx::GraphicsWidget[/font] provides the following interface:

class Graphics;class GraphicsWidget : public QWidget{ Q_OBJECTpublic: GraphicsWidget(Graphics &graphics, QWidget *parent = 0); ~GraphicsWidget();protected: virtual QPaintEngine* paintEngine() const; virtual void resizeEvent(QResizeEvent *event); Graphics &graphics();private: friend class GraphicsDevice; Graphics *g;};Internally, it implements the basics of registering and unresigstering itself from the device in the way you would expect:

QGx::GraphicsWidget::GraphicsWidget(Graphics &graphics, QWidget *parent) : QWidget(parent), g(&graphics){ setAttribute(Qt::WA_PaintOnScreen); setAttribute(Qt::WA_NoSystemBackground); setMinimumSize(64, 64); setSizePolicy(QSizePolicy::MinimumExpanding, QSizePolicy::MinimumExpanding); g->device.registerWidget(this);}QGx::GraphicsWidget::~GraphicsWidget(){ if(g) g->device.unregisterWidget(this);}QPaintEngine* QGx::GraphicsWidget::paintEngine() const{ return 0;}void QGx::GraphicsWidget::resizeEvent(QResizeEvent *event){ QWidget::resizeEvent(event); if(g) g->device.scheduleReset();}QGx::Graphics &QGx::GraphicsWidget::graphics(){ return *g;}So at the moment I have a simple test Qt application that allows you to add and remove graphics views from a [font='courier new']QVBoxLayout[/font] when buttons are clicked. [font='courier new']MyWidget[/font] is an example derived from [font='courier new']QGx::GraphicsWidget[/font], like so:

class MyWidget : public QGx::GraphicsWidget{public: MyWidget(QGx::Graphics &graphics, Gx::Color color, QWidget *parent);protected: virtual void paintEvent(QPaintEvent *event);private: Gx::Color color;};MyWidget::MyWidget(QGx::Graphics &graphics, Gx::Color color, QWidget *parent) : QGx::GraphicsWidget(graphics, parent), color(color){}void MyWidget::paintEvent(QPaintEvent *event){ if(graphics().device.isLost()) return; graphics().device.begin(this); graphics().device.clear(color, 1.0f); graphics().device.end(this);}So just like any other Qt widget really. In the [font='courier new']paintEvent()[/font], we first check to ensure the device is valid and, if so, we use the protected [font='courier new']graphics()[/font] method to get hold of the device and the resources map and use all the methods we inherit from [font='courier new']Gx::GraphicsDevice[/font] to do whatever rendering we like. The rendering is now constrained to this particular graphics view, whose size is taken from the widget.

If the widget is resized, this triggers a [font='courier new']reset()[/font] internally.

The main loop of the program fires a timer over and over, which calls a method on [font='courier new']MainWindow[/font]. This method calls back a method on [font='courier new']QGx::Graphics[/font] which checks for device lost and so on:

bool QGx::Graphics::testForReset(){ if(device.isLost()) { if(!device.isReadyToReset()) { return false; } } if(device.isReadyToReset()) { device.scheduleReset(); } if(device.needsResetting()) { for(auto &r: resources) { if(r.isDeviceBound()) { r.release(); } } if(!device.reset()) { return false; } for(auto &r: resources) { if(r.isDeviceBound()) { r.reset(device); } } deviceReset(*this); } return true;}The [font='courier new']MainWindow[/font] timer method then emits a [font='courier new']render()[/font] signal, which we connect to the [font='courier new']update()[/font] slot on any [font='courier new']QGx::GraphicsWidget[/font] classes we create.

void MainWindow::renderTimeout(){ if(!graphics.testForReset()) { return; } emit render();}So when we hit the button to create a new view in [font='courier new']MainWindow[/font], it is just standard [font='courier new']QWidget[/font] stuff now, except we have to pass the [font='courier new']QGx::Graphics[/font] to the widget's constructor.

void MainWindow::buttonPressed(){ static Gx::Color colors[3] = { Gx::Color(1, 0, 0, 1), Gx::Color(0, 1, 0, 1), Gx::Color(0, 0, 1, 1) }; static int index = 0; MyWidget *widget = new MyWidget(graphics, colors[index++], this); if(index >= 3) index = 0; connect(this, SIGNAL(render()), widget, SLOT(update())); centralWidget()->layout()->addWidget(widget);}And when we hit the button to delete the last view in the list:

void MainWindow::removePressed(){ QLayoutItem *i = centralWidget()->layout()->takeAt(centralWidget()->layout()->count() - 1); delete i->widget(); delete i;}That automatically takes care of unregistering the widget from the [font='courier new']QGx::GraphicsDevice[/font]'s internal list.

So all ready to go when I want a GUI application that uses Direct3D windows now. Thankfully my old level editor is working well enough to not have to worry about that quite yet.

I've added the [font='courier new']GxPhysics[/font] module to Gx now as well. It isn't fully formed yet, but it is essentially a [font='courier new']Gx::Physics[/font] object representing the world, a [font='courier new']Gx::Body[/font] object that wraps up Bullet's [font='courier new']btRigidBody[/font] and a whole bunch of shapes deriving from [font='courier new']Gx::Shape[/font] which wrap individual Bullet shape classes such as [font='courier new']Gx::Polyhedron[/font] for [font='courier new']btConvexHull[/font] and [font='courier new']Gx::Sphere[/font] for [font='courier new']btSphere[/font] and so on.

I've started a bare-bones game project now that can load physics objects and graphic meshes from my old level editor and display the level and a debug render of the physics polyhedra. As I'm working now, I'm adding things to [font='courier new']Gx[/font] as and when I need them which is proving a reasonable work-flow - lots of jumping between two projects and lots of clean/rebuild going on, but at least things feel a bit tidier and setting up new projects in the future will be a lot easier.

Hopefully soon I'll achieve something pretty enough to post some screenshots instead of all this code. We'll see :)

Thanks for stopping by.

## Framework Fun

About time I posted an update here. Been a while.

I have kind of lost a bit of heart in Om, my scripting language, since I realised that circular references lead to memory leaks unless I implement some kind of garbage collection. I've not given up on it - I will indeed try at some point to add GC to Om, but I decided I needed to start a new game project to cheer myself up.

Whenever I start a game, I end up copying loads of files from the previous project into the new one. This is a habit I got into maybe ten years ago, when I discovered that trying to maintain my own libraries across several projects at once was a nightmare. I'd modify the library to do something in the current project and from that point on, only the current project would compile ever again :)

But, I'm 42 years old, I should be mature enough to be able to maintain my own personal library. So I've been working on creating a game framework, strictly for personal use. I've called it [font='courier new']Gx[/font], which stands for, erm, Glorious Xylophones or something.

Setting up a library is a good exercise actually. It's making me really think about and improve upon the shared code that all my game projects end up using. The last couple of days, I've been working on my resource mapping classes which I primarily use for managing graphics resources (but there was no reason to not write the map stuff in a more generic way).

I have a [font='courier new']Gx::GraphicsResource[/font] base class that all of the graphics resources use as an interface. It contains methods like [font='courier new']release()[/font], [font='courier new']reset()[/font], [font='courier new']isDeviceBound()[/font] and so on which allow us to treat all the resources in a uniform manner during a device reset.

Graphics resources in my games end up needing to be stored in such a way that I can iterate over them before and after a device reset. There are also always two types of graphics resource - what I'll tentatively call "global" resources and what we can call "local" resources for the sake of this discussion.

For example, I commonly have vertex declarations defined once on startup for the various vertex types I'm using and these are shared across the code. But an [font='courier new']Entity[/font] may well need to own a vertex buffer of its own, and the buffer's lifetime should match that of the [font='courier new']Entity[/font].

So the resource map supports two types of storage - one by a [font='courier new']std::string[/font] url for the global resources and another by using handles which allows us to RAII-away the resources when the owner of the handle goes bye-bye.

An improvement I've made while porting all this into [font='courier new']Gx[/font] is to make the handles more strongly typed, using templates, so that you cannot accidentally use a handle from one map type in another. Previously the handles were just a loose wrapper around an index into the map but now I've also added some type information.

We start with [font='courier new']Gx::ResourceId[/font]. In my previous projects this was just a [font='courier new']typedef[/font] for [font='courier new']unsigned int[/font] which meant it had no information about what it was pointing at. Now, we typedef [font='courier new']Gx::Index[/font] from [font='courier new']DWORD[/font] (from the Win API) and make [font='courier new']Gx::ResourceId[/font] a template class.

This lives in [font='courier new']GxResourceHandle.h[/font] where we also define the generic [font='courier new']Gx::ResourceHandle[/font] and [font='courier new']Gx::TypedResourceHandle[/font]. The former you cast to type when you access it, the latter binds the type to the handle when it is declared.

#include #include namespace Gx{template class ResourceMap;template class ResourceId{public: ResourceId() : id(invalidIndex) { } bool valid() const { return id != invalidIndex; }private: friend class ResourceMap; explicit ResourceId(Index id) : id(id) { } Index id;};template class ResourceHandle{public: ResourceHandle() : id(invalidIndex), map(nullptr) { } ~ResourceHandle(){ destroyed(id); } template T &value(); template const T &value() const; Gx::Signal destroyed;private: friend class ResourceMap; Index id; ResourceMap *map;};template class TypedResourceHandle{public: TypedResourceHandle() : id(invalidIndex), map(nullptr) { } ~TypedResourceHandle(){ destroyed(id); } T &value(); const T &value() const; Signal destroyed;private: friend class ResourceMap; Index id; ResourceMap *map;};}[font='courier new']Gx::Signal[/font] is part of my signals/slots implementation which I wrote an article for the site about a while back. It uses varadic templates to provide a reasonably efficient system for communication between objects that are ignorant of each other's type.

The actual implementation of the [font='courier new']value()[/font] methods is implemented in [font='courier new']GxResourceMap.h[/font] as they need to access methods of the map but divding the files up this way means I can just use forward-declaration in other headers for classes that own a handle and just include the actual map in the .cpps.

We also have Gx::SharedResourceHandle for consistency. This is a very simple, copyable class that you can use to extract out one of the global resources by url. This just allows us to access a global resource using the same handle interface as the others, and binds its type to the handle when declared to avoid accidentally casting the wrong type.

template class SharedResourceHandle{public: SharedResourceHandle() : id(invalidIndex), map(nullptr) { } T &value(); const T &value() const;private: friend class ResourceMap; Index id; ResourceMap *map;};[font='courier new']GxResourceMap.h[/font] is a larger file, so I'll just present the interface to the map, then look at some usage code.

namespace Gx{template class ResourceMap{public: class iterator { public: bool operator==(const iterator &o) const { return v == o.v && i == o.i; } bool operator!=(const iterator &o) const { return v != o.v || i != o.i; } iterator &operator++(){ i = next(i); return *this; } iterator operator++(int){ Index n = i; i = next(i); return iterator(v, n); } Base &operator*(){ return *((*v)); } Base *operator->(){ return (*v); } private: friend class ResourceMap; iterator(PodVector *v, Index i) : v(v), i(i) { } Index next(Index i){ if(i == v->size()) return i; ++i; while(i < v->size() && !((*v))) ++i; return i; } PodVector *v; Index i; }; class const_iterator { public: bool operator==(const iterator &o) const { return v == o.v && i == o.i; } bool operator!=(const iterator &o) const { return v != o.v || i != o.i; } const_iterator &operator++(){ i = next(i); return *this; } const_iterator operator++(int){ Index n = i; i = next(i); return const_iterator(v, n); } const Base &operator*() const { return *((*v)); } const Base *operator->() const { return (*v); } private: friend class ResourceMap; const_iterator(const PodVector *v, Index i) : v(v), i(i) { } Index next(Index i){ if(i == v->size()) return i; ++i; while(i < v->size() && !((*v))) ++i; return i; } const PodVector *v; Index i; }; ResourceMap(){ } ~ResourceMap(); template T &add(const std::string &url, T *p); template T &add(ResourceHandle &handle, T *p); template T &add(TypedResourceHandle &handle, T *p); template T &get(const ResourceId &id){ return *static_cast(resources[id.id]); } template T &get(const std::string &url){ return get(id(url)); } ResourceId id(const std::string &url) const; Base *operator[](Index id){ return resources[id]; } iterator begin(){ return iterator(&resources, 0); } iterator end(){ return iterator(&resources, resources.size()); } const_iterator begin() const { return iterator(&resources, 0); } const_iterator end() const { return iterator(&resources, resources.size()); }private: Index internalAdd(Base *resource); void handleDestroyed(Index id); Receiver receiver; PodVector resources; PodVector free; std::map mapping;};Nothing particularly earth-shattering in the implementations. But now we have the map methods defined, we can implement the [font='courier new']value()[/font] methods of [font='courier new']Gx::ResourceHandle[/font] and [font='courier new']Gx::TypedResourceHandle:[/font]

template template T &ResourceHandle::value(){ return *(static_cast((*map)[id])); }template template const T &ResourceHandle::value() const { return *(static_cast((*map)[id])); }template T &TypedResourceHandle::value(){ return *(static_cast((*map)[id])); }template const T &TypedResourceHandle::value() const { return *(static_cast((*map)[id])); }template T &SharedResourceHandle::value(){ return *(static_cast((*map)[id])); }template const T &SharedResourceHandle::value() const{ return *(static_cast((*map)[id])); }In a debug mode, we could use [font='courier new']dynamic_cast[/font] checks here to ensure we have the correct type, but this has never actually been an issue when using this type of map for me, so I'm just using [font='courier new']static_cast[/font] now.

The other point of interest regarding the handles is in the [font='courier new']add()[/font] methods.

template template T &ResourceMap::add(const std::string &url, T *p){ Index id = internalAdd(p); mapping[url] = id; return *p;}template template T &ResourceMap::add(ResourceHandle &handle, T *p){ handle.id = internalAdd(p); handle.map = this; receiver.connect(handle.destroyed, this, &ResourceMap::handleDestroyed); return *p;}template template T &ResourceMap::add(TypedResourceHandle &handle, T *p){ handle.id = internalAdd(p); handle.map = this; receiver.connect(handle.destroyed, this, &ResourceMap::handleDestroyed); return *p;}The simplest is the [font='courier new']std::string[/font]-based url add, that simply uses a [font='courier new']std::map[/font] to keep the assocaition.

The handles are non-copyable, but we pass them by reference into the add methods and we have declared the [font='courier new']ResourceMap[/font] as a friend of the handle classes, so it can populate their internals.

The [font='courier new']Gx::ResourceMap[/font] also has a [font='courier new']Gx::Receiver[/font], to which we can connect [font='courier new']Gx::Signals[/font], so we connect the handles [font='courier new']destroyed(Index)[/font] signal up so that when the handles go out of scope, the map is informed and can remove the resources they point to.

template void ResourceMap::handleDestroyed(Index id){ delete resources[id]; resources[id] = 0; free.push_back(id);}We also have a [font='courier new']shared()[/font] method to get back a global, url-based resource so we can use it with a consistent handle interface:

template template SharedResourceHandle ResourceMap::shared(const ResourceId &id){ SharedResourceHandle handle; handle.id = id.id; handle.map = this; return handle;}template template SharedResourceHandle ResourceMap::shared(const std::string &url){ return shared(id(url));}Simple as that. So now we can look at some usage. [font='courier new']Gx::Application[/font] contains a protected [font='courier new']Gx::Graphics[/font] object, which is a composition of a [font='courier new']Gx::GraphicsDeice[/font] and [font='courier new']Gx::ResourceMap[/font]. So lets have examples of url, generic resource and typed resource handles.

class Application : public Gx::Application{public: virtual bool createResources(); virtual void render(float blend); Gx::ResourceHandle vertexBuffer; Gx::TypedResourceHandle vertexShader; Gx::SharedResourceHandle decHandle};bool Application::createResources(){ graphics.resources.add("colorvertexdec", new Gx::VertexDeclaration(/* ... */)).reset(graphics.device); graphics.resources.add(vertexBuffer, new VertexBuffer(/* ... */)).reset(graphics.device); graphics.resources.add(vertexShader, new VertexShader(/* ... */)).reset(graphics.device); decHandle = graphics.resources.shared("colorvertexdec"); return true;}void Application::render(float blend){ graphics.device.setVertexDeclaration(graphics.resources.get("colorvertexdec")); // or auto id = graphics.resources.id("colorvertexdec"); // auto -> Gx::ResourceId graphics.device.setVertexDeclaration(graphics.resources.get(id)); // or graphics.device.setVertexDeclaration(decHandle.value()); Gx::VertexBuffer &buffer = vertexBuffer.value(); // generic handle, cast on access buffer.begin(D3DLOCK_DISCARD); /* ... */ buffer.end(); Gx::VertexShader &shader = vertexShader.value(); // typed handle, type encoded in declaration Gx::Matrix wvp = /* ... */ graphics.device.setVertexShader(shader); graphics.device.vertexShader().setMatrix("worldviewproj", wvp); graphics.device.renderTriangleList(buffer);}The vertex declaration stays in the map until it is manually removed or the map is destroyed, whereas the buffer and shader have their lifetimes controlled by the owning [font='courier new']Application[/font] object.

Internally then in [font='courier new']Gx::Application[/font], we handle device reset and the game loop like this:

int Gx::Application::exec(){ ShowWindow(hw, SW_SHOW); MSG msg; PeekMessage(&msg, NULL, 0, 0, PM_NOREMOVE); Timer timer; const float delta = 1.0f / 60.0f; float accumulator = delta; while(msg.message != WM_QUIT) { while(PeekMessage(&msg, NULL, 0, 0, PM_REMOVE)) { TranslateMessage(&msg); DispatchMessage(&msg); } if(!graphics.device.isLost()) { if(graphics.device.isReadyToReset()) { for(auto &r: graphics.resources) { if(r.isDeviceBound()) { r.release(); } } if(!graphics.device.reset()) { return errorWindow("Unable to reset graphics device"); } for(auto &r: graphics.resources) { if(r.isDeviceBound()) { if(!r.reset(graphics.device)) { return errorWindow("Unable to reset graphics resource"); } } } if(!graphicsDeviceReset()) { return errorWindow("Unable to reset graphics resources"); } } float t = timer.elapsed(Gx::Timer::Option::Restart); accumulator += t; while(accumulator >= delta) { update(delta); accumulator -= delta; } graphics.device.begin(); render(accumulator / delta); graphics.device.end(); } } return 0;}So any resources currently in the map that are device-bound are released and reset at the appropriate time. Otherwise, a standard Gaffer-On-Games. fix-your-timestep game loop calling a couple of virtual methods for the class deriving from [font='courier new']Gx::Application[/font] to fill in.

I'm not actually sure if the generic [font='courier new']Gx::ResourceHandle[/font] is needed since as far as I can remember, I have always known the type that a handle should point at when I declare the handle, but that's the thing about assembling a library - you tend to start thinking in slightly different ways and YAGNI doesn't apply quite as strongly.

[font='courier new']Gx[/font] is currently composed of:

GxCore GxCoreTypes.h GxDataStream.h GxFlags.h GxPodVector.h GxPtrVector.h GxResourceHandle.h GxResourceMap.h GxScopedPtr.h GxSignal.h GxSize.h GxStringFormat.h GxTimer.cpp GxTimer.h GxWindows.cpp GxWindows.hGxGraphics GxCubeMap.h GxDepthStencilSurface.h GxDisplaySettings.h GxFont.h GxGraphics.h GxGraphicsBuffer.h GxGraphicsDevice.h GxGraphicsResource.h GxRenderContext.h GxShader.h GxTexture.h GxVertexDeclaration.h GxVertexElement.hGxMaths GxMathTypess.h GxMatrix.h GxQuaternion.h GxRay.h GxVec2.h GxVec3.hGxApplication GxApplication.hPlan is to add [font='courier new']GxPhysics[/font], a wrapper around Bullet that I already have written but will rewrite as part of the port and [font='courier new']GxAnimation[/font], my own implemetation of a skeleton animation system closely tied to my 3D model editor (Charm).

It is difficult sometimes deciding what should be in the [font='courier new']Gx[/font] library and what is specific to the game. Plan is, once I have the basics of [font='courier new']GxPhysics[/font] working in my test driver application, to start the game project and set up the includes to go directly into the source tree of the [font='courier new']Gx[/font] project, so I can easily change [font='courier new']Gx[/font] while I am working on the game.

Plan is to write a 3D platformer and KEEP IT SIMPLE this time. I always end up trying to write Tomb Raider when I should be trying to write more of a Super Mario 3D style platform game. So I'm going to have a simpler central character - no edge grabbing for now, nice air-controls for cartoon jumping and just try to stay on the straight and narrow and not get over-ambitious.

That is all for now. Thanks for stopping by.

## Om Module system

Thought I'd try blogging about this before I started implementing anything to try to get it straight in my head. I've decided to change how the Om module system works a little, based on playing around with writing a driver to allow me to execute Om scripts from within Windows explorer.

At the moment, state of play is that you can do this:

import print;print("hello world");The import statement tells the compiler that the symbol print is to be looked up in the modules list that the engine maintains in its shared state. It doesn't have to be populated until the first time that print is used - it just informs the compiler to emit the OpCode::Type::GetMd instruction when it finds the symbol.

Currently, the host application has to call Om::Engine::addModule(name, value) to put the relevant module value into place.

What I want to do differently is for the import statement itself to take care of this. I also want two different types of module to be avaialble.

I was playing around writing DLLs to support extending Om with native code. It turns out to be pretty easy (using the same toolchain at least) to write a DLL that can expose native code as an Om module. For example:

#include #include Om::Value print(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶ms){ for(auto &p: params) std::cout << p.toString().c_str(); return Om::Value();}extern "C" __declspec(dllexport) void omExtend(Om::Engine *engine, Om::Value *result){ engine->addModule("print", engine->makeFunction(print)); *result = Om::Value();}Having built the Om library as a DLL as well, I just get QtCreator to build the above as a DLL, linking to the stub libOm.a. I can then, in a driver program, call LoadLibrary on the DLL, find the omExtend method and call it, passing in a pointer to the engine and a value and I then have the native print function available as a module in the engine.

Equally it is possible to write an Om script that returns a value, evaluate this with the engine and add that to the modules list to have it available to other scripts.

So what I'm thinking is that I want there to be two forms of the import statement.

import print;import "path/to/print.om";import "path/to/print.om.dll";In all cases, the symbol "print" should be given to the compiler as a module symbol to be checked for with OpCode::Type::GetMd when it is encountered in the source.

Assuming the file these are in is located in C:/Projects/script.txt, and the Om::Engine has a search paths variable set up as something like "C:/Om/extend;C:/MyOmModules", the first statement should look for the following in the following order:

C:/Projects/print.omC:/Projects/print.om.dllC:/Om/extend/print.omC:/Om/extend/print.om.dllC:/MyOmModules/print.omC:/MyOmModules/print.om.dllThe second import statement should look in

C:/Projects/path/to/print.omC:/Om/extend/print.omCL/MyOmModules/print.omand the third in

C:/Projects/path/to/symbol.om.dllC:/Om/extend/symbol.om.dllCL/MyOmModules/symbol.om.dllWhen the first one is found, it is evaluated in the relevant way and added to the module list.

But when should this happen? I think perhaps the first time the module is actually used as this allows us to create some circular relationships that will otherwise be impossible. Obviously this means there could be some unexpectedly long operations occurring in unexpected places, but I don't think it would be a problem.

So in both cases, the compiler is going to have to store the argument to import in the text cache and embed the id in with the OpCode::Type::GetMd instruction, along with the id for just the symbol. So the psuedocode for the GetMd will be something like:

bool Machine::md(uint path, uint symbol, Om::Result &result){ auto m = state.modules.find(symbol); if(m == state.modules.end()) { m = loadModule(state, path); if(m == state.modules.end()) { result = Om::ValueProxy::makeError(state, stringFormat("module not found - ", state.tc.text(id)), mapToLine()); return false; } } vs.push_back(m->value); inc(state, vs.back()); return true;}Where loadModule looks something like:

pod_map::iterator loadModule(State &state, uint pathId){ pod_string path = state.tc.text(pathId); auto s = doSearch(path); Om::Value v; if(s.type == om) { v = state.engine.evaluate(s.path, Om::Engine::EvaluateType::File); if(v.type() == Om::Type::Error) return state.modules.end(); } else if(s.type == dll) { HMODULE h = LoadLibrary(s.path); FARPROC p = GetProcAddress(h, "omExtend"); // check for errors, return state.modules.end() typedef void(*Func)(Om::Engine*,Om::Value*); reinterpret_cast(p)(&state.engine, &v); } inc(state, v); return state.modules.insert(v);}or whatever.

This should then mean that, like with C and C++, you can reference both a standard installation directory (maybe stored in an OM environment variable or set by options to the driver program, as well as using a local directory structure for a current project.

Can't quite decide if this is the right approach or not so need to ponder a little more. Playing a bit of Skyrim at the moment so that is helping me unwind my mind.

## Om - postponing the worry

I've decided to just postpone worrying about cycles and memory leaks in Om for the time being. I may have to address this in the future, but for now I've discovered that there are actually so many things the user can do that cause leaks, infinite loops, stack explosions etc that I cannot efficiently detect or prevent, that the memory leak issue with a cycle is only one small issue.

So I'll put this at the back of my mind and continue on with what is otherwise a pretty solid little language.

This morning I've been playing around with building Om as a static library and linking it into a standalone test application. The static library is 900 kb in size which is nice and small, and the test application, which just loads a script and prints some output to std::cout at the moment, feels quite nice and tidy. Figuring out the PRO file settings to build the lib was easy enough. If I'm going to start developing Om in this model I'll need a better workflow but for now I'll probably keep developing Om in its test bed context, which is a command line application, then update the file in my lib directory when I hit milestones.

Obviously I'm planning to use Om in my games as a scripting backend, but I've also thought I'd like to be able to use it on a day to day basis for sort of batch and general purpose usages, so I think I'm going to put together something that allows me to write an Om script in a text file, then execute it from Windows explorer with a right-click or something, so I'll probably get working on some kind of context program for this. I come across tasks in my day to day work that I could easily benefit from being able to run a quick Om script to solve so be nice to have this working on my computer.

As well as the context to execute it, this is going to involve slowly developing an Om standard library for things like file streams and so on. Best if this is just done as I go along and I need things. Perhaps this will form another static library for now that I'll link into my desktop context program. Om was designed to be extensible in this way, so that the library can be developed independently from the core language itself. We shall see how well this works out in practice.

I think it unlikely that cycles will be accidentally created in normal use and part of me just wants to say "Don't do this" rather than start to try to develop some kind of complex garbage collection system. I don't see how any system is going to work in all cases anyway and keeping this simple and robust is my chief concern.

I'll be updating this post later. Few bits to do this morning and time is a'wasting.

## Mission Control - Om has a problem.

I realised today why JavaScript doesn't have deterministic destruction. Thought I was so smart basing Om around it.

Cycles.

If one object A references an object B and B references A, Om will never release the entities when they go out of scope. Memory leaks.

This is a well enough studied problem to assume there is no available solution. So where does this leave Om?

It is hardly worth turning it into a garbage collected language. That defeats the whole point of the language in the first place.

Equally, it is too easy to accidentally create cycles, especially with long chains of objects or lists, to just say "Don't do this in Om or you leak memory". I would lose all heart in the project then.

So we are in a quandary. Maybe have to toast an interesting project goodbye and move on.

What really hurts is whenever I go online to research this, everyone is describing the approach I have been using for the last few years as naive :)

To be fair, most of the recent work on Om has been copy-pasting code from version 3 into version 4, so in a way it is nice to have a new problem to think about. Toying with the idea of a hybrid system that does strict reference counting unless it (somehow) detects cycles then falls back on asynchronous garbage collection. I see other languages have accepted this. A real shame though. But I'm unlikely to be the first living human to find a better solution, frankly. I'm good enough at what I do to be sure of that much.

Any suggestions from the floor?

[EDIT] It is even worse than I thought, with lists.

var o = [ ];var n = [ o ];o.push(n);When I try to output the state at the end of the program now, we get an infinitely recursive loop going on until Om runs out of stack space.

This is pretty much impossible to detect. Easy enough in a silly example like this, but a general solution just doesn't exist I don't think.

I'm pretty much starting to lean towards just making it a rule that circular references can cause memory leaks or infinite loops and leaving it at that at the moment. Leaves a bad taste in my mouth but I'm not sure what else I can do.

## The Om Programming Language :)

The Om Programming Language :)

Rather a long journal entry today. I hope that someone sticks with it as I'm really getting quite excited about how my scripting langauge is starting to develop now.

I started trying to write a general overview of Om last night as I've been posting about my scripting language here for a while and never really provided such a thing, but I was quickly overwhelmed. It's really hard to write a concise overview, partly because the language has developed far more features than I realised until I took a step back but also because it is generally hard to know where to start and what order to discuss things in, given the interlated nature of language features.

Om is designed to be a lightweight scripting language that is easy to integrate into existing C++ projects, implemented entirely in standard C++ itself and provide a reasonable level of efficiency of code execution in terms of speed and resource usage. It is not trying to compete with Google V8 and similar. I wouldn't be that foolish. But while the syntax of Om is quite similar to Javascript, there are a number of differences that have motivated the developent of the language in the first place.

Firstly, there is no garbage collection in Om and it offers completely deterministic destruction of objects to enable RAII style coding - something I personally find hard to live without. Complex objects are carefully tracked by reference count and are released at the exact point their reference count reaches zero and, in the case of an Om::Type::Object, an Om::Type::Function can be set up to be called at this point.

Secondly, the syntax for using objects is slightly simpler than Javascript in that there is no need for a new keyword. The closest we come to constructors in Om are free functions that return an instance of an Om::Type::Object.

Finally, extending the language by the provision of native-side code is designed to be extremely simple. There is one Om::Function typedef, defined as:

Om::Value someFunction(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶meters);

This single type of function can be used to provide rich modules of shared native code to be accessed from the script as well as allowing the script to pass pretty much anything back to the host application.

I'm going to focus on Om::Type::Object in this post and gloss over the other details which I hope will be fairly obvious from the example code. The only thing to bear in mind is that Om is entirely dynamically typed, with variables inferring their type from what is assigned to them, carrying their type around along with their value.

Also bear in mind that Om::Type::Functions are entirely first-class entities in Om and can be assigned to variables or passed as parameters as simply as one would pass an Om::Type::Int or any other type.

So to pick a random example, here's a simple example, entirely in script for now, of how one might implement a Person class.

Om::Type::Objects are declared with the syntax { }, and are essentially a string-value mapping with some special properties discussed later on. Unlike Javascript, Om::Type::List, declared with [ ] syntax, is a completely separate type from Om::Type::Object and functions as a resizable, heterogenus array of values.

import print;var makePerson = func(name, age){ return { name = name; age = age; };};var people = [ makePerson("Paul", 42), makePerson("Eddie", 23), makePerson("Jill", 78) ];for(var p: people){ print(p.name, " is ", p.age, " years old");}Okay, so let's now think in a more OO way and make the description a method on a person instead.

import print;var makePerson = func(name, age){ return { name = name; age = age; describe = func { print(this.name, " is ", this.age, " years old."); }; };};var people = [ makePerson("Paul", 42), makePerson("Eddie", 23), makePerson("Jill", 78) ];for(var p: people){ p.describe();}Om supports prototype-based inheritence in a very simple fashion.

import print;var proto ={ hair = "brown";};var makePerson = func(name, age){ return { prototype = proto; name = name; age = age; describe = func { print(this.name, " is ", this.age, " years old and has ", this.hair, " coloured hair."); }; };};var people = [ makePerson("Paul", 42), makePerson("Eddie", 23), makePerson("Jill", 78) ];people[1].hair = "blonde";for(var p: people){ p.describe();}Note that all instances of the person object now share the "brown" value when we are reading, but when we write the "blonde" value to Eddie before the output loop, Eddie then has his own "hair" property which overrides the one in the prototype. This is a very simple system to implement but extremely flexible.

Note the this.value syntax has to be explicit in Om. The reason is that the function has no idea it is a member function as it is being compiled. Indeed it is quite possible to call the same function once as a method on an object and then again as a free function.

import print;var proto ={ hair = "brown";};var makePerson = func(name, age){ return { prototype = proto; name = name; age = age; describe = func { print(this.name, " is ", this.age, " years old and has ", this.hair, " coloured hair."); }; };};var people = [ makePerson("Paul", 42), makePerson("Eddie", 23), makePerson("Jill", 78) ];people[1].hair = "blonde";var speak = func(word){ if(this.type == "object") { print(this.name, " says ", word); } else { print(word, " is generally spoken :)"); }};speak("hello");people[0].prototype.speak = speak;for(var p: people){ p.describe(); p.speak("hello");}This outputs:

Om: hello is generally spoken :)Om: Paul is 42 years old and has brown coloured hair.Om: Paul says helloOm: Eddie is 23 years old and has blonde coloured hair.Om: Eddie says helloOm: Jill is 78 years old and has brown coloured hair.Om: Jill says helloLastly for now, if we assign an Om::Type::Function taking no parameters to an objects destructor property, this will be called when the object is destroyed.

import print;var proto ={ hair = "brown"; destructor = func { print("goodbye from the prototype"); };};var makePerson = func(name, age){ return { prototype = proto; name = name; age = age; describe = func { print(this.name, " is ", this.age, " years old and has ", this.hair, " coloured hair."); }; destructor = func { print("goodbye from ", this.name); }; };};var people = [ makePerson("Paul", 42), makePerson("Eddie", 23), makePerson("Jill", 78) ];people[1].hair = "blonde";var speak = func(word){ if(this.type == "object") { print(this.name, " says ", word); } else { print(word, " is generally spoken :)"); }};speak("hello");people[0].prototype.speak = speak;for(var p: people){ p.describe(); p.speak("hello");}people[2] = null;print("end of program");The above program will output the following:

Om: hello is generally spoken :)Om: Paul is 42 years old and has brown coloured hair.Om: Paul says helloOm: Eddie is 23 years old and has blonde coloured hair.Om: Eddie says helloOm: Jill is 78 years old and has brown coloured hair.Om: Jill says helloOm: goodbye from JillOm: end of programOm: goodbye from PaulOm: goodbye from EddieOm: goodbye from the prototypeNote how assigning null to people[2] destroys Jill at that point, since that causes Jill's refernece count to drop to zero.

Om::Type::Object has a built in members property that returns an Om::Type::List of the names of its members. Om::Type::Object supports lookup by both the dot operator and via dynamic text and the subscript operator so you can use these together to implement a form of reflection.

import print;var o ={ name = "Paul"; age = 41; car = "Rover";};for(var m: o.members){ print(m, " = ", o[m]);}Using the subscript operator is far less efficient than the dot operator so should only be employed when the name of the property is not known. Using the dot operator in the VM equates to doing a binary search for an unsigned integer in a sorted array whereas using the subscript operator requires actual text comparisons at runtime.

Final note on Om::Type::Object is that, like Om::Type::List and Om::Type::String, default copy is by reference.

import print;var o = { name = "Paul"; };var c = o;o.name = "Eddie";print(c.name);This will output "Eddie", not "Paul". However, all types support the clone() method so we can explicitly perform a deep copy here instead.

import print;var o = { name = "Paul"; };var c = o.clone();o.name = "Eddie";print(c.name);This will output "Paul" as expected.

clone() is supported by every type although does nothing in the case of the value types. Evem constants can use the dot operator in Om and the following is all perfectly legal and well-defined:

import print;print(10.type); // prints "int"var n = 10.clone(); // equivalent to var n = 10 :)var s = "hello".length; /// s = 5print({ name = "Paul"; age = 42; }.members.length); // prints 2print({ name = "Eddie"; age = 23; }.members.length.type); // prints "int"Now we have a bit of an overview of the langauge itself, let's take a look at how the C++ API is used to integrate Om scripting into an existing C++ application.

The two key classes exposed by the API are Om::Engine and Om::Value.

#incude "om/OmEngine.h"int main(){ Om::Engine engine; Om::Value v = engine.evaluate("return (1 + 2) * 3;", Om::Engine::EvaluateType::String); if(v.type() == Om::Type::Error) { std::cerr << "Error: " << v.toError().text << "\n"; return -1; } std::cout << "v is " << v.toInt() << "\n"; // will print "v is 9"}When reference types like Om::Type::String or Om::Type::Object are stored in Om::Values, the Om::Value takes care of keeping track of reference counts and so on, seamlessly from the user's point of view.

Om::Value can directly construct value types, but the constructors are marked explicit to avoid accidental conversions.

void f(){ Om::Value i(123); // Om::Type::Int Om::Value f(12.34f); // Om::Type::Float Om::Value b(true); // Om::Type::Bool}Reference types have to be generated from the Om::Engine.

void f(Om::Engine &engine){ Om::Value s = engine.makeString("hello"); Om::Value o = engine,makeObject(); o.setProperty("name", engine.makeString("Paul")); o.setProperty("age", Om::Value(42));}If we construct an Om::Value with an Om::Type::Function, it is compiled and stored, but not executed until we choose to later on.

int main(){ Om::Engine engine; Om::Value f = engine.evaluate("return func(a, b){ return a + b; };", Om::Engine::EvaluateType::String); if(f != Om::Type::Error) { Om::Value r = engine.execute(f, Om::Value(), { Om::Value(2), Om::Value(3) }); std::cout << "result " << r.toInt() << "\n"; // prints "result 5" }}In more detail, the execute method is Om::Value Om::Engine::execute(const Om::Value &function, const Om::Value &object, const Om::ValueList ¶meters), allowing you to pass in an optional this-object and a parameter list to the function.

Om provides a simple but flexible mechanism for writing and reusing modular code. There is no preprocessing or file inclusion in Om. The compiler is only ever looking at exactly one source file (or string) at a time.

Om::Engine provides the addModule(const Om::String &id, const Om::Value &value) method. Any type of Om::Value can be added to the modules list and then imported into another script.

For example, all the previous examples begin with import print; As Om is entirely unaware of the context in which it is running, I have set up a simple native-side function to print values to std::cout in the test bed. C++-side, this looks like this:

void out(std::ostream &os, const Om::Value &value){ if(value.type() == Om::Type::List) { os << "["; for(int i = 0; i < value.count(); ++i) { os << " "; out(os, value.property(i)); } os << " ] "; } else if(value.type() == Om::Type::Data) { os << value.toData(); } else { os << value.toString(); // toString() provides a text representation of most types }}Om::Value printFunc(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶ms){ std::cout << "Om: "; for(auto p: params) { out(std::cout, p); } std::cout << "\n"; return Om::Value();}int main(){ Om::Engine engine; Om::Engine::OutputFlags flags(Om::Engine::OutputFlag::HideDefinedStrings); engine.addModule("print", engine.makeFunction(printFunc)); engine.evaluate("sample.txt", Om::Engine::EvaluateType::File);}The import keyword is scope aware and only introduces the symbol into the import's scope.

var n = 20;if(n > 10){ import print; print(n);}print("end"); // compile error - print symbol not foundThe actual module lookup is peformed at runtime so it is quite possible to compile a function that references modules that habe not yet been added to the engine, as long as they are added before the function is executed. As a result it is possible to create two-way relationships between modules without issues with circular dependancy.

In the print example, the module is simply an Om::Type::Function.

Let's look at a slightly more complex example using the script to define a module instead - a modular reworking of the Person examples above. Firstly we define the Person module in a normal script file:

import print;return{ base = { hair = "brown"; }; make = func(name, age) { return { prototype = this.base; name = name; age = age; describe = func { print(this.name, " is ", this.age, " years old and has ", this.hair, " coloured hair."); }; destructor = func { print("goodbye from ", this.name); }; }; };};Note we are returning an Om::Type::Object here, which gives as a place to store our prototype instance as well as the make function. The make function is the Om equivalent of a constructor here.

In the C++ setup, we can simply do:

int main(){ Om::Engine engine; Om::Engine::OutputFlags flags(Om::Engine::OutputFlag::HideDefinedStrings); engine.addModule("print", engine.makeFunction(printFunc)); engine.addModule("person", engine.evaluate("person.txt", Om::Engine::EvaluateType::File)); engine.evaluate("sample.txt", Om::Engine::EvaluateType::File);}Note in the real world, one would evaluate person.txt into an Om::Value so one could check for compiler errors. The evaluate method will return an Om::Type::Error rather than the object if errors are thrown up by the compiler.

We can now use this module in sample.txt as follows:

import person;var people = [ person.make("Paul", 42), person.make("Eddie", 23), person.make("Jill", 78) ];people[1].hair = "blonde";for(var p: people){ p.describe();}Note that there is no need to import print; into sample.txt now as it is not used directly.

It is also possible to extend Om with types implemented in native code. For example, because Om::Type::Strings are immutable, it is not optimal to concatenate lots of strings together in the script as it produces a great deal of temporary values. Much as in other langauges, what we really need is a stringBuilder that can do this kind of concatenation more efficiently. I'll now describe how to create such a facility in native C++ to make available to the scripts.

A special Om::Type provided for use in the C++ API is Om::Type::Data. This allows the user to store a void* pointer in an Om::Value. We can then access this data from the object instance in the usual way and use it to implement custom object types that interface with C++ code.

Our string builder is going to be based on std::ostringstream, so first of all we can define a representation in C++.

class Rep{public: Rep(){ } Rep(const std::string &s){ os << s; } std::ostringstream os;};Next, we need to provide a function that the script can call to create an instance of the string builder. In this function, we assign the properties of the string builder, using other native functions. I didn't want to provide a void* constructor for Om::Value as that could potentially lead to some dangerous conversions, even with an explicit constructor, so instead there is a static fromData() method instead to make this even more explicit.

Om::Value makeObject(Om::Engine &engine, const std::string &init){ Om::Value o = engine.makeObject(); o.setProperty("data", Om::Value::fromData(new Rep(init))); return o;}Now we can add the methods the script needs to be able to call on the string builder, specifically add() and value(). Om::Value provides a convenience template function, toUserType>(), to make it slightly more concise to cast back the pointer.

Om::Value add(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶ms){ if(params.count() != 1 || params[0].type() != Om::Type::String) return engine.makeError("incorrect parameters"); Rep *rep = object.property("data").toUserType(); rep->os << params[0].toString().c_str(); return Om::Value();}Om::Value value(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶ms){ Rep *rep = object.property("data").toUserType(); return engine.makeString(rep->os.str().c_str());}Om::Value makeObject(Om::Engine &engine, const std::string &init){ Om::Value o = engine.makeObject(); o.setProperty("data", Om::Value::fromData(new Rep(init))); o.setProperty("add", engine.makeFunction(add)); o.setProperty("value", engine.makeFunction(value)); return o;}For the final step, we need to define a destructor for the object so we can clean up the allocated memory. Note we are really just using existing features of the language here rather than having to implement any special functionality.

Om::Value destroy(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶ms){ delete object.property("data").toUserType(); return Om::Value();}Om::Value makeObject(Om::Engine &engine, const std::string &init){ Om::Value o = engine.makeObject(); o.setProperty("data", Om::Value::fromData(new Rep(init))); o.setProperty("add", engine.makeFunction(add)); o.setProperty("value", engine.makeFunction(value)); o.setProperty("destructor", engine.makeFunction(destroy)); return o;}Caution needs to be taken here though. Much like C++'s rule of three (or five), if you are providing a custom destructor and using the Om::Type::Data type, you almost certainly also need to overload clone() or terrible things will happen. The built-in clone() will do a by-value copy of the "data" property, meaning you end up with a double-delete if you clone the object in the script.

Since Om::Type::Objects can override any of the built-in methods with their own properties, we can simply add:

Om::Value clone(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶ms){ Rep *rep = object.property("data").toUserType(); return makeObject(engine, rep->os.str());}Om::Value makeObject(Om::Engine &engine, const std::string &init){ Om::Value o = engine.makeObject(); o.setProperty("data", Om::Value::fromData(new Rep(init))); o.setProperty("add", engine.makeFunction(add)); o.setProperty("value", engine.makeFunction(value)); o.setProperty("destructor", engine.makeFunction(destroy)); o.setProperty("clone", engine.makeFunction(clone)); // will override the built-in clone() for this object specifically return o;}Now we are safe to clone() the object inside the script.

We can define all of the above in a cpp file and provide a simple interface function in the header.

Om::Value omStringBuilder(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶ms){ return makeObject(engine, "");}Then business as usual setting up the module in main().

int main(){ Om::Engine engine; engine.addModule("print", engine.makeFunction(printFunc)); engine.addModule("stringBuilder", engine.makeFunction(omStringBuilder)); engine.evaluate("sample.txt", Om::Engine::EvaluateType::File);}Now off we can go into the script and use our custom type:

import print;import stringBuilder;var s = stringBuilder();s.add("one, ");s.add("two ");s.add("and three.");print(s.value()); // prints "one, two and three."For types that should not be cloned, for eaxmple a wrapper around a file stream or similar, one can instead provide a clone implementation in C++ like this:

Om::Value clone(Om::Engine &engine, const Om::Value &object, const Om::ValueList ¶ms){ return engine.makeError("unable to clone object");}Then, in the script, if an attempt is made to clone the object, a runtime error will be generated and the script will exit. The destructors will still be called though so the memory clean up will still take place.

A couple of other snippets to mention. Script functions can be defined to take a variable number of parameters using the following syntax:

var f = func(a, b, c...){};The ellipses must be attached to the right-most parameter and when calling the function, you must provide values for the normal parameters. Any additional parameters (more than two in this example) are then accessible using the 'c' symbol which will be of Om::Type::List, containing the additinoal parameters.

Better explained with exmaple code here.

import print;var f = func(a, b, c...){ print(a); print(b); print(c);}f(1, 2); // prints 1, 2, [ ] (an empty list)f(1, 2, 3, 4); // prints 1, 2, [ 3, 4 ]var x = func(p...){ for(var i: p) print(i);};x(); // prints nothingx(1, 2, 3, 4); // prints 1 2 3 4The C++ style ternary operator is supported in Om, as well as short-circuit evalution of and and or. These are of particular use in a dynamically typed language as they can be used to concisely avoid evaluating expressions that would generate a runtime error.

var f = func(a);{ print(a.type == "object" ? a.name : "no name"); if(a.type == "object" and a.name == "Paul") doStuff();}In both cases the dot operator would throw a runtime error if the variable was not an object, so avoiding evaluating these is useful.

I think that is enough information for now. If anyone has made it through, thank you for your perserverence and I'm keen to answer any queries anyone may have.

[EDIT] I've just finished implementing Om's version of the switch statement, and also more general break statements for early-exiting out of loops, both of which work much as you would expect. Unlike C an C++, Om's switch is never turned into a jump table so while it isn't quite the efficient beast we know and love, it also doesn't have the C++ switch limitations - the switch expression and even the case expressions can be of absolutely any expression type. Fallthrough works the same as in C++ though.

Next Time: If anyone is interested, I'll maybe start to lift the lid on how all of this is actually implemented. Here, as a sneak peak, is the current complete OpCode set for the virtual machine, upon which everything above is based. A surprisingly small set of codes I think.

namespace OpCode{enum class Type{ Call, Ret, Push, Pop, PopN, Peek, Jmp, JmpT, JmpF, GetLc, PutLc, GetMb, PutMb, GetSc, PutSc, GetNl, PutNl, GetMd, Math, Cmp, Una, Bool, MkEnt, AddCh, FeChk, FeGet, Inc, Dec, Invalid};const char *toString(Type type);const char *parameters(Type type);enum class Math{ Add, Sub, Mul, Div, Invalid};const char *toString(Math type);enum class Cmp{ Eq, Neq, Lt, LtEq, Gt, GtEq, Invalid};const char *toString(Cmp type);enum class Una{ Neg, Not, Invalid};const char *toString(Una type);template const char *text(uint id){ return toString(static_cast(id)); }}

## Om script - template special :)

Following on from my last entry where i "discovered" you could use templates to provide a common interface to two unrelated classes, I've now implemented this technique in Om, my scripting compiler and virtual machine and thought I'd just write this up a bit.

Om has various [font='courier new']Om::Value[/font] types, including [font='courier new']Om::String[/font] and [font='courier new']Om::List[/font]. Strings are obviously strings of characters and lists are lists of [font='courier new']Om::Values[/font], and can themselves contain strings and lists and so on. Both are reference-counted entities under the hood.

Their behaviour and use are very different, but the operations you can perform on them have a great deal of overlap. For example, both have a built-in [font='courier new']length[/font] property that returns either the length of the string or the number of values in the list as an [font='courier new']Om::Int[/font]. You can apply [font='courier new']operator[][/font] with an [font='courier new']Om::Int[/font] argument which, for the list, returns the value at that index but for the string returns an [font='courier new']Om::String[/font] containing the single character at that index (char types are not supported by this language).

In the previous incarnation of this project, I wrote all of this twice, once for the list and once for the string. It wasn't that bad, but it always felt like there should be a way to combine the overlap into one set of code.

So in this latest incarnation, using template specialization, that is what I have been able to do. First, these are a rough-out of the [font='courier new']Entity[/font] classes that are used to store the data under the hood. Note that for the purposes of this topic, the fact that both inherit from [font='courier new']Entity[/font] is irrelevant and all of this would work equally well with completely unrelated classes.

namespace Om{class StringEntity : public Entity{public: pod_string text; // pod_string is an internally used string class};class ListEntity : public Entity{public: pod_vector elements; // pod_vector an internal vector class, TypedValue is explained below};}[font='courier new']TypedValue[/font] is a POD-class used to store values internally that has to have its reference-counting manually incremented and decremented. From a user perspective, the [font='courier new']Om::Value[/font] class is used publically, which looks after this automaticaly. But internally we often need to, for example, remove a [font='courier new']TypedValue[/font] from the stack, use it in a sort of "expiring" state, then decrement it when we are done.

[font='courier new']TypedValue[/font] is defined partially like this:

class TypedValue{public: TypedValue() : t(Om::Type::Null) { *reinterpret_cast(d) = 0; } explicit TypedValue(Om::Type type, int v) : t(type) { *reinterpret_cast(d) = v; } explicit TypedValue(Om::Type type, uint v) : t(type) { *reinterpret_cast(d) = v; } explicit TypedValue(Om::Type type, float v) : t(type) { *reinterpret_cast(d) = v; } explicit TypedValue(Om::Type type, bool v) : t(type) { *reinterpret_cast(d) = v ? 1 : 0; } TypedValue(Om::Type type, const char *data) : t(type) { *reinterpret_cast(d) = data ? *reinterpret_cast(data) : 0; } Om::Type realType() const { return t; } Om::Type userType() const { return t; } const char *data() const { return d; } int toInt() const { return *(reinterpret_cast(d)); } uint toUint() const { return *(reinterpret_cast(d)); } float toFloat() const { return *(reinterpret_cast(d)); } bool toBool() const { return *(reinterpret_cast(d)) ? true : false; }};So you need to use it carefully and ensure that you haev checked the type before you call the [font='courier new']toWhatever()[/font] methods and so on, unlike [font='courier new']Om::Value[/font] that the user uses, which is checked.

Looking at the [font='courier new']length[/font] property first, when the compiler builds a dot-syntax node, it emits instructions to push the target onto the stack, the text ID of the right-hand-side (stored in a reference-counted text cache) and then either a [font='courier new']GetMb[/font] or [font='courier new']PutMb[/font] instruction depending on if we are reading or writing.

So if we have:
var o = "hello";out o.length;We end up with something like:

Abstract Syntax Tree:
block var [o] string [hello] out dot [length] symbol [o] out newlineGenerated Virtual Machine Code
 0: MkEnt string 4 6: GetLc 211: GetMb 216: Out17: OutNl18: PopN 223: RetHere, [font='courier new']length[/font] has an id of 2 in the text cache, so when [font='courier new']GetMb[/font] is called, the left-side of the dot is on top of the stack, and the [font='courier new']length[/font] text id is encoded into the [font='courier new']GetMb[/font] instruction.

To unify the code to be able to have a template function work on either a [font='courier new']StringEntity[/font] or a [font='courier new']ListEntity[/font], we first need a common interface for the two types. I took the generic name [font='courier new']Sequence[/font] to use to represent this and wrote the following header:

template class Sequence { };template<> class Sequence{public: Sequence(StringEntity &e) : e(e) { } int length() const { return static_cast(e.text.length()); } TypedValue get(State &state, int index) const { TypedValue v(Om::Type::String, state.allocate()); StringEntity &c = state.entity(v.toUint()); c.text.append(e.text[index]); return v; } void set(State &state, int index, const TypedValue &v, Om::Value &result) { pod_string c = state.entity(v.toUint()).text; if(c.length() > 1) { result = Om::ValueProxy::makeError(state, stringFormat("cannot assign multiple characters via subsscript - ", c), 0); return; } e.text[index] = c[0]; }private: StringEntity &e;};template<> class Sequence{public: Sequence(ListEntity &e) : e(e) { } int length() const { return static_cast(e.elements.size()); } TypedValue get(State &state, int index) { return e.elements[index]; } void set(State &state, int index, const TypedValue &v, Om::Value &result) { if(!dec(state, e.elements[index], result)) return; e.elements[index] = v; inc(state, v); }private: ListEntity &e;};The [font='courier new']length[/font], [font='courier new']get[/font] and [font='courier new']set[/font] methods are what we need to implement the current operations and are implemented very differently as you can see. For example, the list needs to increment the reference count of its value when adding it to the list, whereas the string has to construct a new [font='courier new']StringEntity[/font] to return the character at the given index.

But having these written, we can now express operations in a function templated on the entity type.

In the [font='courier new']Machine[/font] class, executing a [font='courier new']GetMb[/font] or [font='courier new']SetMb[/font] instruction is handled by first looking for the built-in methods.

InternalMethod findInternalMethod(Om::Type type, uint id){ TRACE; InternalMethod m; switch(type) { case Om::Type::String: m = sequence_method(id); break; case Om::Type::List: m = sequence_method(id); break; default: break; } if(m.valid()) return m; // snip, handle other things return InternalMethod();}So here, if we have a string or a list, the first thing we do is call [font='courier new']sequence_method[/font], templated on the correct type and the compiler generates a version of [font='courier new']sequence_method[/font] for each type for us. We only have to write it once:

template void sequence_length(State &state, const TypedValue &v, Stack &vs, Om::Value &result){ TRACE; vs.push_back(TypedValue(Om::Type::Int, static_cast(Sequence(state.entity(v.toUint())).length())));}template InternalMethod sequence_method(uint id){ TRACE; switch(id) { case DefinedStrings::Length: return Properties::sequence_length; default: break; } return InternalMethod();}Note how the [font='courier new']Sequence[/font] interface is then used to extract the [font='courier new']length[/font] from the entity.

Similarly, with [font='courier new']operator[][/font] on a string or a list, the compiler will push the target onto the stack, then the expression inside the square brackets, then call [font='courier new']GetSc[/font] or [font='courier new']PutSc[/font]. We translate this into a call to the template method [font='courier new']rangeOp[/font].

bool Machine::sc(AccessType type, Om::Value &result){ TRACE; TypedValue v = vs.pop_back(); TypedValue o = vs.pop_back(); TypedValueGuard guard({ v, o }); // snip, handle other uses switch(o.realType()) { case Om::Type::String: rangeOp(state, type, vs, o, v, result); break; case Om::Type::List: rangeOp(state, type, vs, o, v, result); break; default: result = Om::ValueProxy::makeError(state, stringFormat("subscript applied to invalid type - ", Om::typeToString(o.userType())), mapToLine()); } return guard.release(state, result);}The [font='courier new']rangeOp[/font] method looks like this:

template void rangeOp(State &state, Machine::AccessType type, Stack &vs, const TypedValue &o, const TypedValue &v, Om::Value &result){ Sequence seq(state.entity(o.toUint())); process(state, type, seq, v, vs, result);}We construct a local [font='courier new']Sequence[/font] so we can pass it by non-const reference into the guts of the system, then we call a template method to implement the reading or writing using [font='courier new']operator[][/font], all expressed via the [font='courier new']Sequence[/font] interface instead of [font='courier new']StringEntity[/font] or [font='courier new']ListEntity[/font].

template void process(State &state, Machine::AccessType type, Sequence &s, const TypedValue &v, Stack &vs, Om::Value &result){ if(v.realType() != Om::Type::Int) { result = Om::ValueProxy::makeError(state, stringFormat("subscript expression of invalid type - ", Om::typeToString(v.userType())), 0); return; } int index = v.toUint(); if(index < 0 || index >= s.length()) { result = Om::ValueProxy::makeError(state, stringFormat("subscript expression out of range - ", index), 0); return; } if(type == Machine::AccessType::Read) { vs.push_back(s.get(state, index)); inc(state, vs.back()); } else if(type == Machine::AccessType::Write) { TypedValue c = vs.pop_back(); s.set(state, index, c, result); dec(state, c, result); vs.push_back(v); }}Again, the two versions the C++ compiler will generate here are quite different, but we have been able to express it once in the same way.

As I now add more built-in methods and properties to strings and lists, I can now easily decide whether each new addition shoudl be just for a string, just for a list or for both and implement it in the relevant place, just once, for it to work as I wish. I can also extend the [font='courier new']Sequence[/font] interface if I need to support any other operations, for example removing items, clearing items out etc.

In the unlikely event that some other kind of sequence container is added to the language, again I can just specialise Sequence for that container and it will all just snap into place.

In summary, I'm sure this is well studied and knowm but I had never before thought about using template specialisation to create a common interface for unrelated classes and, for this particular situation where code size is pretty much not a consideration but efficiency and maintainability are key, it has turned out to be a success.

There is a certain amount of boiler-plate needed at the start but once this is done, this approach seems to extend very nicely and prevents a great deal of duplication of code that, previously, had felt icky and wrong to me.

So there you go. Slightly longer entry this time :) In the unlikely event that anyone has made it this far and is still reading, I'd like to wish you an excellent Christmas if you are into that sort of thing, and all the best for the new year. Thanks for stopping by.

## Compile time "polymorphism"

So here is something I just discovered I can do in C++ that I didn't know about.

template class wrapper{public: wrapper(const T &t){ } int length() const { return -1; }};template<> class wrapper{public: wrapper(const StringEntity &e) : e(e) { } int length() const { return e.text.length(); } StringEntity e;};template<> class wrapper{public: wrapper(const ListEntity &e) : e(e) { } int length() const { return e.elements.size(); } ListEntity e;};template void process(const wrapper &w){ std::cout << "process: " << w.length() << "\n";}int main(){ StringEntity s("hello"); ListEntity l; wrapper se(s); wrapper le(l); process(se); process(le);}Compile time polymorphism, but of completely unrelated types and no use of virtual methods.

My scripting language has both String and List types which are completely unrelated, but have a lot of overlap in terms of the operations the VM needs to do on them.

StringEntity and ListEntity do happen to both inherit a base class, but that is irrelevant for the subject here - could add a specialisation for any other class here.

This approach should mean I can define a common interface without affecting any of the other Entity classes, and use template functions to do the processing.

Call today's entry a snippet since I'm actually at work :)

## Om scripting revisited

Howdy.

Been quiet for a while but have been getting back into my scripting language compiler/virtual machine. No particular reason, just has grabbed my attention for the last few weeks.

Today I started adding support for range-syntax on strings and lists:
var s = "hello";var m = [ "one", 2, 3.0 ]; out s[1...3];out m[...func { return 2; }() ];

The ellipse syntax in a subscrpt means you can have two optional expressions to mark the low and high indices to slice at. If omitted, they are 0 and count - 1 implicitly. You can even do s[...] as shorthand for s if you want :)

I realised though that I now have two systems, one for single subscript s[20]; and one for ranged, where these could be combined into a single system, where a single subscript is represented by a range with the low and high set to the same value under the hood. Since I'm aiming for small and clean implementation over high performance, I've now ripped out all of the subscript handling code and going to start again with this in mind.

However, something I also want to do is switch Om over to work with UNICODE so I'm wondering if now would be a good time to tackle this, while I have a load of code ripped out that won't therefore need to be rewritten. I'm not quite sure what is going to be involved in UNICODE support but I presume the less code I have to worry about, the better.

Ho hum. Regarding Om, I'll just briefly recap its main purposes.

Om is a duck-typed scripting language that draws on JavaScript but provides an entirely deterministic approach to object lifetimes via careful reference tracking. There is no (and no need for) garbage collection since objects are released in a completely deterministic way.

One of the types, Om::Object, which works like a string-value map, has a couple of special properties. You can assign a parameterless function to a property called destructor which will then be called when the Om::Object is released.

Om::Objects also support prototype-based inheritance. If the prototype property of an Om::Object points to another Om::Object, that forms a prototype chain. When properties are being read from, the VM searches up the prototype chain for the value, checking to ensure no infinite prototype look has been accidentally created. When writing, the property is always set on the local object. This is how I believe JavaScript inheritance works. It is actually very clean and simple to implement.

Unlike JS, Om::List is a separate type to Om::Object. Lists are declared like [ 1, 2, 3 ]; and so on.

This is the full list of types:
namespace Om {enum class Type { Null, Int, Float, Bool, String, Function, List, Object, Error };

The first four are value-types and the rest are reference types, so the value that is stored is a unsigned int id that points to an Entity inside the machine.

Wish me luck on my UNICODE experiments. wctombs or something isn't it?

## Bow and Arrow beginnings [updated]

I've been working a little bit on the view mode with the bow this week. Not spent a lot of time on it but Saturday morning seems to be a good time to update my journal since I have an hour to work on the game and can't really get into anything else properly, so we are only about half-way through this.

This is the camera view with the bow attached to the hand. The bow is just attached same as on my previous entry and there is a new animation to switch into when in this mode. Here it is set up in Charm, the model and animation editor.

In motion, we apply the up/down rotation from the view to the Chest rather than the Head of the character now, which moves the arm as well as the head, meaning the bow continues to point where you look.

I should say I'm entirely unhappy with the quality of the graphics here. Need a much more detailed hand for example. I'm toying with the idea of using a different model for the view mode and normal mode, although that will present difficulties of its own so it may just be we have to suck it up and build a higher-resolution model for the player. The current model is very low poly, which is fine from a distance, but this was always just a stop-gap model anyway and not the final for any game.

I need to make a decision about the arrow. Do I attach it to the other hand, and move it into position using the draw animation, then use this to inform the initial transform of the arrow when it is "fired", or do I keep the arrow completely separate and just ensure by tweaking the bow animation that the arrow rests in the right place when being drawn back?

I can see pros and cons to both approaches really. One thing that is nice is that, either way, it will be trivial to swap out the bow and the arrow meshes for other ones, so once this is working it will be easy to have a sort of Thief/Thief 2 style bow and arrow system with different bows and arrows having different abilities and powers.

It is all a bit fiddly though. I'm going to live with the crappy graphics for now until I have the actual system working the way I want. I don't think my attachment system is working quite right and need to iron out some issues there - seems the initial transform I apply to the bow in object space is somehow causing some drift in the position when in aiming mode which must be a simple enough issue to find and fix.

Sure it will all come together in the end, and it will be very satisfying to fire that first arrow into the world.

[UPDATE]

Thought it wasn't worth a new entry for this, but I've added an arrow mesh now. Threw a very quick arrow model together in Charm.

I decided to go down the route of attaching the arrow to the other hand, so it can be guided into place using the animation. Took some fiddling around to get it right but seems to work okay.

The video below shows a special external view of the animation first, then switches to what you would actually see in-game, with the camera moving to the over-the-shoulder view.

It is actually two animations, one moving from idle to the bow drawn position, then another animation all in the bow drawn position with a tiny amount of breathing added. This is the further modified by applying a portion of the up/down rotation of the camera to the Chest bone of the player.

To get this working, I had to add a facility to my animation system to set an animation to looping or non-looping. They were all looping before, which was wrong here as it was switching briefly back to idle then back into bow drawn when the animation event fired on the EnterBowDrawn animation.

It was quite scary how little I could remember about how that code worked and was quite good to revisit it. Have used it extensively for a year or two, but not actually opened the source files in almost as long :)

So the theory is now that when the player presses the Fire key, we take the current matrix being applied to the player's arrow and pass this to a system that spawns an arrow to be fired off into the world.

This matrix should generate both the start transform of the new arrow as well as its movement vector, since we know the arrow is pointing straight down in local space, so transforming (0, -1, 0) by the matrix should give the direction vector for the arrow as well. You can just see the green line I've added to the below.

Here we are taking the transform and using it to generate a vector from the tail end of the arrow ((0, 0, 0) in local space of the arrow model) and extending it out via a directional vector.

void Pc::prepareScene(SceneParams ¶ms, float blend){ KeyFrame key = machine.generateKeyFrame(blend); // snip Matrix am = data.animation.skeleton.placementTransform(data.animation.skeleton.index("Right Hand"), identityMatrix(), world); arrow->setTransform(am); Vec3 d = transformNormal(Vec3(0, -1, 0), am); Vec3 p = transformCoord(Vec3(0, 0, 0), am); debugAddLine(p, p + (d * 4), makeColor(0, 255, 0));}We will on the same frame just hide the player arrow and we should be ready to go.

Need to design up some kind of inter-system communication for this that will extend to other things. Not sure on the details of this yet as I need to think about what else it is likely to be used for.

Thanks for stopping by as usual and hope this was of interest.

## View Mode implementation and wittering

Now I have a chase camera working that can be assigned to move to different targets, I've been working on a "view mode" for the player. Holding down the right-mouse button moves the camera to a shoulder-view mode so the player can look around.

Here I have put the camera in a position right behind the head, so that you can see in more detail that the orientation of the head changes to match the direction you are looking. You'll see this first in the video later, then it switches to the real shoulder-view mode which is just above the right shoulder but the head animation is less obvious.

To implement the head animation, I now generate my animation [font='courier new']KeyFrame[/font] inside the [font='courier new']PcStateMachine[/font].

void Pc::prepareScene(SceneParams ¶ms, float blend){ KeyFrame key = machine.generateKeyFrame(blend); data.animation.skeleton.setKeyFrame(key); // snip}We can then generate the key frame from the animation as usual, but apply further modifications to it before we return it.

KeyFrame PcStateMachine::generateKeyFrame(float blend){ KeyFrame key = data.animation.controller.keyFrame(blend, data.animation.map); if(curr == PcState::Looking) { key.transforms[data.animation.skeleton.index("Neck")].rotation *= axisRotationQuaternion(Vec3(-data.ang.value(blend).y, 0, 0)); } return key;}We can save time by looking up the "Neck" index ahead of time. Bone names are set in Charm, the model editor, and are stored as part of the binary model format to avoid working directly with numeric indicies. Here we just apply an extra bit of rotation to the neck based on the accumulated angle we are looking in, which is stored in [font='courier new']PcData::ang[/font], a [font='courier new']BlendVec2[/font].

So we can do whatever we want to here, depending on the current internal state of the [font='courier new']PcStateMachine[/font] and [font='courier new']PcData[/font].

The player rotates horizontally to match the angle you are looking at, and because the view down is limited, there is no need for a foot animation to show this turning action. When you release right button, the camera pulls back out and you are facing whatever direction you last looked in.

void PcStateMachine::updateLooking(GameState &state, float delta){ Vec2 a = data.ang.value(); a.x += pi(); data.rot = FadeValue(axisRotationQuaternion(Vec3(0, a.x, 0)));}As above, [font='courier new']PcData::ang[/font] is the accumulator for the angle the camera is looking in and [font='courier new']PcData::rot[/font] is a quaternion representing the direction the actual player is currently facing, so when looking, changing one updates the other.

Here is all of this in motion.

While this is a nice feature on its own for looking around in the game, it is also prepartion for the bow aiming mode I plan to tackle soon. When the player is holding a bow, view mode will include moving into a bow-drawn animation, and the arm will move with the head to point the bow at whatever point is looked at. Hopefully the position of the camera will mean I can get away with just a partial animation for aiming the bow, and the superimposing of an arrow mesh over the top. We shall see how this works out.

There are some issues at the moment with the camera intersecting the player model, and indeed have always been issues with the camera intersecting the level geometry that I haven't quite decided how to tackle yet. One approach might be to just model the camera as a sphere, and add the minimum separation vector for the sphere to the final camera position each frame but that could lead to some nasty jumps.

I don't really want to have to apply full path-finding logic to the camera. A possible solution might be to detect when the camera is jumping beyond a certain threshold and then apply a very fast fade-out/fade-in effect to cover the jump. Need to just try a few different approaches and see how they look.

Thanks, as always, for stopping by and I hope this was of interest to someone.

## Rag Doll is back

Just a quick post to say I've re-integrated my rag doll system back into the current iteration of the game. This is based on making a set of Bullet shapes from the current skeleton configuration on the fly, then handing control of these over to Bullet.

New feature here is that the new chase camera system then continues to follow a particular shape seamlessly when you "die".

The first half of the video below is as the player will see things, then I run through some death sequences again with some debug drawing on, so you can see the relevant shapes. While the player is alive, the large capsule is the kinematic shape that is handled manualy by the character controller system, just using Bullet to do manual intersection tests, then when in rag doll mode, Bullet is handling the whole thing.

I had all this working before so was just a case of re-writing the integration code, which happens at quite a high level so that the entire PcStateMachine code is not run once the rag doll has taken over. I just had to add some simple methods to the rag doll to extract the position of a particular shape from the current position to feed to the chase camera system.

Not much else to report at the moment. I'm going to look at working on a "view" mode next, whereby if you hold down the right mouse button, the camera mode switches to a behind-the-shoulder view and you can then rotate the player by moving the mouse and look up and down in more detail, in preparation for the bow-aiming mode but I'm not quite sure how this is going to work yet.

The player logic has become split over quite a range of areas and it is sometimes hard to know what should go where so constant refactoring is happening here.

Thanks as usual for stopping by. Your interest is much appreciated.

## Mumbling about the Camera system

In preparation for being able to do some stuff with the bow aiming, I've ripped out and rewritten the [font='courier new']Camera[/font] system for the game. I've split what was one system into two classes now, [font='courier new']ViewTransform[/font] which represents a position and orientation for a point of view with the orientation being represented by a quaternion and a [font='courier new']Camera[/font] class which takes care of moving between a current and a target [font='courier new']ViewTransform[/font] and providing a current view matrix based on its internal state.

In the video below, you can see that we have manual control of the camera while holding the right mouse button down, but pressing the N and M keys set hard-coded targets for the camera at opposite ends of the map which the camera will move to whenever those keys are pressed. All can be interrupted by the others at any point.

[font='courier new']ViewTransform[/font] allows us to think in terms of both rotation (quaternion) and angles (x, y rotations) but keeps all its rotations stored in a quaternion to make interpolating between two [font='courier new']ViewTransforms[/font] a breeze.

class ViewTransform{public: ViewTransform(); ViewTransform(const Vec3 &pos, const Quaternion &rot); ViewTransform(const Vec3 &pos, const Vec2 &angle); Vec3 position() const; void setPosition(const Vec3 &value); Quaternion rotation() const; void setRotation(const Quaternion &value); Vec2 angle() const; void setAngle(const Vec2 &value); void flatVectors(Vec3 &look, Vec3 &right) const;private: Vec3 pos; Quaternion rot;};[font='courier new']Camera[/font] just maintains a current [font='courier new']ViewTransform[/font] and an optional target transform and, depending on its mode, will work out the current interpolated [font='courier new']ViewTransform[/font] between the two.

class Camera{public: Camera(); enum class Mode { Manual, Tracking, Null }; void store(); void update(float delta); ViewTransform transform() const; ViewTransform transform(float blend) const; void setTransform(const ViewTransform &transform); void setTarget(const ViewTransform &transform); Mode mode() const; Matrix viewMatrix(float blend) const; Vec3 position(float blend) const;private: Mode md; ViewTransform cur; ViewTransform tar; BlendFloat t;};We don't really need to go into details here. It is all quite fire-and-forget really. When the camera is in [font='courier new']Mode::Tracking[/font], it is moving between a current and a target position, otherwise what it returns is just based on the current position.

The [font='courier new']flatVectors()[/font] method is used when the player works out what direction to move in based on the current orientation of the camera. Eventually we'll replace this with some kind of chase camera system.

So before I can get into the bow drawing and other combat camera activity, I think I need to get the normal chase camera system working. Not sure hwo this should work yet.

There is the original Tomb Raider system of the camera trying to stay behind the player at all times as the player rotates, or the newer Tomb Raider system whereby the camera stays a fixed distance from the player, but the controller controls the angle of the camera by rotating left and right and up and down, then movement is always in the direction of the camera towards the player. Not sure which is better here.

I'm already finding it difficult to track how this should work across all the different units that look after the player now. There is [font='courier new']Pc[/font] which is the highest level unit that connects the player to the [font='courier new']Entity[/font] system, the [font='courier new']PcData[/font] unit which is owned by the [font='courier new']Pc[/font] class and provides all the state to the [font='courier new']PcStateMachine[/font] class which looks after switching the player into all of its different states.

Hard to get a good overview, especially on a weekend morning when I'm half-asleep which is when I am doing most of the work on this these days :).

I'll post this now and then have some time to reflect I think. Sorry if it is largely mumble. That is how it is in my head at the moment.

[LATER EDIT]

Woke up a bit as the day wore on and managed to figure out what I needed to get the basic chase camera working, as per this video:

It was quite a lot easier than I expected actually. You'll see I've opted for the version where the player controls the rotation with the mouse and the direction the player runs in is relative to whatever direction the camera is currently facing. Feels quite nice and natural.

Using the targetting system here, so there is a small delay between the player moving and the camera tracking actually leads to a nice smoothing out of the camera motion which I personally find quite pleasing.

I'm going to fiddle about a bit more with this, see what else I can do with this system. Been meaning to get round to getting the chase camera working for months though, so a significant step here.

Thanks for stopping by.