On Embedding General-Purpose Scripting Languages

Published July 22, 2015
Advertisement
I spend a lot of time thinking about programming languages. Most often it's from the perspective of a consumer, i.e. a programmer using Language X. Sometimes it's from the viewpoint of someone who wants to explore Language Y, either for fun or for practical usage. Occasionally I see languages from the vantage of a language creator.

Recently, I've sunk a lot of brain cycles into thinking about languages as a non-programmer. More specifically, I've been thinking about the tendency for large (and sometimes small) games to use "scripting" languages to do various things.

I think there are basically two (often overlapping) desires in play when using an embedded scripting language. One is to make programmers more productive. Often this is believed to be possible by using "higher level" languages to write certain types of logic. The other common desire is to make non-programmers more productive, by enabling them to directly modify logic in the host program/game.


The more I ponder this, the more I'm convinced that in most cases, neither desire actually justifies the use of a general-purpose scripting language.


Lies About Language Productivity
It is fairly common in programming circles to hear assertions like "Language X is more productive than Language Y." A popular tag for this used to be "rapid development" although I hear that term less nowadays. I will not attempt to tackle the debate about whether or not X can be more productive than Y. However, what I would like to point out is that this is actually irrelevant!

If we're talking about embedding a scripting language into a host program, like a game, we aren't comparing X and Y. We're comparing "X+interop layers+Y" to Y by itself. This distinction may seem pedantic, but it's crucial. If we wanted to add X to our Y-based program in order to boost programmer productivity, we have to be careful here.

Depending on how the job is done, adding a language to a program can actually be a massive net loss of productivity. Suppose we have a C++ program, written by C++ programmers, who implement a Lua interpreter in their program. Now the average programmer working on the program must know not only two different languages, but must also be intimately familiar with how the two interoperate.

If we designed APIs like this, we'd revile the result as a leaky abstraction, and rightly so. I content that embedding a general-purpose language is almost always a net loss, for the following reasons:

  • Programmer knowledge is now required to span multiple languages
  • Programmers MUST be fluent in the boundary layer between languages
  • Interop is likely less performant than staying in a single language, especially if experts in the interop are unavailable
  • Impedance mismatches are guaranteed to crop up, i.e. languages think of things differently and have different idioms
  • The mental overhead of managing multiple languages is taxing and leads to higher bug rates and nastier bugs
So if you want to embed a general-purpose scripting language in your game to make development more productive, don't. Focus on making better tools and making better architectures. A good architecture allows you to build high-level logic almost as if it were scripting anyways. There is no need to do it in a different language.

Note that this is in no way an argument against using, say, C# to write your level editor and C++ to write your game.


What About Designers?
The other motivation I mentioned above for doing scripting systems is to allow non-programmers to do powerful things.

I'm going to point out something that feels painfully obvious to me, but in harsh experience, far too many people just cannot wrap their head around:

Shoving a general-purpose programming language in a non-programmer's face is a great way to lose team members and make people very frustrated.

I can safely assume that most of my readers are not astronauts. So if I were to cram you into an Apollo capsule and yell GOOD GODDAMN LUCK while slamming it shut, I can safely assume that you'd react negatively. You're not trained to think about the tools in front of you. You're not informed as to what they're for. You don't know what to expect after the door closes and the last echoes of my jerk voice fade away. You're in an alien environment and can't even tell what will happen next.

That is what putting a designer in front of a general-purpose scripting engine feels like.


Why do I keep mentioning languages as "general purpose"? Because there is an alternative, which works, and works well.



Domain-Specific Languages
Many people conflate DSLs with "very limited languages." This is unfair. A DSL can be immensely powerful and still be confined to a particular domain. It can even be Turing complete if you really want.

I half-jokingly posted on Twitter the other day that Excel combined with a verb system would make a great DSL for game designers.


And the more I think about it, the less I think that's a terrible joke of an idea.

I might just give it a shot someday.
13 likes 9 comments

Comments

ferrous

I spent some time on a tool that could import basic excel formulas for a project that ended up getting cancelled.

July 23, 2015 10:28 PM
tnovelli

Yeah... I considered Lua, but I didn't see the advantage of Turing-completeness for RPG scripting. I've done that with Javascript, in JS games where everything's exposed, and I ended up systematizing everything for consistency's sake. Not using all that power. Stuck with JS syntax oddities.

So I went with a fixed-function DSL. It's just data. You could definitely use a spreadsheet.. CSV would be convenient. One reason not to: spreadsheets tend to mangle your data. For example, a new version has different defaults for unicode. At least I can trust my text editors.

July 24, 2015 12:48 AM
Hodgman
On the programmer productivity point, you're assuming a situation where the game programmers spend a lot of time in both languages and hence have to interop a lot. This may often be the case, but in the Lua-based console games that I've shipped, Lua has been the primary language for the game codebase, and C++ the primary language for the Engine code-base. A gameplay programmer could go an entire project without writing any C++ code at all (and vice versa for a tech programmer) -- so the dual language + interop penalties only applied to a small number of the staff, for the small periods of time where they were working on the game/engine API boundary... Or when optimizing gameplay code by porting it to SPU jobs - but such work already requires the type of programmer who can juggle 6 abstractions at once while trying to strip them all bare :lol:
July 24, 2015 12:56 AM
LoneDwarf
If your project requires knowledge of both languages then you likely have a design problem. The interop/glue layer is usually solved by one guy and never talked about again. You think it's bad when a someone comes and asks you questions about lua, wait until they ask about your DSL.

For The Warriors we used lua for scripting for a few reasons.
  • PS2 dev kits were expensive and only core engine devs had them. Scripting could be done without dev kits.
  • Build times for C++ on any reasonable sized project will add up in lost time.
  • It's easier to hire someone with lua experience and for people to find resources to learn.
  • A lua script was a way to reduce memory instead of having all the C++ loaded. It's easy to forget that that C++ code is always loaded and taking memory. Sure you could use dlls if you wanted but who really wants that?
Since that project I haven't done anything without lua. The biggest downfall is that the tools for debugging lua aren't at the same level as C++. Nothing is perfect but I think you under estimate the work involved in a DSL and the amount of work it will continue to demand.

Sure a generalized solution will always cost you something in speed and/or memory but I am fine spending it. I remember something about how Carmack used very little asm for Doom but felt the speed tradeoff was worth it.
July 24, 2015 06:43 PM
ApochPiQ

On the programmer productivity point, you're assuming a situation where the game programmers spend a lot of time in both languages and hence have to interop a lot. This may often be the case, but in the Lua-based console games that I've shipped, Lua has been the primary language for the game codebase, and C++ the primary language for the Engine code-base. A gameplay programmer could go an entire project without writing any C++ code at all (and vice versa for a tech programmer) -- so the dual language + interop penalties only applied to a small number of the staff, for the small periods of time where they were working on the game/engine API boundary... Or when optimizing gameplay code by porting it to SPU jobs - but such work already requires the type of programmer who can juggle 6 abstractions at once while trying to strip them all bare laugh.png


That doesn't change my point, really; instead of having N C++ programmers you now need to have M C++ programmers and Q Lua programmers, where (M+Q) is probably > N. You're also condemning those Q programmers to living with subpar tooling and debugging experiences. Unless they are highly accustomed to being productive in Lua already, they will probably take a very long time to achieve the same degree of proficiency as they already have in C++.

To be fair, it isn't exactly difficult to find programmers who know Language X better than they know C++. My point is that if you want programmers to be able to be fluent in any part of the game code, they need more knowledge than if they only had to use one language. Maybe this can be minimized to a small number of people in an ideal case, but my experience is that eventually having a language divide in a codebase makes it harder to work on, not easier.

Consider the case where you need to fix an issue that lies partially on each side of the language divide. Either you bottleneck these issues on your small number of ninjas, or you require a larger number of people to be deeply familiar not only with the languages involved but in how they interact. Even the sheer communication and scheduling overhead of handing off such issues between various programmers is a subtle but dangerous cost. IME these bugs usually mean one "high level language" programmer bangs his head against the wall for some amount of time before bringing in a "multilingual" developer to help diagnose it. And even then sometimes you need to bring in C++ specialists and scripting specialists before a resolution can be found.

I guess what I'm trying to highlight is that in real life the idealism of "high level logic is written in a scripting language" misses out on the realities of how costly that division really is. My point is that a lot of game developers are in denial about the impact of language divisions (probably just through being unaware of the extents of the problem).
July 24, 2015 07:08 PM
ApochPiQ

If your project requires knowledge of both languages then you likely have a design problem. The interop/glue layer is usually solved by one guy and never talked about again. You think it's bad when a someone comes and asks you questions about lua, wait until they ask about your DSL.


I've deployed several DSLs in successful games. I'll take a well-defined and easy-to-use DSL over Lua any fucking day of the week, hands down.


Since that project I haven't done anything without lua. The biggest downfall is that the tools for debugging lua aren't at the same level as C++. Nothing is perfect but I think you under estimate the work involved in a DSL and the amount of work it will continue to demand.


Bad DSLs are a liability, sure. But I would say that bad general-purpose languages are a vastly larger liability.
July 24, 2015 07:10 PM
LoneDwarf

Really you take lua and make it domain specific. The idea of creating a whole language is ridiculous. The fact that you have deployed several successful games using a DSL needs to be taken with a grain of salt also. I bet if you asked different members of the team you will get different answers. GTA3 was pretty successful from what I heard and the scripting for it was god awful. This is in no way a dig against the people that wrote it as I highly respect what they did but the DSL didn't contribute to it's success in any way.

Also putting effort into something like lua gives you the ability to use it on the next project with a new domain. I use it for my build system, batching operations on assets, basically anything not C++. For me lua gives variables, flow control, maps and the ability to call into C. Just the right amount of glue anyone would want. Why write a parser and then all the docs on how your DSL magic layer works. I have already shipped while you make your DSL.

July 24, 2015 08:39 PM
Juliean

On a hobbiest note, I've made some totally different experiences with scripting languages (mostly visual scripting with c++).

First off, our graduation project for university was programmed in 3-4 months with 4 programmers. We were using Unreal 4, which uses both C++ and their visual blueprint system. I could probably write a separate article about this, but even though we had a lot of interop (base classes in C++, specific abilities/behaviour/UI in blueprint), we had almost no issues with it and I can say that we would have never been as productive if we had eigther only C++ (I just imagine how painfull it would have been to implement the UI purely in C++ with like 100 minor changes per day with 1 min+ compile/start time; plus doing things like selecting assets in the script code per drop down was a real time-saver), or only Blueprints (having C++ for base functionality like a base player class with overridable methods is very important too). So the mixture of both (visual) scripting and C++ was an enormous advantage here, even though to be fair that might also be accountable to how phenomenal Unreals C++ script bindings are (you can inherit from C++-classes in script, and pretty much use all your C++ methods in script without handling the script binding yourself).

The other case is my own, where I have developed my own visual scripting system similar to Unreals'. I'm using it currently for porting a game that was written with Rpg-Maker which had kind of a DSL with their "event system", for stuff like cutscenes, NPCs etc... Now apart from that my new scripting system allows me to be more productive in comparison, I also cannot imagine how it would be doing this in C++ (a new class for every NPC that does something special in a scene? Yuk). Now I have too little experience with modern DSLs that I can pass any judgment on whether or not I would be more productive for specializing behaviour and cutscenes in comparison to my visual scripting system, but I can't imagine that it would be much better for my cases. I hope I'm not too biased towards my own work, at least in direct comparison to the Rpg-Maker it fairs way better...

So yeah, take it with a grain of salt, I obviously lack much professional experience and am therefore only talking about limited-scale projects in type of manpower (the game I'm porting is an 10 hours+ 2D action Rpg, so I would at least consider it medium-scale). Just thought I'd leave that there anyways.

July 25, 2015 03:23 PM
Ravyne

I think the DSL approach has a great deal of merit, and gets far less consideration than it should. Part of this may be that C++ isn't a great language for implementing DSLs, at least not embedded ones (BTW, are we talking about embedded or external DSLs?)

I don't think using something widespread as a scripting language (say, Lua or C# or even Ecmascript) is all that bad if done well, but its definately a larger liability than I think people realize. I might go one further and say that the apps that use scripting most-successfully fall into two categories -- one where the API surface of what's essentially a DSL anyways happens to be expressed in/through teh scripting language, and a second like has been said where its really only the inner-most core of the engine and the scripting interface are written in the host language, with all gameplay logic implemented therein (even some of the lighter-weight traditional engine components, perhaps). In the case where its really just the expression language of the DSL, the extra expressiveness afforded by the scripting language itself really is a liability, useful as it may be at times to expedite solutions to unanticipated needs.

I do tend to agree though that for small teams, scripting probably is not the best solution. The people likely to be wearing the scripter hat are also likely to be wearing the coder hat as well. I think most "designers" on indie-scale teams are probably either competent coders, or would be better served by tools that have no need for Turing completeness. Full-blown scripting langauges seem to me to only be a defensible proposition when it enables teams to scale up, and when people have more-discreet roles.

July 29, 2015 07:27 PM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Profile
Author
Advertisement
Advertisement