Why YOU should embed a web server in your game engine

Published March 31, 2014
Advertisement
I'm going to give you a sales pitch. It might sound outlandish at first, but bear with me - I think, by the end, you'll agree that this is a Good Thing(TM).

You should embed a tiny web server in your game engine.


It doesn't need to ship with the game, of course; this is a development tool. If you're into letting people mod your work, though, consider leaving the server active.

Don't roll your own. That takes too much work and is a security liability. There are plenty of tiny HTTP servers available in the open source world, in virtually any relevant language you can name; pick one and go with it.

What does this server do, you ask? Simple. It provides a window into the game world that can rival even the best IDE debugger's offerings. It gives designers and content creators a way to iterate on their work almost instantly instead of spending hours in the build/test/tweak/rebuild cycle. And it offers a mechanism for directly manipulating your game state for easy experimentation and exploration.

Sound fantastical? Too good to be true? Read on!


Debugging Unhinged
Ever wanted to dump a list of every NPC in your game and view its current state? Done. Stuck on a bug that only repros after several hours of play on someone else's machine, who doesn't have a debugger installed? No problem! Just dump the relevant data through your web server and you can diagnose it right at their desk. Frustrated by the difficulty of reading complex data structures in your debugger's interface? Worry no more; you can output the data you want and view it in any way that makes sense to you. No messing with visualizers or other IDE extensions.


The Ultimate Iteration Tool
This was actually my original motivation for building one of these suckers. Suppose you're working on tuning a game feature, and you need to edit a few numbers repeatedly. Make a tweak, test the changes, tweak some more, rinse, repeat. If you have to run through a build cycle every time this tweaking happens, you're in for a lot of pain.

Worse, you will lose context every time you wait on a build. Even if all you need to do is restart the engine or reload a level, your brain will stop processing the task at hand and get distracted a tiny bit (or a lot). If you could edit those numbers and instantly see the game change in response, you can keep that feedback loop tight and fast.

The best part? Iterating instantly means you'll not only have time to try a lot of ideas, but you'll find that over time it changes the kinds of ideas you have in the first place.

This is powerful.


OK, OK, sign me up. How do I do it?
Pick some lightweight HTTP server code. Plunk it into your engine and set it up to listen on some innocuous port (I like 8088). You're going to need to implement three things: the landing page, the "features", and the web-facing UI.

First, the landing page. This should just be a list of things you can go do in the game. As you add features, it should grow. (Hint: generate this page dynamically. Trust me, it's cooler that way.) This is the first thing a user sees when hitting your web page, and should help guide them into what they want to see or do.

Secondly, the features. This is the bulk of the work. In my system, I have a simple central API where I can register a "web console feature." This is basically just a set of callbacks. When someone opens the web page for the game, they click a link for a feature, and the appropriate callback is invoked.

Each feature is just a well-defined unit of functionality, such as "list all NPCs" or "drill into detail on a specific NPC" or "show me the stats of all the weapons in the game" or whatever. From the code side, the implementation is easy enough, but once you start wiring this up, chances are you'll get addicted to it and find yourself wiring up more and more code to the system.

Now for the UI bit, and the secret sauce. Don't generate HTML in your web server. Instead, generate XML documents. This allows you to keep the code that emits the web page constant, as long as the data it puts out remains in the same form. When you want to actually use this XML, then you write a simple XSLT layer that turns it into something pretty.

This last bit of separation is crucial. It means you can update your view of the game data without rebuilding any engine code. It also means that you can have a web dev or someone else comfortable with XSLT and HTML build the UI for you. You can even do fancy JavaScript, CSS, and whatever else your browser supports. Just remember to keep the original data in simple XML, so that the UI layer can be upgraded at any time, without touching the engine itself. This is especially vital on larger teams where you don't want to constantly make updates to the "NPC code file" every time you change the color scheme of the UI.


Bonus Mode
Here's a handful of ideas to run with, that make this system really shine:

  • Give each game object a unique ID. Offer a URL that lets you enter an ID and see details about the corresponding game object.
  • Attach state information as well as static information. Don't just say this NPC has 50 hit points, show his max as well. Hell, show his entire character sheet or whatever.
  • Don't forget this doesn't have to be read-only. Letting people edit the game via the web console is insanely powerful and a great time-saver.
  • Mock up a rough draft and hand it to a Web UX designer to build a really shiny product out of it.
  • Try to make URLs stable and easy to share between developers.

Do all this stuff, and I promise your team will love you. YOU might even love you, I don't know. Either way... enjoy your new fame and glory!
18 likes 22 comments

Comments

Ashaman73

Have to sign this.

My game implemented a webserver and it was one of the most useful things I have added to my engine in the recent decade.

Additionally , using HTML5 canvas + Javascript + Ajax helps a lot to rapidly implementing really useful tools just by using your common webbrowser.

March 31, 2014 05:40 AM
3Ddreamer

Good day,

This sounds great for Indy developers, but documentation and other accountability issues weigh heavier the larger the development company becomes. Management software by itself demands tracking people's contribution for a wide variety of reasons.

Like I say, great for the small and nimble developer but when investors and large organization must be tracked and paid, maybe not so good? This could be a detour from the highway to true game development scalability.

Any thoughts?

March 31, 2014 06:36 AM
ApochPiQ
Honestly, I think you have it dead backwards.

For a small developer, building fancy tools is often considered (rightly or wrongly) to be an opportunity cost; the same energy could just as well go towards directly building product. I personally argue that this sort of tool is a force multiplier at any scale, so it actually gives the savvy indie an edge over larger teams with bulkier processes in place.

As far as large teams go... managing a team at scale is all about minimizing friction. If you have a pipeline or process that gets in the way of people doing their jobs, you will ultimately lose to someone else who is more nimble (not necessarily smaller, mind).

Honestly, I don't see how better development tools in any way impede tracking contributions of individual team members. A web console is orthogonal to asset/version control, and if you're not doing good version control, you're basically fucked anyways. Developers still use the standard build pipeline to commit changes - they just get a much easier mechanism for experimenting and discovering what those changes should be.

As for scalability... I have to come back to the idea of minimizing friction. Adding people to a project increases friction in many ways - communication overhead, politics, difficulty of reaching group consensus, and so on. Past a certain point, if you aren't careful, process actually drowns out productivity, and that's when your team is well down the road to obsolescence. Any decent team of large scale will tell you that developing tools to improve productivity is among the best ways to counteract the burdens of process overhead.


Out of curiosity, what's your experience with managing (or even working in) a team of, say, over 50 people? I'm not sure I understand your perspective and I'm wondering if there's some kind of trend towards a different mode of thinking in certain sectors. (To be honest I've only worked with really great teams, so I may be biased.)
March 31, 2014 07:02 AM
_the_phantom_
Stuff like this indeed vital for large teams; in fact any system where you can get information and adjust it on the fly is vital for large projects.

While not using a webserver the previous version of an engine I worked on had a system which let you expose values to the end user over the network to allow for live tweaking (and saving) of them so that you could adjust game values on the fly (and that was any value!).

The last version had a larger integration where this system was expanded to allow some interactivity between our world editor and any running instance of the game (so if you had 3 instances all connected to the dev-pc and the world editor open you could move the camera in the world editor and see the changes reflected on all running instances), as well as allow for changing of material parameters and other graphics 'tweaks' as were exposed; something the artists loved.

By the third iteration we had a deeper link system over the network to the extent where you could add an object in the editor and have it spawn in the running instances, you could instruct the engine to "hot load" resources from disk meaning while the game is running you could, for example, update a texture and have it reloaded on the fly (this extended to anything; models, scene lists, material data, anything) and the UI team had built a deep introspection system for the UI layer which allowed them to inspect the whole state of the UI at any given moment and receive live updates to things (animation states and positions for example).

While I've left the company now further plans exist to improve the hot-load feature set to include code loading.

The funny thing is until you use this stuff, even the basic 2nd implementation, you don't really "get" how useful it is; when discussing the 3rd iteration of the engine one team, who hadn't used the live link stuff, wasn't that bothered and wanted to forget it figuring what they had with the first version was enough (!) fortunately those who HAD used the 2nd version managed to convince them how awesome it was to use and they were sold in the end.

While all this is slightly different to the original blog post the point is that this kind of introspection, tweaking and updating code is awesome to have, regardless of team size smile.png
March 31, 2014 08:42 AM
tnovelli

I'm doing a lot of HTML5 games, so I get all this 'for free'... adding it to a native-code game is an interesting idea.

I guess I'd implement most of it in javascript, like this:

- HTTP server serves index.html and console.js directly from disk, with no processing

- console.js does AJAX requests to get data as needed

- server looks up object by GUID and calls its .toJSON()

March 31, 2014 01:03 PM
Dwarf King

This sounds very interesting. I really like to read your blogs. This one is not only interesting but also seems very useful.

March 31, 2014 01:37 PM
evanofsky

This is an awesome idea. However, as a web developer, I beg you, drop the XML/XSLT bit. In my experience, it's a beautiful theory that turns out to be a real pain in practice.

A better idea is to write it as a RESTful JSON service. JSON is a million times easier to use in Javascript, you still get clean separation of responsibility, and the web developer doesn't have to worry about crazy XSLT transformations, they can use their templating engine of choice.

Sorry, just picking on the one thing in this blog that I actually have knowledge of. :)

March 31, 2014 04:51 PM
Ravyne

I was actually going to ask something along the same lines. I don't think XML should be dismissed out-of-hand, I actually tend to like it in its place, but XSLT and such isn't knowledge that most non-web-devs have.

I wonder if JSON would be easier for many things. It might also make it easier to interface other tools with it, aside from a web-browser interface (although, I think the browser is great for knocking up quick UI). Maybe even an SQL-like query language would be cool -- like Linq in .net-land.

Any thoughts on XML v. JSON, or which other web technologies you think might be useful for implementation?

March 31, 2014 05:28 PM
Agony

This is exactly what I'm going to be needing in the very near future. I just didn't realize it until reading this article, thanks so much!

I'm working on a city builder and I'm deep in the bowels of getting the simulation code fleshed out. What that naturally means is that there is a lot of data that affects the game, but is not intended to be directly viewable in-game. I've been dreading the process of writing throw-away UI components within the game, and I've found direct debugging to be very impractical for assessing simulation behavior. Writing out massive log files and manually examining them, or even writing visualizers for the data, is also quite a pain.

But having an HTTP-accessible API sounds awesome. I've already done some successful experiments moving my finance information out of spreadsheets and into a more web-friendly form, so I think I'm psychologically prepped for doing a similar thing to my games.

March 31, 2014 05:32 PM
ApochPiQ
I'm not really religious about the XML/XSLT thing. The point of that is more that you should stream data from your engine's web service, not usable content. Separating the UI implementation from the raw data that comes from the game is powerful because it lets you vary them independently.

I personally really like the XML/XSLT combo, because I have access to a lot of library code that makes generating great-looking and feeling pages really easy. If JSON is a more suitable approach for your given project, go for it.

I've had zero need to write more than trivial JavaScript in my web console implementations, so JSON isn't really a win to me. Implementing a trivial set of code in the server is more valuable to me than having a rich, RESTful API; so I literally have three URIs in my server: the root landing page, /edit, and /submit.

As with anything, don't just drink koolaid and try to fly; use what makes sense for your project and apply your own reasoning to determining what that might be.
March 31, 2014 05:54 PM
Liort

See the article in this magazine (page 18):

http://twvideo01.ubm-us.net/o1/vault/GD_Mag_Archives/GDM_January_2013.pdf

Basically, the whole engine is said to be built on top of a web server and is totally browser based. maybe some ideas can be borrowed from that article that are similar to what's discussed in this blog post.

March 31, 2014 10:36 PM
Jason Z

This is a great post - if you have any implementation details (i.e. which web server you are using) and any friction points that you ran into, I would appreciate hearing about them.

In general, the web interface sounds more or less like a normal game console extended to support the common browser interaction model. Especially if you are only wiring up specific commands to the interface, it is very much like a traditional console. The benefit (and I think this is one of your key points from above) is that you get to separate the visualization of the data from your engine, which is nice.

Regarding the REST interface, I think there are quite a few off the shelf REST libraries available too, so there shouldn't be any issue with adding it. The URL scheme is actually no more complex than the list of links that your landing page contains, so I don't see any issue there. If I recall correctly, BitSquid is actually running all of their tools over a JSON enabled API...

April 01, 2014 07:49 PM
Agony

This is a great post - if you have any implementation details (i.e. which web server you are using) and any friction points that you ran into, I would appreciate hearing about them.

For what it's worth, here are my experiences from yesterday. After some quick research, I personally went with Boost.Asio, since I'm already using other Boost libraries, have a small amount of experience with that library already, and I like its style. But the included HTTP server is really only sample code, not necessarily production-ready, so I supplemented it with http-parser for parsing HTTP requests.

I haven't actually implemented any relevant endpoints/handlers; only enough to verify that the connection works and the browser gets the expected response. We'll see what additional difficulties I face when I start using it heavily.

Thus far, I've experienced three problems.

The first is that I forgot to actually run the IO service code. All of Boost.Asio examples seem to use run(), which is a blocking function with its own internal loop, but there's also conveniently a poll() function that I can just call once each game loop which processes all ready requests. Seems to work well.

The second was determining how to handle keep-alive. I don't know if I'm doing it right, but it didn't seem too hard to figure out how to adapt the sample HTTP server to keep connections open (it auto-closed every connection after a single request/response).

A little bit of extra logic to examine the headers in the response for "Connection: close", and I think it should handle both client and server requests to close a connection properly.

The third was handling the response headers effectively. In particular, adding the Content-Length header in the server code if it wasn't manually added to the response by the request handler. But also headers added by the handler that should affect the server behavior, such as "Connection: close".

That all seems to be working well, so now my biggest hangup is my general inexperience with web services, and thus the lack of knowledge about using various client-side tools to consume any API that I might implement.

April 02, 2014 03:10 PM
tnovelli

Andy, http-parser sounds useful.. thanks.

For persistent connections I think you want WebSocket. It's a recent extension to the http protocol, meant to supersede old hacks like 'COMET' and 'long polling'. (Here's a little test code: http://tnovelli.net/junk/ws2/ ... note that the server isn't running so it won't work..)

April 03, 2014 12:00 PM
swiftcoder

Any recommendations on simple embeddable http server libraries for C++? We do this regularly for debug tools at work, but that is in Java, which is a tad more capable out of the box. And I'm hesitant to take a dependency on Boost just for ASIO (my game is not networked beyond debug tools)...

April 06, 2014 05:14 AM
Agony

Any recommendations on simple embeddable http server libraries for C++? We do this regularly for debug tools at work, but that is in Java, which is a tad more capable out of the box. And I'm hesitant to take a dependency on Boost just for ASIO (my game is not networked beyond debug tools)...

An alternative to Boost that I was considering is lighttpd, written in C, using the BSD license.

I had also briefly looked at Libmicrohttpd (LGPL) and Mongoose (GPL), but licenses dissuaded me. Your relation to various licenses might differ.

April 07, 2014 05:03 PM
Liort

In the article from GDMAG (that i linked above), they mentioned Mongoose: https://code.google.com/p/mongoose/

April 07, 2014 05:25 PM
Kwizatz

Any recommendations on simple embeddable http server libraries for C++? We do this regularly for debug tools at work, but that is in Java, which is a tad more capable out of the box. And I'm hesitant to take a dependency on Boost just for ASIO (my game is not networked beyond debug tools)...

An alternative to Boost that I was considering is lighttpd, written in C, using the BSD license.

I had also briefly looked at Libmicrohttpd (LGPL) and Mongoose (GPL), but licenses dissuaded me. Your relation to various licenses might differ.

I am on the same boat, the problem is lighttpd is not suitable for embedding, or the developers don't care about that, so you're on your own (reference). Different kind of embedding... should have known.

libmicrohttpd and thttpd would be my options, myself leaning towards thttpd because of the BSD license variant, though libmicro is LGPL 2.1, so no problems with propietary code as long as it is a shared library.

The problem with both of those is that there is no Windows port, if you use cygwin, you get GPLed, so you'd pretty much would have to create the build scripts to build them with MSVC or MinGW, not that it would be impossible or too complex, but definitely extra work you were not expecting to have.

April 08, 2014 06:18 PM
swiftcoder

I'm really looking for something that I can drop into my own build system - I don't like dealing with heavyweight dependencies. Makes me miss python, where things like itty exist...

April 08, 2014 07:00 PM
Servant of the Lord
What is the benefit of using HTTP over UDP or TCP and a dumb/simple receiving application?
April 10, 2014 12:48 AM
ApochPiQ

What is the benefit of using HTTP over UDP or TCP and a dumb/simple receiving application?


Why bother with that? Why not just build the UI for your debugging etc. directly into the game then?

The point of using known, commodity protocols and formats is you do less work. Your approach requires a shitload more work, and maintenance cost on top of that.

Would you rather write, debug, maintain, and constantly add features to your dumb-and-formerly-simple app, or just use an existing browser that already has all the stuff you could ever want for UI and presentation layers?
April 10, 2014 04:46 AM
swiftcoder

What is the benefit of using HTTP over UDP or TCP and a dumb/simple receiving application?

jQuery.

Need to be able to configure a list of post-process filters via drag-n-drop? That's a full afternoon with most desktop UI frameworks, and a couple of minutes with jQuery.
April 10, 2014 05:52 AM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Profile
Author
Advertisement
Advertisement