Jump to content

  • Log In with Google      Sign In   
  • Create Account

Continuous Refinement


Posted by , 17 January 2011 - - - - - - · 632 views

I'm pleased to announce that the GDNet+ member webspace is now fully back online - and, for the first time in about 3 years, accessible by FTP once more!

Just FTP into members.gamedev.net, using your regular username and password, to access your personal space. Old GDNet+ members should find all their files ready and waiting for them. We've also increased the space quota to 100MB per user, and we'll look at increasing this further as things get settled in.

Anything you upload into your webspace is accessible over HTTP, too...

Old GDNet+ members can browse to the same addresses that they've always used. We'll probably be retiring these addresses at some point, but we'll make sure we let you know before we do.

New GDNet+ members, log in, and browse your way to http://www.gamedev.net/subscribe - take a look at the bottom of the page for your address info.

GDNet Slim

Posted by , 16 July 2010 - - - - - - · 497 views

For the past three days or so, I've taken some time away from working on V5 to see if there aren't some things I can do for the current site, V4. As you're no doubt aware, we're in a bit of a tight spot on cashflow right now - much like everyone else in the industry - so I figured I'd see if there wasn't anything I could do to bring down our hosting costs. Messing with our hardware and datacenter setup is beyond my remit; I'm only the software guy here, but that software has been churning out an average of 15 terabytes of data every month, and bandwidth ain't free. Not to mention that it makes the site load more slowly for you.

So, what exactly have I done about it? 97 commits to Subversion in the past three days, that's what [grin]

  • I spent about 4 hours optimizing and refactoring the site's CSS. Historically the site's had one large (28kb) CSS file per theme, with lots of duplication between themes; this is now one shared (16kb) and one theme-specific (11kb) file. A whopping 1kb saving, hurrah! Might not seem like much, but now that all the common stuff is in one file, it makes it easier to optimize, and also means that the optimizations will be picked up by people on every theme.

  • I totally rewrote the markup (and CSS) for the header banner you see up top there. It used to be this big 3-row table, with 0-height cells, lots of sliced-up background imagery, etc. It's now 4 divs. Much, much cleaner.

  • I put all the little icons from the nav bar into a sprite map, and got them all to be displayed by CSS. So, now, instead of making 15 separate requests to the server, you only make 1, and now there are no image tags in the header of every page.

  • I rewrote the little popup you get when you mouse over the 'recent topics' on the front page. The javascript library we were using to do this weighed in at 50KB (!!!); even minified it was still 23KB. I had a look into a jQuery solution, as we can embed a version of jQuery hosted by one of the big CDNs, but then realised that the whole thing could just be CSS instead. So, it is. That's a 50KB saving on bandwidth for every brand new visitor to our site's front page right there, which is substantial.

  • I stripped a bunch of <br> tags out of the markup and replaced them with margins (specified in the cached CSS files, naturally).

  • I updated our Google Analytics code. This wasn't strictly necessary, but I wanted to do it, and in the process I discovered that none of the forum pages have actually been including it properly up until now. The visitor graph in Analytics since I fixed it has a spike that looks like we've just been featured on CNN or something [grin]

  • I tidied up the breadcrumb, search box, and footer code. Again, mostly just getting rid of tables and replacing them with CSS.

  • I killed some of the 'xmlns' attributes that get left in our output due to the way we're using XSLT. There's still a bunch of them around, but I covered forum topics, which are the most popular offender. At some point I'll go back in and do all the other cases.

  • I redid the markup for the headers in 'printable version' articles. The gain from this won't be too huge, but it's often where Google searches end up, so it won't be nothing either. Also because I HATE TABLES AND WILL MAKE LOVE TO CSS IF IT IS EVER INCARNATE AS A TANGIBLE ENTITY.

  • I started switching the site over to using Google Ad Manager, instead of our in-house copy of BanMan. This is quite a big deal; the switch has been far from painless for me, and it's still ongoing, but the benefits are numerous. Firstly, instead of the ad images consuming our bandwidth, they'll consume Google's. Secondly, instead of the ad system consuming our CPU cycles, it'll consume Google's. Thirdly, instead of the ad data store consuming our disk space, it'll consume Google's. I'm pretty much fine with this, and for whatever reason, Google are too.

  • I made us a new version of the OpenGL|ES logo. It's shinier!

That's pretty much everything for now. It's a little difficult to get a picture of how much total change it's made, but the HTML for the site front page has dropped from 95kb to 85kb. I guess I'll find out if it's actually made a serious dent when I hear the bandwidth figures in a few days.

What's the downside to all this? I've been acting with basically no regard to old versions of IE. Chrome is my primary development browser now, with Firefox a close second; I check that things work in IE8, particularly when using unusual CSS pseudoclasses like :hover and :first-child, but anything prior to IE8 - and especially anything prior to IE6 - can go die in a fire, basically. I know, I know, you can't do anything about it, your machine is locked down by corporate, I understand... and I don't care. These days, I think I'd be comfortable accusing any sysadmin who hasn't upgraded all their machines to at least IE7 of criminal negligence.

I guess the site will probably still work in old versions of IE. I'm not actively trying to shoot them down. Yet. By and large, things should degrade gracefully.

To end, here are some excerpts from my SVN logs that you may enjoy.

2010-07-15 00:29:18 dropped prototype and clientscripts.js from the page header. (over 120kb for a new visitor!)
2010-07-15 00:32:50 also dropped menu.js, as the menus have been CSS powered for some time now

2010-07-15 03:24:27 killed the empty child! \m/

2010-07-15 04:33:49 tidied up breadcrumb + search boxes
2010-07-15 04:34:38 oops
2010-07-15 04:35:45 added a floatclearer
2010-07-15 04:37:03 try again

2010-07-16 02:21:38 updated 'printable' articles to use GAM
2010-07-16 02:23:11 forgot the <script> tags :)

2010-07-16 02:28:21 switched the box ad on the printable-article page, and changed it to use the adslots code
2010-07-16 02:28:50 uups
2010-07-16 02:29:31 it's times like this I wish the site was written in pure javascript

2010-07-17 00:17:26 new 'recent threads' code, using pure CSS for the popups
2010-07-17 00:20:52 added a little space between 'recent threads' entries
2010-07-17 00:23:10 bye bye, overlib! :D

Activity streams

Posted by , 05 May 2010 - - - - - - · 321 views

Today's chocolate-chunk-o'-LINQ:

public ActivityLogEntry[] GetActivitiesFiltered(DateTime? startTime, DateTime? endTime, Uri[] actors, Uri[] actorTypes, Uri[] objects, Uri[] objectTypes, Uri[] verbs, int? maxToFetch)
using(var context = new ActivityEntities())
var entries = context.Activities.AsQueryable();

if (startTime.HasValue)
entries = entries.Where(act => act.timestamp >= startTime.Value);
if (endTime.HasValue)
entries = entries.Where(act => act.timestamp <= endTime.Value);

if (actors != null)
entries = actors.Length == 1 ? entries.Where(ent => ent.actorUri == actors.First().ToString())
: entries.Join(actors, ent => ent.actorUri, act => act.ToString(), (ent, act) => ent);
if (actorTypes != null)
entries = actorTypes.Length == 1 ? entries.Where(ent => ent.actorType == actorTypes.First().ToString())
: entries.Join(actorTypes, ent => ent.actorType, act => act.ToString(), (ent, act) => ent);

if (objects != null)
entries = objects.Length == 1 ? entries.Where(ent => ent.objectUri == objects.First().ToString())
: entries.Join(objects, ent => ent.objectUri, act => act.ToString(), (ent, act) => ent);
if (objectTypes != null)
entries = objectTypes.Length == 1 ? entries.Where(ent => ent.objectType == objectTypes.First().ToString())
: entries.Join(objectTypes, ent => ent.objectType, act => act.ToString(), (ent, act) => ent);

if (verbs != null)
entries =
act => act.ActivityVerbs.Join(verbs, v => v.verb, w => w.ToString(), (v, w) => w).Any());

if (maxToFetch.HasValue)
entries = entries.Take(maxToFetch.Value);

return entries.Select(MakeFromEntity).ToArray();

V5: User accounts and profiles

Posted by , 18 April 2010 - - - - - - · 406 views

At the moment I'm working on the code for managing user accounts. This encompasses logging into accounts, creating new accounts, changing your password, and so on. There are some interesting features and design requirements that make this a non-trivial thing to do, so maybe it'll be interesting for you to read about it.

Federated Identity: A world of pain

Probably the biggest thing affecting the way user accounts get handled is the fact that we're supporting federated identity in V5. No longer will you need a username and password with us; if you'd prefer, you'll be able to sign in using a Facebook or LinkedIn account, or a generic OpenID. This is potentially a pretty big deal for our signup rate; given that pretty much every piece of content on the site will have a 'share' button that can spam a link to the social network of your choice, we're going to get an increase in people coming from sites like Facebook, and we want to make it as easy as possible for them to interact with the site, post comments, and so on.

Simply authenticating with these external sites would be easy, but we also want to fetch profile data from them... and while technically straightforward, the data privacy policies complicate matters. You're not allowed to store any data you retrieve from Facebook for more than 24 hours (with a few exceptions, like the numeric user ID); LinkedIn has a similar, though less explicit, policy. But if you come to the site from Facebook and post a comment, who do we attribute that comment to a week later? We can't store your name, avatar, or anything like that for more than a day.

What we have to do is simply re-fetch the data from Facebook when we need it. We can cache whatever we've fetched for up to 24 hours, but after that we drop it from our cache and wait until somebody needs it again. As well as storing your Facebook user ID, we also store the session key needed to talk to Facebook about you. The session has the special 'offline access' permission set on it, so we can keep using the same session key even when you're signed out of Facebook - it lasts until you 'disconnect' us (remove us from your FB applications listing).

So, all we need is a table of (facebookUserID, facebookSessionKey, expires, ...) to store all our cached Facebook data. We can run a job every 10 minutes or so, and for any entry that's approaching the 24 hour limit, we wipe all the data except the user ID and the session key. When the profile data is needed again, we go and refetch it from Facebook. Simples.

A similar approach is taken with LinkedIn - using OAuth, rather than a proprietary platform - and will be taken with OpenID, using some of the extensions to the standard. There are no explicit privacy policy concerns with OpenID, but another benefit of doing this is that we'll automatically be synchronising our data with external sites - so if you change your profile information on your OpenID site, it'll update here too.

What's in a name?

One of the problems this is going to make much more acute is duplicate names. At the moment, it's no big deal to ask every user to pick a unique nickname, but if you're coming from Facebook or LinkedIn then it's much more natural to just use your real name. But we can't ask people to pick unique real names! What happens when two John Smiths both come to use the site?

Also, plenty of users won't want to go by their real names. Just because you've come to the site from Facebook or LinkedIn doesn't mean you're happy advertising who you are.

The end requirement is that we want every user to have a unique 'display name,' which can be constructed from their first/last name, their nickname, or a combination thereof. The rules will be something like:

  1. Offer the user the option to display their real name. If they turn it down, they have to pick a nickname that doesn't match any of the existing display names, and the nickname will be their display name.

  2. If they enter their real name, and there's no other user with that real name, their real name can be their display name, and a nickname is optional.

  3. If their real name is already in use as a display name, then they have to pick a nickname that will cause their display name to be unique.

Going by both real name and nickname will probably be displayed like:

Richard "Superpig" Fine

while going by real names or nicknames would just be what you'd expect - "Richard Fine" and "Superpig" respectively.

Cobbling bits together

A further complication is that the user might get some of their profile information from the external site, but not all. LinkedIn, for example, doesn't provide any kind of email address. And what if the user wants to present a slightly different identity on GDNet? Maybe they go by 'T-Bird Smith' on Facebook, but they'd rather go by the slightly more professional 'Tom Smith' on GDNet.

Enter the 'profile map.' The map specifies, for each field of the profile, where it comes from: LinkedIn, Facebook, GDNetV4, GDNetV5, and so on. Whenever the site needs to load somebody's profile into memory, the accounts service begins by fetching the profile map, and then the necessary LinkedIn/Facebook/V4/V5 database rows, combining fields across them to populate the user profile data structure. (This structure is then cached in-memory to avoid having to assemble stuff from the DB every time).

Here comes the new stuff, same as the old stuff

One other thing about this architecture is that it finally answers the question of how to handle existing (V4) user accounts: they just get treated like another identity provider, same as Facebook or LinkedIn. At some point we'll convert every V4 account into a V5 account, but treating it like an external identity provider for now will make it very easy to run the two sites side-by-side until that time.

V5: What I've been working on recently

Posted by , 02 March 2010 - - - - - - · 600 views

Well, I could tell you, but maybe it'd be easier just to show you.

(Let me know if you get any errors out of it. I'm aware of two issues at the moment: one, that ads don't load in IE; and two, that sometimes a page displays a generic 'something went wrong' message which goes away when you refresh. I'm fairly sure the second is something to do with an idle timeout somewhere because it only happens after nobody's touched the pages for a bit).

More to come.

EDIT: Here's another one.

Service process account install gotcha

Posted by , 11 February 2010 - - - - - - · 281 views

Here's a little something that had me stumped for 15 mins. The info on the net about it is pretty sparse so maybe this will help somebody.

I was trying to install the GDNet service processes on the backend server. Every service process needs its own user account - it makes security, auditing, and SQL Server access a lot neater. Normally when you install a service process that uses a user account, you get prompted for the username and password of the account the service should use. I want the installs to be unattended, so I hardcoded the usernames and passwords into each service process:

public class RendererServiceInstaller : System.Management.Instrumentation.DefaultManagementInstaller
public RendererServiceInstaller()
var process = new ServiceProcessInstaller
Account = ServiceAccount.User,
Password = "xxxxxxxxxxxxxxxxxx"
var service = new ServiceInstaller
DisplayName = "GDNet V5 Rendering Service",
Description = "Service that renders GDNet XML into XHTML for output to users.",
ServiceName = "V5Renderer"


var evtLog = new EventLogInstaller {Source=RendererService.EventLogSourceName, Log="GDNet" };

When I tried running InstallUtil to install the service, though, I got this error:

System.ComponentModel.Win32Exception: No mapping between account names and security IDs was done

I'd granted the accounts in question the 'Log on as a service' and 'Log on locally' rights, I could start processes as them by hand over remote desktop, so what was the problem?

Look at the username I'm using: it's a domain account, so it's in the form DOMAIN\accountname. Look at what's separating those two components. It's a backslash. Backslashes are special in C# (and many other languages). As far as the compiler was concerned, the account name wasn't 'GDNET\v5_render', it was 'GDNET', then a vertical tab, then "5_render".

Sticking an @ on the front of the username string has fixed it.

V5: Fun with MSBuild

Posted by , 05 February 2010 - - - - - - · 464 views

Now that Pulse is churning away at the codebase, I've spent time today doing further tidying of the build and deploy process. Once the site goes live I want to be able to get changes deployed quickly and safely, and I want to be able to start deploying builds for the Staff to look at within the next few days, so I'm doing what I can to get the pipeline right now. Fortunately, the tooling in this area is all really pretty good.

The first problem is versioning. As I mentioned in my last journal entry, I wanted to get MSBuild to stamp all my executables (I should have said 'assemblies' because a lot of these are libraries) with the build number. When dealing with rapidly-changing, opaque binaries spread across multiple computers, being able to ensure that your files are in sync is critical.

MSBuild, the standard build engine used for .NET projects, is highly flexible and extensible; it's very easy to just drop in new kinds of build task and add them to your project file, and best of all, Visual Studio is fine with it - it can't show them in the UI most of the time, but it can respect them, execute them, and generally not screw them up while working with the project file like normal. There's also a lot of drop-in build tasks freely available all over the net. For this, I'm using the AssemblyInfoTask (though I may upgrade to the MSBuild Extension Pack). The task takes the necessary version parts - major, minor, build, revision, plus any of the product/company name, copyright info etc that you usually find in a Win32 file version resource - and updates the project's AssemblyInfo.cs file with them prior to build. That's a little skeevy as it means the AssemblyInfo.cs file - which is under SVN - keeps getting local changes, but I can live with it. I've written a .targets file that incorporates the task just before the main compile phase, like so:


<AssemblyInfoFiles Include="**\AssemblyInfo.*" Exclude="**\.svn\**"/>

<UsingTask AssemblyFile="bin/Release/AssemblyInfoTask.dll" TaskName="AssemblyInfo"/>


<Target Name="UpdateAssemblyInfoFiles"
<AssemblyInfo AssemblyInfoFiles="@(AssemblyInfoFiles)"
<Output TaskParameter="MaxAssemblyVersion" PropertyName="MaxAssemblyVersion"/>
<Output TaskParameter="MaxAssemblyFileVersion" PropertyName="MaxAssemblyFileVersion"/>

This could be made somewhat more efficient - I don't strictly need to pull the version bits out into a separate PropertyGroup, for example, and could just write them directly into the attributes on the AssemblyInfo element. Still, it gets the job done. All I then need to do is add an <Import> statement into my .csproj file pointing at this .targets file, and the build step is magically included.

Note how the build and revision numbers are actually variables - PulseBuildNumber and PulseSvnRevision. I'm passing those in as arguments to MSBuild when I launch it. You can do this on the command-line using the /p switch, though because I'm using Pulse, it's actually got an XML config file that I use to feed inputs to MSBuild:

<msbuild build-file="V5.sln" configuration="Deploy">
<build-property name="PulseBuildNumber" value="${build.number}"/>
<build-property name="PulseSvnRevision" value="${build.revision}"/>

$(build.number) and $(build.revision) are, in turn, built-in variables defined by Pulse whenever it launches a build. See the data pipeline!

I had a good question from @naim_kingston on Twitter, asking why I use both the build number and the SVN revision number - aren't they redundant? In theory, yes; I should only need the SVN revision number, and then should be able to check out that revision of the code, build it, and always get the same result. In practice, though, I might not always get the same result because there are elements of the environment that may have changed. For example, maybe I'm using a different version of the compiler, or of the build tasks library. Storing the build number as well allows me to more quickly correlate a particular binary to its entry in Pulse's build log, so I can very quickly go to Pulse and download the right .pdb files, MSBuild output files, and so on, and always be confident that what I'm getting is from exactly the same build, rather than just one that used the same code.

So, that's got versioning sorted. I need to add the <Import> element to more of my project files, but I've got the main service projects covered for now. I'll add more as I go along.

Next, app.config files. It's common to want to change stuff in these files, such as the address at which a service can be found (e.g. from "db-server.gamedev.net" to "localhost"), but changing the app.config file directly means you have to remember not to check it into SVN, and it's kinda pesky to have it always showing up as 'modified' in the Pending Changes window. What would be better would be if I could have a second file of 'local overrides' that should be used in preference to the app.config file, falling back to app.config for stuff I don't care to change.

MSBuild to the rescue once more. This time I've used the MSBuild Community Tasks, which includes a task called "XmlMassUpdate" - given two XML files, it takes the nodes from one, and adds, inserts, or replaces them into the other. There's also some custom attributes for removing nodes from the target file. Another .targets file integrates the task into my build pipeline, and presto: I have an app.local.config file in each project, svn:ignored to stop it from pestering me, that MSBuild neatly integrates on every local build.

The next challenge I face is how to get each service from a ZIP file in Pulse to a correctly installed and registered presence on the relevant server. There's more to this than just XCOPY - most of the services need to be registered as Windows Services, have event log sources and WMI classes created, etc - and ideally it should happen without me logging into each machine by hand and copying files around + running InstallUtil. The answer is probably going to be to build MSI files. Anyway, that's for later. For now, I'll sleep.

V5: Continuous Integration and Deployment

Posted by , 04 February 2010 - - - - - - · 254 views

I spent a bit of time recently doing some work on V5's build pipeline, implementing continuous integration and making the deploy-to-servers process a bit more formal. Unlike most web developers, I'm a big fan of pre-deployment testing and verification, so a well-established build process is a key part of that.

Continuous Integration, for those who aren't familiar with it, is the simple idea that your code should be continually being built. Every change you check into source control should get compiled, packaged, and tested on all your target platforms - automatically, of course. It's a great way to catch build errors in other configurations or on platforms other than the one you're developing on.

Many people go for CI servers built around CruiseControl, but after researching the options when I was back at NaturalMotion, I selected, used, and fell in love with Zutubi Pulse. So, it's now running on GDNet, a nice complement to our issue tracker and source control system.

Pulse is great. It's got an easy-to-understand but elegant and powerful web UI, built-in support for a bunch of external build systems (such as MSBuild), it's trivial to install... but the best thing, really, is the support. Zutubi is, as far as I can tell, two guys in Australia - Jason and Daniel. Yet, between them, forum questions get answered within minutes, with detailed and helpful responses; feature requests get logged and show up in a point release a week later; their JIRA instance is publicly accessible; and they still, somehow, find time to blog about build systems, agile programming, unit testing, and so on. If I ever meet these men, I am buying them a drink. Each.

Two further things that are more relevant to the average GDNetter: Firstly, they have free licenses available for open-source projects and for small teams (2 people / 2 projects), and secondly, I'm told they've got a number of game developers as customers... so they've got quite a lot of familiarity with our use-cases, and Pulse handles things like '4GB of art assets' pretty well. I'd definitely recommend checking Pulse out if you've got the hardware to spare.

The other nice thing about having a CI server is it provides an authoritative 'release provider' within the GDNet LAN: a clear, single source for new releases of the site software to be deployed to our machines. I've done some work tonight to have Pulse capture the executables and content directories as zip-file 'artifacts;' next I'll get MSBuild to actually stamp the executables with the build number, and I'll look into ways to quickly and efficiently deploy the artifacts to the machines that need them. Eventually, doing a new release of the GDNet site will just be a question of clicking a 'trigger' button, and watching the progress bar tick for a bit [grin]

Recent Entries

Recent Comments