Sign in to follow this  
johnnyk

Boost/STL in games?

Recommended Posts

Do the Boost and STL libraries have a place in console games? I've been exploring the Boost library (www.boost.org) and find it pretty fascinating, but of the three game companies I've worked for none have ever used even the STL, making it pretty unlikely that Boost will be used - usually the reason is due to poor PS2 compilers that wouldn't support the template specification needed, or problems with the memory taken up by the stl (which seems like a cop out since it is customizable). Has anyone shipped a game on a console using either of these libraries? It would certainly be nice to not have to learn another dynamic array interface again, thats for sure :) Let alone the myriad of things boost can do. -John

Share this post


Link to post
Share on other sites
Quote:
Original post by johnnyk
Do the Boost and STL libraries have a place in console games?

I think it's safe to say they're frowned upon because they are over-engineered, have difficult-to-predict memory behaviors, and give obscure compiler errors that do not facilitate easy debugging.

Share this post


Link to post
Share on other sites
Quote:
Original post by Christer Ericson
have difficult-to-predict memory behaviors


Which is why you can write your own allocator for them, plus I'm pretty sure they have well defined charactistics when it comes to memory useage.

Quote:

and give obscure compiler errors that do not facilitate easy debugging.


Thats just a lack of familarity with the error messages templates throw at you, which will big, arent that hard to understand if you apply a bit of brain power to them (infact, often the key to the error is within the first few characters or the last few).

Share this post


Link to post
Share on other sites
We're using the STL in our Nintendo DS project, and the DS isn't the most powerful machine out there. And we're not even messing around with custom allocators.

It's part of the language, so long as you use it properly there's no problems with it.

Share this post


Link to post
Share on other sites
Quote:
Original post by phantom
Which is why you can write your own allocator for them, plus I'm pretty sure they have well defined charactistics when it comes to memory useage.
On a continuously streaming game like God of War we need 100% deterministic memory behavior; "pretty sure" doesn't cut it.

STL is fine for tools, of course, where we don't have such strict requirements.

Share this post


Link to post
Share on other sites
Quote:
Original post by Christer Ericson
Quote:
Original post by phantom
Which is why you can write your own allocator for them, plus I'm pretty sure they have well defined charactistics when it comes to memory useage.
On a continuously streaming game like God of War we need 100% deterministic memory behavior; "pretty sure" doesn't cut it.

STL is fine for tools, of course, where we don't have such strict requirements.


He said "pretty sure" in the context of the default allocator having well defined characteristics -- which is why he suggested custom allocators as an alternative. If you use a custom allocator, you can make the STL containers' memory behavior as predictable as you like. Take a look at Boost.Pool (and pool_alloc in particular), for an example.


jfl.

Share this post


Link to post
Share on other sites
Quote:
Original post by Christer Ericson
On a continuously streaming game like God of War we need 100% deterministic memory behavior; "pretty sure" doesn't cut it.


You have the source, it's easy to figure out how the library implementation you are using does things. It's not like there are trade secrets in there. Now (and please don't take that as a personal attack), that assumes you're able to read code and grok the conventions used.

It's not as if the library relies on rand() to decide whether to allocate more memory for its container or not. And while you're at it, you might decide to modify (*gasp*) your standard library, if you don't like, say, capacity doubling on vector reallocation. But this assumes a level of comfort with the library that I don't expect to find in many places, considering how much balking there seems to be about simply using it.

Quote:
STL is fine for tools, of course, where we don't have such strict requirements.


Agreed. That's a context in which there shouldn't be any objection except, as usual, unfamiliarity.

Share this post


Link to post
Share on other sites
100% great.

Pretty much what phantom said.

You want deterministic memory usage... write your own allocator. Why reinvent the wheel? IMHO the best thing about the STL and to a lesser extent Boost is that it is a standard. Learning specifics to an engine or company because they've decided to roll their own rather than use something most people should already be familiar with is crazy. The exceptions occur when you want to be doing something REALLY specific or are working with a legacy codebase.

Quote:
Original post by BlueHabu
I don’t know what "continuously streaming" means, but I am sure you have done some performance analysis of the STL on the PS2. Care to share?


I'd be very suprised if he didn't mean continuously streaming in content from DVD, HDD or similar.

Share this post


Link to post
Share on other sites
Absolutely we use STL on console games. The main problem of lots of small allocations is trivially dealt with by writing your own allocator. The neat thing about the STL is that you can guarantee that not a single byte is allocated without you knowing about it and being able to change its behavior.

-me

Share this post


Link to post
Share on other sites
Quote:
Original post by meh
You want deterministic memory usage... write your own allocator. Why reinvent the wheel?
What I tried to point out, without having to go into a lot of detail is that we have very strict memory use (see below) and trying to coax something to adhere to our rules just isn't worth the effort especially when we can write a version that suits our needs in half the time it would take to "fix" some other system.

Quote:
Quote:
Original post by BlueHabu
I don’t know what "continuously streaming" means, but I am sure you have done some performance analysis of the STL on the PS2. Care to share?
I'd be very suprised if he didn't mean continuously streaming in content from DVD, HDD or similar.

Right. And for a game that streams data continuously for, say, 15 hours of gameplay, you need memory use to be fully deterministic so that we don't have scenarios like on one of the thousands of different paths a player make take through the game, the code runs out of memory 5 minutes before the end of the game.

The difference from e.g. level-based games is that there is no point where you can "flush" your state (such as inbetween levels); we have a persistent state that is unique across the whole game. At no point should it be possible to run out of memory due to fragmentation or similar.

But to each his own.

Share this post


Link to post
Share on other sites
Quote:
Original post by Christer Ericson
What I tried to point out, without having to go into a lot of detail is that we have very strict memory use (see below) and trying to coax something to adhere to our rules just isn't worth the effort especially when we can write a version that suits our needs in half the time it would take to "fix" some other system.


To be honest it all really boils down to familliarity with the STL and somewhat of a religious issue. It's trivial to "fix" the STL for games(console or otherwise) w/r/t memory management/fragmentation if you know what you're doing. Even then though, there are a lot of people who just won't use the STL either due to not being familliar with it or because of something akin to whether or not an opening curly brace should be on a newline or not.

The bottom line is preference. If you like the STL and know it well and want to use it for console games it's trivial to "fix" it. If you just don't like the STL or aren't really familliar with it, then it's easy to write your own container. There are zero technical reasons that you cannot use it. It's just an OpenGL v. DirectX religious war.

-me

Share this post


Link to post
Share on other sites
Quote:
Original post by Christer Ericson
Quote:
Original post by meh
You want deterministic memory usage... write your own allocator. Why reinvent the wheel?
What I tried to point out, without having to go into a lot of detail is that we have very strict memory use (see below) and trying to coax something to adhere to our rules just isn't worth the effort especially when we can write a version that suits our needs in half the time it would take to "fix" some other system.


This is assuming that writing/debugging/mantaining an appropriate STL allocator takes twice as long as writing/debugging/mantaining an appropriate non-STL allocator plus the special containers (plural) that use it (not to mention any bugs caused out of unfamiliarity with project P's containers by newer teammates).

Which sure dosn't make sense to me unless the allocator interface is badly incompatible with whatever you're trying to accomplish. And I dare say that I don't think that it is for preventing memory fragmentation, or similar.

There will be exceptions, of course, if you're more familiar with what you may have repeatedly written, than you are with the STL, but that is a matter of personal experience rather than STL being somehow harder in general.

Share this post


Link to post
Share on other sites
Quote:
Do the Boost and STL libraries have a place in console games?


Can be done, and is sometimes being done. However, on a console, you have to really know what's going on in your code, and knowing what's going on with malloc() and free() is a whole lot simpler than knowing what's going on with a few dozen highly templatized classes. Thus, the learning curve is high, and adoption low.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Jesus Christ...I am truly amazed with the things I see posted on this forum.

Christopher is, and I quote from his site, Director of Tools and Technology for internal development at Sony Computer Entertainment America in Santa Monica, and you people are...who exactly? Someone called phantom, another called funny... Learn to show some respect to the TRUE professionals of the industry that you claim to support and STOP BEING ELITISTS.

To Christopher

Don't waste your time with these bozos. You are a professional who speaks eponymously and civilly on the Internet, both of which are your credit. It seems to me that some of these guys are just hoping to "debate" with someone of your STATURE so that they can tell their friends at work (i.e. tech support offices) how they PAWNED a professional of the game industry. I remember there was another thread where another author, like yourself, debated with some of these guys and ended up being insulted multiple times (sarcastic comments and the like). I am appalled...

Share this post


Link to post
Share on other sites
He is providing great help, and although your right, not everyone is a professional like him, or probably as intelligent as him, people are just giving their opinions and thoughts. Maybe they learned things wrong or their opinions are skewed, don't go off on them, and he's not wasting his time. These forums are one of the greatest assets to up and coming and recreational game developers. They are an amazing resource, and he is helping out in conversations that hopefully people can learn some things from, I know I did, and it helps, so don't say he shouldn't be in here discussing these issues. People take them seriously and his input helps us learn what is the best way of doing certain things and just helps people learn things to a more full extent. So thank you Mr. Ericson, cheers,

-Karakadin (a current CS major who really appreciates you being active in the forums / community)

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous Poster
Jesus Christ...I am truly amazed with the things I see posted on this forum.

Christopher is, and I quote from his site, Director of Tools and Technology for internal development at Sony Computer Entertainment America in Santa Monica, and you people are...who exactly? Someone called phantom, another called funny... Learn to show some respect to the TRUE professionals of the industry that you claim to support and STOP BEING ELITISTS.


This is commonly referred to as the fallacy of authority. That is, somebody's being an authority doesn't make them right, although it certainly indicates that such might be more likely. In any case, people are pointing out a number of valid contradictions to the assertion that STL/boost are useless on consoles, so citing authority isn't going to cut it as a rebuttal. And yes, some of us might just be professionals as well.

In any case, I think the first question that has to be asked is WHAT PARTS of STL/boost are in question? It's not like they are all or nothing propositions--you can choose to use as much or as little of them as you want. Are we talking iostreams? In my opinion (just mine mind you) those barely even cut it for PC games, let alone consoles... Perhaps collection classes? Well, that depends on the collection. std::vector, for example, tends to have extremely deterministic behavior--much moreso than, say std::map (which again shouldn't really be used much in PC games either). As far as boost goes, some of its classes are specifically designed to PROMOTE deterministic memory behavior. Memory pools anyone? On the other hand, I certainly wouldn't try to do any sophistocated template metaprogramming (boost::mpl) in a console game (but again that pretty much applies to PC games as well IMO). And on top of all that there's the potential, as has been mentioned by other posters, to write custom allocator classes to get complete control over memory usage.

To put it bluntly, there are a good number of reasons that a categorical rejection of standard libraries as being universally useless on consoles is, shall we say, somewhat misinformed. I haven't done any console programming per se, but I have used such techniques in other embedded systems programming (often on even tighter memory and processor budgets) quite successfully. It's just a matter of putting in a bit of time and effort to learn the inner workings of the library in question (not difficult because source is obviously available) so you KNOW what's going on. Even then if you don't like it, you can always change the parts that don't do what you want rather than reinventing the wheel.

But please, appeals to so called "authority" will not avail you without meaningful argumentation to back them up...

EDIT: I think the original poster hit the nail on the head anyway. It's not so much a matter of template libraries being ill suited to consoles, it's a matter of compilers for consoles--at least those I've seen--being incredibly ill suited to template libraries. They just don't tend to be anywhere near as sophistocated as their desktop brethren.

Share this post


Link to post
Share on other sites
Quote:
Original post by Shinkage
But please, appeals to so called "authority" will not avail you without meaningful argumentation to back them up...


Unfortunately, NDAs tend to preclude such argumentation.

Share this post


Link to post
Share on other sites
Quote:
Original post by Fruny
Unfortunately, NDAs tend to preclude such argumentation.


You misunderstand me. It's not about proving your CV, it's about proving your argument, which no amount of demonstration-by-authority-fiat will accomplish. Not that I'm saying he's wrong, he has valid points to be sure (as does the opposition), but don't accept them "because he's an authority."

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous Poster
Jesus Christ...I am truly amazed with the things I see posted on this forum.


What, you mean a debate on a subject in which intelligent professionals can believe or support differing viewpoints, where they can discuss the logic behind those positions and ponder their stengths, weaknesses, and their effect? I'm amazed too, that such can exist on the internet, in a condensating world where people like you merely write off the opposing side as not being worth talking to.

Unlike you... me and those replying to this topic, we wish to engage in this discussion not because Mr Ericson has a large epeen, but rather to see what he has to bring to the table of discussion (which he has joined by his own volition). We wish to both teach and learn. Telling him what to do, to ignore us, is disrespectful to both us and him. He is more than capable of making that decision himself if he so chooses.

Share this post


Link to post
Share on other sites
Quote:
Original post by Palidine
To be honest it all really boils down to familliarity with the STL and somewhat of a religious issue. It's trivial to "fix" the STL for games(console or otherwise) w/r/t memory management/fragmentation if you know what you're doing. Even then though, there are a lot of people who just won't use the STL either due to not being familliar with it or because of something akin to whether or not an opening curly brace should be on a newline or not.

The bottom line is preference. If you like the STL and know it well and want to use it for console games it's trivial to "fix" it. If you just don't like the STL or aren't really familliar with it, then it's easy to write your own container. There are zero technical reasons that you cannot use it. It's just an OpenGL v. DirectX religious war.

-me


I believe this hits the crux of the matter pretty accurately.

Also Shinkage makes the very valid point that you don't have to use the entire library.

The AP should have the balls to state who he is before he goes ahead blathering about position.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this