Jump to content

  • Log In with Google      Sign In   
  • Create Account


lawnjelly

Member Since 20 Mar 2012
Offline Last Active Jan 23 2014 05:34 AM
-----

#5067111 Difference between software engineer and programmer

Posted by lawnjelly on 03 June 2013 - 09:15 AM

Job title inflation, imo.wink.png

 

http://www.bbc.co.uk/news/magazine-18983009

 

It's like calling a toilet cleaner a 'sanitation engineer'.




#5034102 Game engine Memory Manager

Posted by lawnjelly on 19 February 2013 - 04:47 AM

p.s.

 

 

(don't hate me for using singletons smile.png the engine only uses two) 

 

Two too many.




#5034100 Game engine Memory Manager

Posted by lawnjelly on 19 February 2013 - 04:44 AM

In my experience, tight memory management can be incredibly useful in certain situations, on certain platforms, and less necessary in others.

 

Keeping tight control of memory is particularly useful on things like consoles, devices with limited memory, and especially situations where there is no swapping to disk to page in 'extra memory' when you run out. In these situations, if you run out of memory, either your application handles it gracefully (says 'cannot do this' or whatever) or you get a crash.

 

Certain applications, like games and rocket control software, or plane autopilots, I see as more 'mission critical' so I don't want them to fail or crash under any circumstances. Whereas for e.g. a word processor, it is more acceptable when trying to load a document if it says 'cannot load document, not enough memory on this device' (although obviously you'd try and design to prevent this happening). But playing a game it's no good if it says 'cannot load level 5, out of memory', as you cannot progress in the game.wacko.png

 

So for games that are anything other than very simple ones, myself I would tend to use a memory manager. However, for general applications / editors etc which have to adapt to what particular document / documents they are editing, where they are allowed to 'fail' due to out of memory errors, I'm much more likely to just use directly or indirectly the OS allocators.smile.png

 

If you do preallocate blocks of memory for each of your game 'modules', you are right in saying it is useful in advance to know how much memory to allocate. Preallocating blocks for different modules can be very useful when you need to work to a memory budget, particularly with a team of programmers, rather than just putting it all together and 'hoping it doesn't run out of memory'. For some areas, this will be easy to workout (e.g max number of sound buffers, things like that). 

 

For others, particularly game level resources, the memory requirements may change from level to level. You may want certain levels to have more sound data than others, some more texture data, etc etc. However, a way around this, rather than having set limits for sound data / textures / geometry etc, is to have these data shared in a 'level file'. And have a certain maximum size for your level file data.cool.png

 

For tracking memory leaks, as the others say, just because you are using your own allocator it doesn't automatically 'fix' leaks. However, you should design your allocator so that along with the allocation it can store things like the line number and filename (in some type of debug build). Then on exit, you can report any allocations unfreed after cleanup, and other statistics, like the maximum memory used in each module etc.

 

You can also put 'check' regions of a few bytes around allocations, to detect when you have written outside of bounds, off the end of arrays etc.ph34r.png

 

There are also 3rd party systems you can use for most of this leak detection and bounds checking. Although these may not be available on your target platform .. so having your own can be very useful. It's the kind of thing you can write once and reuse in other projects, and great to have in your 'toolbox'.




#5030400 pointers while serializing

Posted by lawnjelly on 09 February 2013 - 10:32 AM

You can also store the pointers as offsets from the start of a structure.

 

Then when you load this structure into memory, you can 'fixup' the pointers by adding the offset to the actual memory address of the start of the loaded structure and saving the result back, and voila, your pointers are valid again.

 

0 Start of structure
1 Pointer to Chicken (offset 3)
2 Pointer to Duck (offset 6)
3 Chicken
4 ..
5 ..
6 Duck
7 ..
8 ..

 




#5010964 "Correct" Music Note Structure?

Posted by lawnjelly on 15 December 2012 - 09:43 AM

More stuff:

Note pitch: I'd stick with just a note number like MIDI for now, and the 12 note western scale. 99% of music is written like this, and handling other systems is a bit more advanced and something you can tap on later. Storing notes as float frequencies I wouldn't recommend for several reasons : accuracy (say you transpose down, then up later) .. the wavelengths don't have a linear relationship with note number. You might want to do operations based on the relative pitches of notes, or detect chords etc. All of this would be stupidly difficult just trying to store wavelength / frequencies. Besides the fact your source instruments may have different base frequencies anyway and these would need to be compensated for.

Pan: Why limit yourself to stereo pan? What about surround sound?

Channels / instrument info on a note: Would you want the note to determine this, or the track and / or pattern? Having a 'grouping' feature for notes can be useful though. Remember you are going to want to be able to do stuff like edit the instruments you are using quickly and easily, and not change this for every note.

What happens when by accident you set 2 bunches of notes to the same instrument ID (if storing on the notes?) you have then lost their 'individuality'. Better to store something else that then maps to the instrument.

Volume: This is usually key velocity rather than volume (there is midi volume as well, but you wouldn't store this per note, but as a separate event), which in midi is 0-127. There is also release velocity, which may or may not be used by the instrument.

There's also other stuff like pitch bend, aftertouch etc, which you can store as a separate event.

Note name / ID: Why try and store this on the note? If your pattern has e.g. an array or vector of 35 notes, then you know its ID as you access it.

An example to start with might be something like this:

class Note
{
public:
int m_iStartTime; // in PPQN. this could be negative? if you want some notes to start before the official start of pattern
unsigned int m_uiLength; // in PPQN
unsigned int m_uiKey; // e.g. like MIDI have middle C as 60
unsigned int m_uiVelocity; // 0-127?
unsigned int m_uiReleaseVelocity; // 0-127?
};

Once you have a simple system working then it will become more obvious where to add things.

To reiterate on the notes side of things, don't worry so much about space saving, just concentrate on simplicity. Note data doesn't tend to be that large. It's more when you get to the audio side you need to pay attention to the data structures / bottlenecks.

And rather than just having a struct-like class you can use accessor functions so the actual data underneath can be anything you want.


#5010955 "Correct" Music Note Structure?

Posted by lawnjelly on 15 December 2012 - 08:55 AM

Bit of a brainfart here, but hopefully something useful: Posted Image

As L. Spiro says, don't store you times as floats or something like that, store than as ticks.

Sequencers commonly work on a scale of PPQN (pulses per quarter note), so if you adjust tempo (once off or gradually throughout a song) it just *works*. The PPQN values are usually things like 48,96,192 etc.

Bear in mind that if you are doing 4/4 music that's all good, but if you are using triplets, or groove, you'll want the PPQN divisible by 3, and with enough precision for your 'groove'.

You'll also probably want to store your note timings as offsets from the start of a pattern, rather than the start of the song. This way you can several instances of the same pattern at different parts in the song.

Also instead of storing things as e.g. char[3] to save space, it's probably more sensible just to make them 4 byte unsigned int / ints and keep your structures 4 byte aligned so you (or the processor) aren't faffing about for no reason. You can always compress them on import / export, if you really need to.

Another reason for PPQN is so you easily change the output sample rate (assuming you are going to do some audio instead of purely MIDI).

I've done several audio / sequencing apps and don't think I stored anything as floats. PPQN can be used to calculate the exact sample for an instrument to start / end (and you might precache this kind of info). You could possibly use something more accurate to get within sample accuracy for timing, but I've never bothered myself.

It's really worth using a plugin architecture for different components of a sequencing / audio app, I'd highly recommend it. You can make effects (reverb, delay, chorus etc) plugins, and instruments plugins. You could potentially also use VST plugins or similar if you can work out their interface (you may find some open source apps that have managed this).Posted Image

I'm currently rewriting a sequencer / audio app I wrote a few years ago, and have actually moved to using plugins for things like quantization / legato / groove / argeggios. Have a think about whether you want to be able to do stuff like 'undo' quantization, keep original values, or have a modification 'stack' applied to notes.

I don't think you'll get the exact structures bang on first time, it's the kind of thing you write a first version, then realise there's a better way of doing it, redo it, etc etc. But it is fairly easy to get something usable. You may also spend as much time on user interface / editing features as the stuff 'under the hood'.

As for APIs, I have so far cheated and don't actually use MIDI input or output (although I have done that in the distant past and it wasn't that difficult I don't think). I have just been writing a MIDI file importer though refreshing my memory lol.

If you want realtime MIDI input you'll have to pay much more attention to latency and the APIs you use. I was just getting by with the old Win32 audio functions for primary / secondary buffers, but the latency is awful, so using direct sound or I think there may be a new API in windows 7 would be better. Sorry can't help yet in that as I haven't researched it myself yet.

Also I'd add, consider using direct3d or (in my case) opengl to accelerate the graphics side. This way you can easily show the position within a song without overloading the CPU and causing stalls and having your audio stutter.

Once you start doing the audio side a bit of SSE / SIMD stuff helps. And you have to think carefully about how you'll structure your tracks / sends to effects, to make it efficient but also customizable.


#5008463 A game made solely to tell a story

Posted by lawnjelly on 08 December 2012 - 05:27 AM

Sure you can do this. There is a continuum between zero user choice audio visuals, say a 3d movie like toy story, and full free choice games. A lot of game artists also work on movies.Posted Image

When there's no user interaction, there's a lot of shortcuts you can take. Objects may only need to be built with one viewing angle for instance. You don't tend to need physics representations in the same way. And the whole thing can be prerendered with smoother curves and effects. On the other hand the detail expected tends to be higher for movies.

A lot of games have scripted elements. This would be more akin to a movie sequence. Game designers have to choose how scripted a game is (which is good for storytelling), and how much choice there is. Sometimes it can be difficult to force the intended story without giving the player the impression that the game is linear.

As the others say, there is a cutoff point, where your game is so linear, that you might as well make it pre-rendered or a movie, so you can take advantage of those techniques.


#4985394 Should I give up?

Posted by lawnjelly on 30 September 2012 - 10:33 AM

No matter what language you use, there is always something that has to be distributed with it.

With C/C++ programs, you'll have to distribute the proper library DLLs and make sure the user has the right version of the C/C++ runtime installed. Then there are video card drivers. The DirectX user redistributable, etc...

So nothing will change. Just the name of the language!


I see what you are saying, but I'm not sure it's strictly always true, it depends on the type of game / app you are writing, and the market (think e.g. casual games, apps, versus AAA games).

If you write c / c++ and statically link to the runtimes, there are no dependencies due to the language, only core OS dlls (which will always be present).

The need for more and more sprawling dependencies is up to you, your choice of language / tech and what third party stuff you decide to pull in. (Sometimes you can statically link to third party stuff though).

I think azonicrider is right to an extent .. end users are easily put off installing stuff. If your game needs a 15 minute download from a third party and separate installation just to run, they'll probably move on. I know I do. If I see a 'your java needs to be updated' or flash or whatever, I'm like 'forget it'. So there is a good argument for considering your market before deciding what dependencies to rely on.


#4985315 Making game art from hand drawing?

Posted by lawnjelly on 30 September 2012 - 04:48 AM

Before you go down the pure pixel approach, also consider that 2d vector art or 3d art can be much easier to animate, once you start thinking about more than 2fps or something.

I personally find 3d packages incredibly tedious and time consuming, but have recently been converted to 2d vector animation. I'm currently using Anime Studio, but SynFig is another free option.

If you build your art out of vectors, you can then rescale it, animate it (with bones etc), easily change colours etc down the line without needing to do loads of extra work.

As for creating the artwork from line drawings, there is PoTrace in Inkscape and a few other options. In the end I wrote a little app for doing centreline autotracing (from e.g. a scan or photo) which is a bit more what you typically want for easy editing:

http://www.facebook.com/doodletrace

The results still require cleanup though, so I tend to use a combination of this and mouse vector editing in Anime Studio.

End result is something like this (character in little 2d game I am currently making) :

Posted Image


#4983547 Client-Side Prediction & Packet Loss

Posted by lawnjelly on 25 September 2012 - 06:52 AM

A video would be quite useful if you have fraps? It's always going to be a difficult one though. Player interactions is where client side prediction breaks down...Posted Image

As a caveat I haven't personally dealt with this type of scenario, but here's some guesses:

When it does break down I believe it is usually best to rely on the server being authoritative. If you 'see' on the client you've planted a bomb and trapper another player, but as far as the server is concerned, that player has already moved out of the way (their input is ahead of yours, for example), you have a choice, you either let the server be authoritative and you correct your client, or you wind back the server, and say 'hey this client made a hit' and replay everything from there and send the result to everyone.

I personally wouldn't want to wind back the server, it strikes me as a bit of a nightmare in terms of balancing, potential for cheating, etc .. (what happens if there's a stutter on the client, and the other player is not updated, and thus easier to trap?). Posted Image

So with an authoritative server you are going to get situations where either the trapper or the trappee is corrected (or both), and it's going to look kind of annoying, like in a FPS when you get shot round a corner or snapped back. Whether this totally mucks up gameplay is a question - some games maybe are not suitable for this sort of multiplayer I guess. You'll just have to try it and see.Posted Image

It maybe something that works as a splitscreen game (with zero lag), but totally falls down when playing with lag.

The other thing you could do if all else failed, easy solution - don't use client side prediction lol. Then the problem is solved. It just means you'd have to either play with fast connections, or lan games, or try and hide the delay somehow in the gameplay.Posted Image

There may also be other possible solutions with client-client communication, but I don't know much about that.


#4983213 Client-Side Prediction & Packet Loss

Posted by lawnjelly on 24 September 2012 - 07:28 AM

I'm toying with sending inputs in a sort of reliable fashion. It's a queue of inputs, and I keep sending them until they are acknowledged. I also have a bandwidth throttle to keep the bandwidth usage in check.


This is a fairly standard way of doing it, I've been doing it for a while, there's a gaffer on games article describing this I think:

http://gafferongames.com/

I think (without looking at the code) I send the server calculated position with the ack for the client input. In fact this may all be part of the regular update packet from server to client (with actor positions).

Then when the client gets the ack, it can compare the server calculated position with its own client side predicted, and if it's different, roll back and recalculate the client position based on the input history. And once it has the server ack up to tick 'blah' it only now needs to send input out from tick 'blah' onwards, etc etc.


#4983144 What programmers want from a designer

Posted by lawnjelly on 24 September 2012 - 02:31 AM

A question out to all the code-heads out there. I'd like to get a sense for what you as a programmer like to see out of designers other then cold hard cash to spend a few dozen/hundred hours on writing code. Obviously a good idea is a pretty useful thing but lets get beyond that. When you're reading the classifieds you're looking for something out of a team or individual to instill a sense of commitment and components to indicate a unified design idea but what are those components. In a priority list what matters to you? Art, documentation, pre-recorded audio, etc?


I can't speak for other programmers, particularly beginners (to whom this question might be more relevant).

Being a 'designer' is a very difficult sell to try and start a project yourself. Unless you have prior experience / track record, realistically you are probably going to need money to pay, or have other skills (do you have a good looking girlfriend, who is *really* committed to the project?).

If we take money out of the equation, other skills would be (as AlterOfScience says) things like producing artwork / programming, and *possibly* running a business (prior experience). I would have thought producing artwork is most likely to be the successful avenue, as that is what most programmers usually can't do themselves / aren't interested in doing. To be realistic, if you have any chance of being part of a non-paid project doing purely 'designing' (probably writing scripts or making levels), it's most likely to be joining an existing project rather than starting one yourself (and consequently working on someone else's ideas, that's what a designer does 99% of the time).

I would suggest to anyone who wants to get started in designing to either start making games themselves, with a game toolkit or mod tool of somekind that doesn't require programming, or to learn to produce artwork of some kind, either 3d or 2d, so they have some marketable 'skillz' to bring to the game. For indie games, artwork doesn't necessarily have to be AAA quality, you just have to be prepared to spend the time learning how to do it, and spending the hours and hours and hours on it.

In short, designer is a hard sell. Designer / artist (primarily) and to a lesser extent designer / programmer is a better bet.

Obviously a good idea is a pretty useful thing

Not really. Posted Image The game world isn't short of ideas. It's short of people who have the skills / time / commitment to do stuff. But I'm sure this has been discussed ad infinitum.


#4977966 What improves better memory performance?

Posted by lawnjelly on 08 September 2012 - 07:02 AM

stuff


Yup, that's pretty much how we ended up doing it too! Snap lol. Posted Image I think it probably ended up as a malloc on a reserved heap on pc build for the level file, and just loading into the prereserved block on consoles.

For the streaming I think I had more chunks (I called them banks), maybe 8 or 16 something like that, then parts had the option to use e.g. 2 banks worth.

For deciding which banks needed to be loaded I used a PVS calculated from the artist's levels and portals, and a potentially loadable set derived programmatically from this. There were areas though I'm sure where the artists had overcooked it and they needed to put in visibility blocks of some kind. I think the tool chain alerted them to this. Worked a charm, especially with decent asynchronous streaming support.

Getting way off topic though there hehe! Posted Image


#4977922 What improves better memory performance?

Posted by lawnjelly on 08 September 2012 - 01:18 AM


The other is that there is no question over the time taken over a deallocation / allocation. It is determined by your code and can be tightly determined - usually a constant very short time.

So, I mostly agree with the rest of your post, but this point isn't quite as straightforward as you suggest.

Malloc/new are not deterministic. The cost of an individual allocation is generally much higher than that of a garbage collector, and it is not a fixed cost. But you do get the (to my mind, dubious) benefit that the performance cost is incurred at the call site (whereas garbage collection incurs a performance cost at an indeterminate later date).

If you actually need deterministic allocation cost, then you have to go with other solutions (probably ahead-of-time allocation: pool allocators, SLAB allocators, etc.)


Ahha .. this may be where the confusion lies.

I didn't want to suggest 'using malloc / free at runtime is better than garbage collectors'. Far from it... they both have related downsides.

In c++, if you override new, you don't need to use OS calls for memory management. You can use whatever system you want for grabbing memory from wherever you want, then you have the opportunity to call the constructor yourself with placement new. Posted Image

In addition there is a distinction between one off allocation / allocations at startup, and their corresponding deletion at shutdown, and dynamic use (i.e. the kind of things you might use lots of times in a frame). The second case is what we are interested in here. For actually reserving your memory at startup, you could use whatever you want .. an OS heap, garbage collected system. Ultimately your memory has got to come from somewhere. Posted Image

(There is also the slightly less stringent case of level load / unload, where you *could* if necessary be a bit more lenient / take some shortcuts on some platforms).

What we are after in games, in an ideal world, for dynamic allocation (things that happen a lot rather than just startup and shutdown) is stability (no failed calls) and constant time (and fast) allocation and deallocation. Posted Image

Sorry I should have been more clear on this. I would on the whole use things like fixed size memory allocators (and potentially other constant time allocators) for things that need to be created / destroyed dynamically (see my first post on page 1). You can use this for constant time incredibly fast allocations / deallocations, suitable for things like nodes in algorithms, even particle type systems.

For things that are truly variable size (levels etc) the tradeoff can be to prereserve space at startup for worst case, and work with that. Alright you lose a bit from the theoretical maximum, but you gain in simplicity and stability. On levels with not much geometry, you can e.g. add more sound, or more textures, and vice versa. For your level file you can prepack into the best format possible, with zero fragmentation, and make use of the whole of your budget in megs. If you need to use more than this, then you need to support streaming of level data on the fly (this is a whole other topic with similar concerns, guess what, you can use fixed size bank slots for this too!).

You can do this for GPU resources too .. reserve e.g. 5000 verts for a character and then stick to that budget or lower for your artwork, and you can guarantee they will always fit in that 'slot'.

You can also pre-designate blank 'slots' for various items in the level data RAM allotment to give more flexibility, if it seems a better idea than deciding ahead of time the maximum number of item 'blah'. If you do this you get the benefit of zero fragmentation, and best use of memory for that level.

In short there are lots of handy 'helper' bits of functionality offered to programmers, like 'general purpose' heaps, variable size strings etc. There are whole languages dedicated to making things 'easier' for the programmer where these things are a given (basic, php etc etc). In most situations this is a real benefit because it makes you much more productive as a programmer - less code, simpler code, less potential for bugs, and the 'costs' are not going to appear to the user.

It's just that in some situations, particularly time critical applications, and those on limited memory devices, it can become worth it to not use some of the helper functionality. An extreme example would be missile control software. You might have limited memory. If your program crashes, people die. If your program takes too long to faff around restructuring the heap, people die. It's only if it works predictably and as per spec that the right people die.

Other examples where you have to be a bit more stringent include things like financial software, medical software, some engineering software.

Would you want the nuke heading towards your neighbours house programmed in java with garbage collection, or c++ with no external allocations? I know know which one I'd rather have heading towards my neighbours. Posted Image

(edit) Some good search terms to google in this area are : 'real time programming', and 'mission critical programming'. (/edit)


#4977804 What improves better memory performance?

Posted by lawnjelly on 07 September 2012 - 02:39 PM


Well it would 'be nice' to be lazy and leave everything to a GC,


Sometimes the developer (or team) may not be quite skilled enough to have a choice and I hope to god that if working in a team, I wouldn't have to use some bodgy monstrosity.

In my spare time, I port software to FreeBSD and notice quite a few cases where developers have tried to hand roll their own stuff, nothing flags up bugs in this type of thing better than porting to an entirely new platform (and older version of GCC). So I suggest using a garbage collector unless you really know how to use the language properly.

While I havn't properly touched managed languages for well over 4 years, the Boehm GC works satisfactory on C++.

For software which needs no clever memory management however, the only solution is tr1/shared_ptr!.


Yup, don't get me wrong, in almost the majority of apps I'd be all for using all the tricks in the book to make things simpler. Garbage collection, you name it.

Sorry if I come off as opinionated on the subject, I was a bit unfair on you Karsten .. I've had to deal with the mess caused in the past and it's not been pleasant. It's not very fair when people's jobs are on the line, and their families depending on them etc.

It's just in the specific case of (professional) games, particularly on fixed low memory devices (and some other software on embedded systems), my personal belief is that controlling the memory yourself can be the best option. That doesn't mean it's necessarily the best approach for people learning .. it's more an approach for making a solid professional product.

The two main reasons I would argue for this are:

Stability
Predictable timing

Stability - no worries about failed allocations .. your game will run each time, every time, no matter how many levels you load, what combinations of objects need to be loaded. There's no, ah but if character B walks round the back of building A, carrying object C and opens the door on level BLAH, then it crashes. Sometimes. Which is pretty much what you don't want to hear about when you are trying to ship something. Or what happens if someone is running such and such a program in the background in a multitasking environment.

Of course it's possible you could get round this to some extent with your Garbage Collection system - if it can allow you to pre-reserve your memory, (depending on its implementation regarding fragmentation), and if you keep a tight handle on your numbers of various objects. But once you get to this extent you are almost doing the work of doing it yourself anyway.

The other is that there is no question over the time taken over a deallocation / allocation. It is determined by your code and can be tightly determined - usually a constant very short time. There's no worry about dropping frames etc. Using a third party allocation / deallocation system leaves you at the mercy of their implementation. That's not to say there aren't good implementations, but there are also bad ones, and worst cases. Windows for example is quite happy to grind to a halt and do some disk swapping when it thinks it's necessary during an allocation / deallocation.

I fully understand that it can be a bit of extra effort (sometimes quite a bit) to manage memory yourself, although it's usually mainly a one off cost setting up your project. But development isn't just the time putting the code together, it's also beta testing, trying lots of different scripts, game levels, combinations of factors. In this situation the more potential problems you can remove the better.

If you are working to a time schedule with milestones and a budget and staff costs to pay, the last thing you want is some vague uncertainty over 'yeah it may take 2 years to beta test this thing'. That's one of the (several) reasons why games get canned / companies go under.

But anyway at the end of the day it's up to whoever is technical lead on a project to make these kind of decisions. Right I'm tired that's enough essaying it's bedtime! Posted Image




PARTNERS