Firstly there is nothing wrong with someone wanting to write a game engine as an educational tool and the topic-poster had a very specific question about game-engine development. “Write games, not engines” is not the universal solution. If it was then guess what: Games would never be made since no one spent the time to write the engines behind them. Not only that but I would be out of both a job and a hobby. Some people, myself included, appreciate the lower-level technology behind game development and enjoy learning about it more than we enjoy making the actual games. It is also true that some people dive too deep before they are ready, but this is not the way to inform them of that fact.

There is indeed nothing wrong with people wanting to throw code together under the gise of some sort of 'learning' experiance BUT I'm willing to bet that ANYONE doing that is going to end up with a mess which isn't fit for purpose WITHOUT a project driving it.

Of course games would be made without an 'engine' behind it - If you start from scratch you don't spent months putting together some grand unified design before working on the game, you attack the game, figure out what the game needs and then write the code for that bit refactoring and rearranging as scope changes. At the end you have a nice chunk of code, a shipped product AND code you can reuse as you know it is fit for purpose.

Yes, there are many people who like hacking on low level stuff, hell I'm one of them, but thinking you can sit in isolation throwing out code which is going to be useful is foolish, more so if you are a beginner/someone new to the process as you have NO IDEA what you want and need to do. To use the overused car example; it would be like someone sitting in a shed who has never designed a car trying to design and build a powerful engine block - sure they might be able to do it but when they come to try and fit it into a car they might be in for a shock...

If you take a step back you'll very quickly see that EVERY useful game engine which has appeared in the last 15 years has, at some point, started off as a game or been driven very firmly by game requirements; the Unreal engine was not developed in isolation and thats part of what makes it good.

And I don't throw this out there as a 'oh this could happen...' I've SEEN it happen in a company where a bunch of very clever people sat down to write an engine in isolation without a game to drive it (only a small test scene) and it didn't survive first contact with a game; large parts had to be rewritten (not refactored, chucked and redone), more had to be refactored, massive holes in the design came to light once someone tried to use it. Even things as fundimental as 'spawn an object in the world at runtime' was missing because it wasn't needed for the tiny world based test case. This wasn't some bunch of newbies, this was people who'd been in the industry sometime, had shipped games and still managed to screw up to the point where only one game will be shipped on this engine before it is going to be (effectively) thrown away and rebuild/salvaged using input from the games which will need to use it in the future.
(It also being done slower this time; before the 'renderer' interface was 'designed' in 2 days... this time out we are about a month in and still not settled on a design, although we are converging on something which everyone seems happy with.)

So if professionals can screw up then I give someone who is new at it no chance of producting something workable without a game as their primary focus.

Sure, if they want to spend months/years on something which they have to massively refactor at the end of the process when they realise 'shit, this doesn't work...' then fine, I don't really care, although I reserve the right to point and laugh at them when they realise this.

Thus, if you want to write something which can be used to power future games then write a game - you'll learn just as much if not more while producing WORKING and USEFUL code at the end of it. Even if you don't 'ship' the game at the end, or get 6 months in and decide to switch game project the point remains; with something guiding you'll still have usable code to run with.

Of course, if you don't want to power a game with your code then do whatever; it doesn't really matter, if you want to just get pretty images on the screen then you can just throw things togeter adhoc... *shrugs*

Whether or not we get it right on the first try is unrelated to our objective. If our plan is to make a game, we will still fail to make the best animation format on our first try.

OK, so you invent an animation format AWESOMEANIM, you get it hooked up and working, you can see a character being animated in a window... how do you know if this is any good for a game to use? What constraints did you place on it? Did they make sense outside of a test renderer? How optimal is it? Is it going to slow down in a 'real' situation rather than your whitebox test space?

So yes, you might well fail to make 'the best' animation format but you won't know this until you (or someone) tries to use it at which point, depending on how badly you've screwed up, you might well end up throwing away 90% of what you've done in order to support what needs to be done. At which point you might argue 'ah! but we get to learn more!' which I can't argue with however I doubt you'll learn that much more as chance are the core principles are going to be the same, you'll just "learn" how to do them differently with a little bit of extra knowledge on top.

The same can be applied to sound pipelines, rendering pipelines (scene selection, sorting, rendering queues etc; all might look 'great' in your whitebox and fail in real life), UI system, message system.. the list goes on.

Without something to aim at you WILL screw up and you WILL produce code which is worthless and you WONT really learn anything more doing it again... well, aside from 'real requirements are important!' which is something so so so many programmers miss because instead of thinking "lets ask who needs to use it.." they start off with "I'm the best one here, so I'll just run with my ideas and not bother asking those who need to use it..." and then act surprised/annoyed when their "solution" turned out to be bullshit.

While I agree with most of your post L. Spiro, I also advocate "practice practice practice".

Make games and then look for reusable code? Why would anyone do that when we know from the start that we need a 3D file format, animation format, camera-animation format, sound format, image format, etc. We know from the start that we need to support basic features such as sound panning, sound volume, camera panning, camera animations, 3D animations, etc.[/quote]
I need to comment on this though. I feel like you've used the word "we" as if everyone is as experienced as you in writing software. Not everyone understands the concepts of these core engine features - nor does every game even require some of the features you've listed above.

How does one even implement 'camera panning' without some concrete requirement of how it's going to be used? I'll usually fall into the "what if the end-user wants to do this or that" trap... and will ultimately end up writing a bunch of code that never gets used. Personally, I'd rather write a feature for a requirement of the game, instead of some generic feature. And then, if I notice this may be a nice chunk of code to use again, factor it out.

Writing software without concrete requirements can be very difficult, which is why I believe writing engines can be so difficult. Writing a game/demo/unit test is so much easier because those applications inherently come with a set of requirements which have purpose, and can help keep a programmer focused.

I'd much rather think of the "showcase game" of my technology as a way to shape the engine I've written for it. I would rather start with an engine that's broad in scope and use my first game with it as a way to iron out bugs in the engine, discover weak points in my engine code, and streamline the parts of the engine code that need it most so that when I make a game in the future, I have a solid tech to start with that I know will work and I don't have to remove a bunch of extraneous hacks that I had to use to force my previous game to even function.[/quote]
I can respect that darkhaven3. +1.

I need to comment on this though. I feel like you've used the word "we" as if everyone is as experienced as you in writing software. Not everyone understands the concepts of these core engine features - nor does every game even require some of the features you've listed above.

Then maybe that individual shouldn't be writing their own game engine just yet. It is common knowledge that a good game in this modern era generally requires that there be a standard format or formats for images in place that you will read from the disk, audio of some form, and some kind of user interface. I believe all of these can be cleanly separated into core components of a game engine, and you don't have to be the most intelligent person in the world or even a programmer at all to understand these requirements.

How does one even implement 'camera panning' without some concrete requirement of how it's going to be used? I'll usually fall into the "what if the end-user wants to do this or that" trap... and will ultimately end up writing a bunch of code that never gets used. Personally, I'd rather write a feature for a requirement of the game, instead of some generic feature. And then, if I notice this may be a nice chunk of code to use again, factor it out.[/quote]

This implies that the programmer has no idea what kind of game he will be programming, or will be somehow working on-the-fly without some kind of design document about how the game should be implemented written down first. If you are writing a text adventure, you know you will not need camera control of any kind. If you are writing some kind of 2D or 3D game, you will certainly need some kind of "camera" in as abstract of a sense as I can put it. Don't need a particular camera feature for your game? #ifdef it out in your game header files or something.

Writing software without concrete requirements can be very difficult, which is why I believe writing engines can be so difficult. Writing a game/demo/unit test is so much easier because those applications inherently come with a set of requirements which have purpose, and can help keep a programmer focused.[/quote]

If you don't have a game design document that clearly outlines the scope and format of the game before you start writing code, I believe you have more serious issues to attend to than deciding how to write your engine. These are all issues that seem easily resolved at the planning stage; if I'm writing a game for Gameboy Advance, I can have it on good faith I'm not going to be needing things like vertex coloring and DX11 tesselation. Likewise, if I'm writing a game on PC and I want it to look fancy, I can implement these features and I will know that I will need to have this information stored in some kind of consistent format.

In short, while I respect that you might find it difficult to write a game engine, it feels like you are assuming that everyone just writes game engines without some kind of idea as to what kind of game the engine will be used for.

In short, while I respect that you might find it difficult to write a game engine, it feels like you are assuming that everyone just writes game engines without some kind of idea as to what kind of game the engine will be used for.[/quote]
Certainly not everyone. I did not mean to make that assumption. My initial comment meant to state that if one is having trouble writing an engine, then don't be afraid to continue writing games/apps and then refactor. The notion of "I must write an engine before I write a game" can be a trap for beginners with less experience.

If one is experienced and disciplined enough to write small unit tests for each feature and prove that their implementations work and their API's are intuitive throughout development, then I agree with your methods completely.

The bottom line is, just write code, fail early and often, write more code, and continue learning. This experience will only make you a better programmer overall.

I think that writing engines isn't a foolish act, given the fact that there are a bunch of engines out there that everyone could afford even for a commercial application. If someone feels good writing their own stuff and feels strong enough to comprehend the concepts behind that or the other technical feature - then let them do it.
I think that if someone is capable of writing a proper engine, then they are capable of writing a proper game too. Writing the game logic and stuff sometimes isn't much easier than writing a renderer for example. In order to make a great use of any engine, a person should know a great deal of 3D math, theory and concepts.
The experience gained while diving into the code is priceless in the long term. You depend only on yourself, your game has no "back box" dll's, you understand how it works , you can put it in your portfolio, you can show your employer that you can write an engine + editor that the previous candidate could only use.
Does anyone have a statistics of how many successful indie games are made using third party engines ?

There are some amazingly helpful people on here who have some really quite brilliant advice but I think sometimes it becomes a bit repetitive where something subjective is in discussion and I can't think of anything more subjective than "games or engines?".

If someone wants to write engines, they will. If they've got to the point where they have an interest in engines, they've probably already made their mind up and nobody on here is going to make them think "wow, writing games not engines, now why didn't I think of that". If they want to write games on the other hand, the same thing applies, that's what they'll do.

I find that 9 times out of 10, the people who come on here wanting to write the next angry birds usually want to write a game, not an engine. That's fine, no need to tell them to write games not engines.

I am writing an engine and I get massive amounts of satisfaction from it, it's the most therapeutic thing I can think of doing. I thrive on finding out how things work and trying to do them myself. Will I end up with a game? Probably not but anyone who wants to write an engine, generally has an idea of the type of game it might be suitable for, 2d, 3d, etc. you don't need a full game spec to write an engine, it's really fun trying to think of how an animation system might be used, or how particle effects might be used. It's also great for coding practice from an architecture perspective - who cares if it makes a game or not. Same goes for creating games, who cares if it makes a great engine or not, as long as you're enjoying yourself - at the end of the day, isn't that what we're here for?

I say let's just agree that if someone wants to do something, they've already made up their mind and let's help them with what they're trying to do. If someone has no clue about games or engines and is asking advice on which is best to start on, then we should be asking questions like "what are you interested in? What are your goals?" and so on.

Debates like these are always healthy but surely this particular subject has been discussed too much....?

The OP asked about shader permutations and it sounds like he wants to write an engine and isn't just messing about - shall we concentrate on that?

I just wanted to very quickly add, I don't disagree with any responses on here, just that a useful-sounding first post has turned, once again, into "write x not y"

If someone wants to write engines, they will. If they've got to the point where they have an interest in engines, they've probably already made their mind up and nobody on here is going to make them think "wow, writing games not engines, now why didn't I think of that". If they want to write games on the other hand, the same thing applies, that's what they'll do.

In my experience most people who 'want to write an engine' do so because they believe that writing an engine is some kind of requirement before writing a game. Most of these people want to write games really but due to hearing the word 'engine' a lot think the wrong thing and go off in the wrong direction generally because they think they must have an engine and don't want to 'do it wrong'.

The OP asked about shader permutations and it sounds like he wants to write an engine and isn't just messing about - shall we concentrate on that?
[/quote]

The OP didn't ask about permutations at all.. he never mentioned them... he asked 'How should shader support be designed in the engine so it is actually easy to use while programming the game?' which says nothing about permutations at all.

Maybe shader permutations factors into this.. maybe it doesn't... of course without knowing what he wants to DO (beyond a nebulous 'make an engine' for some mythical 'game') and how things interact this is close to impossible to really advise on...