Game Engine Creation: What are the biggest challenges faced?

Started by
19 comments, last by SuperG 8 years, 5 months ago
The biggest challenge is writing the tools and level editor to make the engine actually useful.

It's not a challenge, it just takes monster time.

Writing an engine itself isn't actually difficult.

It's false, writing a good engine is not easy at all, you have to deal with threading, motion system, cross platform, easy to use, extensible.....

Its been done so much there are books and tutorials out there that it becomes just a case of following the recipe and adding your own space.

Lot of features, especially in motion system doesn't have any tutorial or books, it's pure research and takes time.

Advertisement

You probably don't have a large enough team to build custom tools for all of that.

I didn't say you have to build custom tools for all that, but you do need to plan the workflows.
If you've got one artist and they want to produce models in Blender, textures in Photoshop and animations in Motion Builder, you need to build your engine around supporting these existing tools. If instead, an artist joins your team and you say "I've got an .obj importer!", they'll immediately turn around and walk back out the door laugh.png

e.g. The artist that I've paired up with for my current game is a die-hard XSI fanboi, so our engine supports using XSI as a modelling tool, material editor, rendering/lighting preview, scene editor and animator smile.png We also have things set up so you can export from XSI into the game while the game is still running for quick iteration.

Most hobby engines don't even have one artist on their team, so they have no idea what the content-creation workflow for their first game is going to be... So they create an engine, start making their first game, try to recruit people... and then... try to solve the content workflow issues that suddenly appear.

As well as answering "How do I write gameplay code?", you need to be able to answer "How do I create UI's?" and "How do I import a skinned mesh?", etc... Unfortunately this is often catch-22, as you want to tailor the answers to those questions to the talents and preferences of your content-creators, who you don't have yet.

So if your first game with the engine is going to be a one-man game, then first you've got to learn how to design levels and create art, and then create the engine around your own workflow preferences for those tasks.

Too bad there are already common formats supported by nearly every single tool tongue.png. Though i don't quite understand the logic of not using more than one format, when said format is usually supported by all systems. Why support Autodesks prioprietary format, when a more common format supported by all 3D tools is Colloda, Obj, and Open Game Exchange? Plus for things like .wav and mp3 files... they are under some hard core licences... so using them in a game engine is no go.

But I am going to add in on my previous post. The hardest part of making a game engine, is not to over-engineer. Which I am now simplifying my code down to leaving actor logic updates as single threaded. And representations as threaded. Maybe co-routines will help out when I need to do something like raycasting. Maybe not.

Too bad there are already common formats supported by nearly every single tool . Though i don't quite understand the logic of not using more than one format, when said format is usually supported by all systems. Why support Autodesks prioprietary format, when a more common format supported by all 3D tools is Colloda, Obj, and Open Game Exchange?

Having existing formats != a workflow.
So you add a collada importer, and tell artists they can put collada files into the engine. Does your importer merge their submeshes, or is that their job? How do you create animation sets from multiple collada files, and do you correct for situations where skeletons between animations and models are slightly different? How do the artists annotate animations with timeline events, such as footfalls? Do you account for the slight differences in how 3DS Max & Blender each choose to write their Collada files (the spec is open to interpretation, extension, and abuse)?
How does the artist select a particular game shader from in their DCC tool? How do they enable shader options, such as parallax mapping? How do you decide which vertex attributes to import from the collada file? If the chosen shader requires a vertex attribute but it's missing from the file (e.g. tangents), do you report an error, or try to create the data yourself? What's the recommended way to import baked normal maps + NBT basis frames without losing precision? How many sets of vertex colours can you support? What naming conventions should artist's use to designate primary and secondary UV sets? Will they create their materials in the DCC tool and export them in the collada file, or will they create the materials later, in your engine? Does the engine automatically create texture atlases when merging sub-meshes of a model, or should the artists manually pack their textures together?
How should assets be created to work with visibility / occlusion / culling systems? Should the artists manually split large models into sub-meshes according to spatial groupings, to allow sub-meshes to be hidden? Do they need to create a separate low-poly occluder mesh? What about a separate collision mesh for the physics system? How are physics materials and audio-materials linked to visual surface materials?
What types of texture layers/channels do your game shaders require - albedo/roughness/metalness/normal, etc? Are some layers optional? Are there naming conventions for these layers? Are the textures DXT compressed by the importer or do the artists need to do it manually? What conventions for specular maps does your game use (specular maps are different for every game as no one seems to be able to agree on conventions for them)? What's the in-engine-preview workflow for texture artists who need to quickly iterate on things like the specular maps? Can they use your game's shaders within their DCC tools for preview purposes, or is there a preview tool, or engine mode designed for texture tweaking/iteration?

Many questions / problems arise from actually trying to produce content for your engine. If there's no solid workflow to create content for the engine, then it's not finished yet.

When you start making a game, even with Unity / Unreal / etc, you need to decide on the workflow that your content creators will follow. Off-the-shelf engines will put constraints on your content-creators -- they'll have their own lists of supported formats, their own naming conventions, material systems, etc, which your content creators will have to adhere to... One major advantage of building your own engine over an off-the-shelf one is that you've got unlimited flexibility in choosing your workflow if you do it up-front. If you don't do it up-front, then you'll be blindly creating the same sorts of constraints completely by accident, which is probably even worse than an off-the-shelf engine. At least off-the-shelf engines have been tested by a large amount of users, so their constraints are probably pretty reasonable.

Too bad there are already common formats supported by nearly every single tool . Though i don't quite understand the logic of not using more than one format, when said format is usually supported by all systems. Why support Autodesks prioprietary format, when a more common format supported by all 3D tools is Colloda, Obj, and Open Game Exchange? Plus for things like .wav and mp3 files... they are under some hard core licences... so using them in a game engine is no go.

Wav is fine, but MP3 decoding/encoding algorithms are covered by patents (having the files exist is fine - the process of decoding them is protected angry.png). However, you use a library like FMOD, they've already paid the licensing fees for that algorithm, so you're free to use MP3 files. That said, if you do use something like FMOD, then you'll likely have a workflow where your source data is in WAV, but then your content build system compiles them into some kind of archive of vorbis files, etc...

Likewise for your collada files -- you would never ship them as part of your game! Your engine's tools would import the necessary data from the collada files and convert them into a much smaller / more efficient format. e.g. A randomly chosen model in my current project is 8MB XSI (source) -> 23MB collada (intermediate) -> 7MB imported (in RAM in the toolchain) -> 4MB engine (compiled to efficient on-disk format) - > 0.5MB engine archive (compressed for streaming). If you make a decision such as "the engine supports collada files", then that's one piece of a workflow -- you also need to make decisions as to what format you'll convert those collada files into, and how/when the conversion takes place (as well as the many, many above questions about how the content of the collada files is treated / constrained).

You have a build system for your code -- what's the build system for content?



Buster2000, on 17 Oct 2015 - 1:31 PM, said:
Writing an engine itself isn't actually difficult.
It's false, writing a good engine is not easy at all, you have to deal with threading, motion system, cross platform, easy to use, extensible.....

None of which are difficult they just take monster time ;)


None of which are difficult they just take monster time ;)

What is difficult in programming then? Unless you are developing a brand new, cutting corners technology, everything in programming can be done given "unlimited" time. Developing a large project for many years in a way that you don't have to ragequit after some time because your codebase became totally unworkable with, can be considered difficult.

So its not just a matter of "I throw code and features at the engine until I have everything I want", but a process that requires planning, knowledge and heavy refactoring unless you have a very good idea of what you are doing. Otherwise you'll just come to a point one day where this feature you want to introduce or this improvement to the workflow will just require you to rewrite pretty much everything, or just start over.

Which could be solved given enough time, ok, but that is pretty much the definition of "difficult" right there.

None of which are difficult they just take monster time ;)

Lot of features are easy to implement only because you have source with the paper or a good paper explaining all, without papers on internet, you would not say that.

None of which are difficult they just take monster time ;)

Lot of features are easy to implement only because you have source with the paper or a good paper explaining all, without papers on internet, you would not say that.

But these papers exist therefore I do say that. The challenge isn't gluing these features together into an engine but writing an interface and tools so that your engine can be reused by other people. Now you may go on about <insert feature that is only available in cutting edge engines and has very little research done on it> but , that isn't what the OP was asking for. He didn't mention physics, he didn't mention cross platform, he didn't mention shaders, he didn't mention motion systems, he didn't even mention what programming language he is using. He just asked for challenges when creating a games engine.

Every single Games Coder I've ever interviewed has had some form of games engine that they have written, but not every coder has written a full game.

Every single Games Coder I've ever interviewed has had some form of games engine that they have written, but not every coder has written a full game.

It's a good point to mention, it's better to use an existing engine to make a game and work on an engine when spare time.

Will echo what many have already said.

The engine runtime is the fun part, and also mostly easy, unless you go for challenging state-of-the-art tech, or try to maximize performance. Creating the scene management, rendering and lighting, physics integration, possible multithreading etc. This can still take a lot of time (easily a man-year) depending on your expertise and how much features you're going to add.

After you've got the runtime done, the rest is to make the system usable for actual game creation. Up to this point you probably haven't needed to make any concrete decisions on how the game projects made with the engine are structured, how assets are imported and cooked into a build, how the game logic or rules are inserted and how they interact with the runtime, how the world data is represented for processes like precalculated lighting or navigation data creation, and how to make all the workflows usable for the creators. Now you're going to have to make a lot of decisions which influence what kind of games you can make with the system, and how usable it turns out in the end.

It helps if you can handle 3D modelling yourself so you can continuously test from a content creator's point. In reality working on the runtime & tools / workflow will very likely intertwine, I just separated them to illustrate the difference.

You can also decide to limit yourself to just creating a coder-oriented runtime library (compare e.g. to Cocos2D or Ogre), rather than a full-blown game engine (like Unity). It will still be a worthwhile learning experience, but probably not something that's directly useful as a game creation tool. Getting to the full-blown stage will certainly take man-years.

One challenge I haven't seen any comments on is - depending on the language - is incorporating type reflection and a Metadata system. The reason I bring this up is that it is not simple nor one sided in a C/C++ code base (which I would recommend for an engine - even more so for learning purposes).

The reason I bring this up is because not many have, it offers usefullness, and when incorporated, makes life a breeze when adding new modules. Many have brought up tool chains and the such but have forgotten that to make something - like a great editor, property reflection is extremely handy. For instance it enables easier data binding to your User Interface as well as writing out assets to a file and loading them in later. Other uses is that incorporating reflection, you may get serialization and a network protocol for free. Finally, adding new modules becomes easier if you design an extensible reflection or Metadata system. For instance, incorporating a scripting language becomes simpler when you can look up function and class definitions automatically in code. As opposed to hard coding them to be bound into the scripting language, on top of having to add new bindings with each new class.

This topic is closed to new replies.

Advertisement