Jump to content

  • Log In with Google      Sign In   
  • Create Account


GameEngine from start to finish(DevLog)


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
15 replies to this topic

#1 ShadowKGames   Members   -  Reputation: 335

Like
3Likes
Like

Posted 16 January 2014 - 10:12 PM

Engine DevLog

 

What I thought might be a cool idea is to create a dev log all about making a game engine from start to finish, I'm not talking just a simple triangle in OpenGL.. I'm talking about everything, from building the API around the renderer to particles / phsyx / post processing and even advanced features, I one day want to implement like a cut down V-ray rendering mechanism.

 

I'll include everything, all the trials and tribulations. The points of utter frustration, the days where I worked so long I can't even remember my name the next day.. I'll approach it from a beginners perspective.

 

Why not just use a tutorial?

 

Well, I'm not going to release all the code because it'll one day be a commercial engine. Even though it's me on my own doing it out of work, also what tutorials generally don't give you is the headache of debugging. They don't include common issues and things you may come across, neither do they go into the actual experience of doing all of this.

 

What's my experience?

 

I've been a hobbyist / professional developer / engineer for 14 years, spanning across multiple different sectors including games / telecommunications / retail etc. 

 

Am I a beginner?

 

Ummm, yes and no.. I've done many projects large and small, I've built engine frameworks (or rip-offs for learning) and worked on games as a developer and artist. But I will be starting from scratch to create a modular engine with as little interclass dependency as possible. So additions and features don't need a re-write, the whole cause of a previous failure. This is a new one for me too, so it'll be very interesting.

 

Can I do this?

 

Yes, if you put the time in.. It's hard work and in many years I have never seen a genius programmer neither have I ever seen someone build an engine with no experience of game creation before. I have seen slackers and hard workers with a thirst for knowledge..

 

The code and target

 

The point in this is to create a modern OpenGL framework, I'll introduce you to the framework / concepts and tools. What I want from my code is:

 

> Little in the way of bugs

> Easy for others to understand

> Maintainable easy framework which can suitably modified

> Doesn't crash

 

I'm not proud and I'm a human, I don't care if someone thinks my code is crap or not.. I do care about my end users.

 

Where to start:

 

Might be a nice introduction, go play a game.. Your favorite game (yes you heard me), instead of focusing on how fun it is.. Start thinking about how it's put together.. I'm talking make notes on what happens when you press a key and you move forward, what happens when a GUI menu item is clicked etc. etc.

 

There's a name for what happens, you move forward (Transformation), you rotate (Quaternion transform), you press a key (Event).. Think about the logic..

 

If I get hit, then I loose life.. Hmm you say an if statement.. 

 

Also a primer on how this all came to be might be a good idea:

 

http://openglbook.com/the-book/preface-what-is-opengl/

 

What language do I choose?

 

Well being honest, the language is kind of irrelevant in the whole scheme of things. But I will be choosing C++, I'll be straight up and say I don't really like C++.. A lot of the quirks infernally annoy me, on basic render tests there has been near 0 impact on performance on a modern PC between C++ and C# with garbage collection.

 

So why?

 

Because most of the source examples I have reverse engineered are in C++, even from the Nvidia developers framework, also you might as well get used to it as GLSL shaders are in C anyway.. it's easy to find a deferred rendering example in C++.. But using an OpenTK wrapper and using C# examples are scarce and it can become painful. Also for better or worse, it's what I'm used to.

 

Will I use libraries?

 

Of course I will, you'd be mad to create an input and sound library.. Even if you're in a team of 20, any shortcut, ANYTHING that can make your life easier use it.. Building an engine is hard enough and takes years, let's not add to the pain. Everything else is all us, the renderer, shader and toolset will be done by us.

 

What do I get out of this?

 

I'm starting from scratch and it's a give and take scenario, some Guru's on here might help me in certain area's.

 

I hope this is a good introduction and I'm looking forward to updating as I go along.


Edited by ShadowKGames, 19 January 2014 - 11:34 AM.


Sponsor:

#2 eduardo_costa   Members   -  Reputation: 327

Like
0Likes
Like

Posted 16 January 2014 - 11:07 PM

As a fellow EngineEngineer (I'm doing one but WebGL in HaXe language) here are some cents.

 

If you're focusing in the end-user you should try C#, as you said the perfomance will not be an issue (look at Unity3d), simple and frequent operations like sorting, delegates, get/set methods and other hi-level stuff done in 1 C# line will help a lot.

 

Just wrap OGL stuff in a static class and you are free to create a higher-level class library that make the calls organized and the GC is a good friend too.

In terms of class modelling I found Unity's way really awesome (Entity-Component based), you have a container (Entity) and Components are attached to it, after that, only destroying it will remove it (it is nice to avoid lost references).

 

I have a 'Object' like class (Resource) which contains useful stuff like unique-id, name string, ... which I register in a global Resource list, useful to simply destroy everything after a scene changes or globally search any stuff based on type or name (Reflection is another awesome feature of C#).

 

I organize stuff in scenes wich is basically a XML [dependecies + hierarchy] that create stuff in RAM/GPU memory and also creates the entities in 3d space with its components + reference to the loaded dependencies. When a new scene is called, destroy everything and restart the process.

 

About OpenGL, I suggest also following Unity's footsteps. Working with the concept of Renderer + Material, I reached a way to sort the rendering stuff based on properties of both elements and then optimizing stuff and fitting RenderTarget operations (E.g. ImageEffects) more easily in the pipeline.

 

Lastly, I will reveal the feature that took most of my time in the 6 months of development. The Collada Loader adapted to correctly load Bones and Animations. It was a real pain. Also the Skinning technique was hard to make it work efficiently in the GPU mode.

 

Nice post! Keep in touch!

 

**Edit**

 

Also create a static class to draw any kind of gizmo on top of everything (lines, wirespheres,wirecube,...) it will help a lot to debug hard stuff (skinning)


Edited by eduardo_costa, 16 January 2014 - 11:12 PM.


#3 L. Spiro   Crossbones+   -  Reputation: 13212

Like
9Likes
Like

Posted 16 January 2014 - 11:19 PM

This was exactly the original purpose of my site.

It was supposed to be a logging of the trials and errors of constructing an engine, not to have a bunch of tutorials.  It eventually shifted focus because there was no reason not to include tutorials and other things, but until now I haven’t really talked a lot about the “trials and tribulations” part.  That is changing now that I am redoing the graphics module.

 

Although my philosophy on using 3rd-party libraries is different, we basically seem to have exactly the same goal with our sites, so readers will be able to see different ways of doing it all, which I would consider a good thing.

 

 

L. Spiro


It is amazing how often people try to be unique, and yet they are always trying to make others be like them. - L. Spiro 2011
I spent most of my life learning the courage it takes to go out and get what I want. Now that I have it, I am not sure exactly what it is that I want. - L. Spiro 2013
I went to my local Subway once to find some guy yelling at the staff. When someone finally came to take my order and asked, “May I help you?”, I replied, “Yeah, I’ll have one asshole to go.”
L. Spiro Engine: http://lspiroengine.com
L. Spiro Engine Forums: http://lspiroengine.com/forums

#4 SeanMiddleditch   Members   -  Reputation: 5078

Like
1Likes
Like

Posted 17 January 2014 - 12:25 AM

If you're focusing in the end-user you should try C#, as you said the perfomance will not be an issue (look at Unity3d), simple and frequent operations like sorting, delegates, get/set methods and other hi-level stuff done in 1 C# line will help a lot.

 

Not arguing with your sentiment, but Unity3D is written in C++; Mono is used for user extensions, not the engine itself.  :)



#5 ShadowKGames   Members   -  Reputation: 335

Like
0Likes
Like

Posted 17 January 2014 - 12:52 AM

This was exactly the original purpose of my site.

It was supposed to be a logging of the trials and errors of constructing an engine, not to have a bunch of tutorials.  It eventually shifted focus because there was no reason not to include tutorials and other things, but until now I haven’t really talked a lot about the “trials and tribulations” part.  That is changing now that I am redoing the graphics module.

 

Although my philosophy on using 3rd-party libraries is different, we basically seem to have exactly the same goal with our sites, so readers will be able to see different ways of doing it all, which I would consider a good thing.

 

 

L. Spiro

 

Sounds great, If you don't mind can I use some of that blog as excerpts? Anyway I'll be going down the OpenGL strictly path.. For a couple of reasons, A) the mantle OpenGL derivative and B) cross compatibility..

 

From a quick scan, you seem to have quite a lot of cool stuff that I'm interested in.. But I'll not be including mobile support, because I don't have a clue smile.png.

 

This is literally a high end renderer and toolset for good looking games. 

 

@ Eduardo

 

Exactly what Sean says, the engine and core concepts will be written in C++.. Then a layer of abstraction for C#. CryEngine is written in C++ but you can use CryMono for example as an end user layer.. So that's the path I'll go down.

 

Also appreciate the input, XML might be a cool way to go for external resources.. 


Edited by ShadowKGames, 17 January 2014 - 01:10 AM.


#6 L. Spiro   Crossbones+   -  Reputation: 13212

Like
2Likes
Like

Posted 17 January 2014 - 01:05 AM

Sounds great, If you don't mind can I use some of that blog as excerpts?

Of course.
 

But I'll not be including mobile support, because I don't have a clue. This is literally a high end renderer and toolset for good looking games.

Don’t underestimate the graphics quality of some mobile games.
I just finished working on this:

 

 

L. Spiro


It is amazing how often people try to be unique, and yet they are always trying to make others be like them. - L. Spiro 2011
I spent most of my life learning the courage it takes to go out and get what I want. Now that I have it, I am not sure exactly what it is that I want. - L. Spiro 2013
I went to my local Subway once to find some guy yelling at the staff. When someone finally came to take my order and asked, “May I help you?”, I replied, “Yeah, I’ll have one asshole to go.”
L. Spiro Engine: http://lspiroengine.com
L. Spiro Engine Forums: http://lspiroengine.com/forums

#7 Solid_Spy   Members   -  Reputation: 401

Like
0Likes
Like

Posted 17 January 2014 - 04:36 AM

 

Also appreciate the input, XML might be a cool way to go for external resources.. 

 

XML is ok, just remember that parsing XML files can be very slow compared to loading binary files, especially in an age where loading times are becoming impacted as bigger resources are demanded. I personally use Assimp in a separate program and generate my own binary files for 3d models and animations from COLLADA, however xml is a good start. Creating your own binary file can be a pain in the ass, but it's kinda cool once you get it done :3



#8 arka80   Members   -  Reputation: 787

Like
0Likes
Like

Posted 17 January 2014 - 06:07 AM

 

 

I hope this is a good introduction and I'm looking forward to updating as I go along.

 

Ok, now give us the link! cool.png



#9 jbadams   Senior Staff   -  Reputation: 17932

Like
2Likes
Like

Posted 17 January 2014 - 06:47 AM

If you're planning to post this here we'd love to read your experiences -- we have a developer journal section of the site that would be perfect for this. :)



#10 eduardo_costa   Members   -  Reputation: 327

Like
0Likes
Like

Posted 17 January 2014 - 09:00 AM

About doing in C++ allowing C# scripting, I found it great!

My experience making these two communicating is really shallow so I could not use this technique!

My point is that making the end-user working in C++ nowadays isn't that productive anymore.

 

Using XML as scene manager was my choice to avoid binary data being loaded with XMLHttpRequest (HTML5+WebGL Engine).

Other data is a mix of XML and Base64 bytes compressed using LZMA.

So a 'mesh' would be:

 

<mesh>

<vertex>base64_float_bytes</vertex>

...

</mesh>



#11 Nathan2222_old   Members   -  Reputation: -400

Like
0Likes
Like

Posted 17 January 2014 - 09:31 AM

Great. Can't wait for the link.
Will you lua or C# for scripting?

UNREAL ENGINE 4:
Total LOC: ~3M Lines
Total Languages: ~32
smile.png
--
GREAT QUOTES:
I can do ALL things through Christ - Jesus Christ
--
Logic will get you from A-Z, imagination gets you everywhere - Albert Einstein
--
The problems of the world cannot be solved by skeptics or cynics whose horizons are limited by the obvious realities. - John F. Kennedy


#12 ShadowKGames   Members   -  Reputation: 335

Like
0Likes
Like

Posted 17 January 2014 - 12:38 PM

@ L. Spiro

 

That looks great, well done:

 

@ Solid Spy

 

Depends on what you're doing with it, you could choose the CryEngine .XML way of listing objects. But I'm inclined to agree, neither do I particularly like that system and I also don't believe it's very user friendly.

 

@  jbadams

 

Yeah I'd love to post it on here, the development blog section seems to be split into several posts. Whereas I'd like to keep updating in sections on a specific post and then getting feedback as I go along. Just like a thread really, is this possible?

 

@ Nathan

 

Yes C#



#13 ShadowKGames   Members   -  Reputation: 335

Like
2Likes
Like

Posted 17 January 2014 - 04:39 PM

Part 1 (Build engines not games)

 

I'll explain the title later smile.png:

 

As said, I'll cover the shear basics and questions asked.. When I originally started the hardest thing was to find what I needed to get going.. So I'll simplify the startup procedure and answer questions I've seen asked many times.

 

What is a game engine?

 

The question is very open to interpretation, diluted further with modern commercial and freeware (Open source) game engines.  Simply put, I see a game engine as a solution to build and develop games quickly. The toolset size is dependant on what a developer is intending to do, you wouldn't focus on orthographic (2D) projection for a 3D game (let's move past the UI for now), but a multi-platform commercial engine like Unity would. There's several features in this tutorial that many engines wouldn't use of need, so they don't have them.. But they are still an engine, then we have open source products like Ogre3D and Horde3D which are rendering engines. Rendering is a term for displaying graphics, whether using Microsoft's implementation direct3D or the multi-platform OpenGL.

 

Why would I build my own engine?

 

A lot of what I say is subjective and it's for you to make your own mind up, but I will say don't attempt to build an engine if the following applies:

 

  • A current engine serves all purposes required
  • You haven't used a game engine before and at the very least, you're unfamiliar with putting a basic game together.
  • You haven't discovered the limitations of a specific engine.

Building an engine can aid in multiple areas:

 

  • Closed source engines are difficult to modify if impossible
  • Deep understanding of not only how games work, but how tools fit together
  • Feature set's can always be improved and added
  • Financial return

Anyone deciding to build an engine has to weigh up is it worth it, this is NOT A COMMITMENT TO MAKE LIGHTLY and in most cases isn't worth the effort.

 

What's the difference between a game and an engine?

 

Not as much as one would expect, a game requires a renderer / physics / input / audio / UI and a way to be executed on a system / load resources / save games.

 

Many a time code is re-used from a game to build an engine around it, if you're going to do it from scratch then you might as well build what is in essence a level editor which allows you to quickly craft scenes and event's.. That's where the engine comes in, because once completed everything is done in the background.

 

What is the toolset we need?

 

Ok a list of tools:

 

SDL 2.0 (http://www.libsdl.org/)

Glew (http://glew.sourceforge.net/)

QT Creator (Because it's cross platform and supports widget's, (VIsual studio can also use QT widgets) (http://qt-project.org/wiki/Category:Tools::QtCreator)

UI (http://cegui.org.uk/) I'm open to other suggestions for vector or widget based solutions

Nvidia Physx 3.1 SDK (https://developer.nvidia.com/physx-sdk-v31)

 

Build engines not games

 

Wrapping this portion up, I saw an article saying build games not engines. Which rings very true, but there is more to it and not everything is black and white. As said, the best path is to find an engine that works for you. Failing that, you will spend a large portion of time building an engine for your game and time will be consumed building toolsets instead of writing an actual game itself. In a nutshell, this word TIME is the greatest challenge here.. A large high fidelity game may take 2 - 3 years to complete, add an additional 2 years for an engine, then ask yourself. Do you have the time?

 

In part 2, I'll be discussing openGL and Math:


Edited by ShadowKGames, 19 January 2014 - 11:40 AM.


#14 jbadams   Senior Staff   -  Reputation: 17932

Like
5Likes
Like

Posted 17 January 2014 - 09:43 PM


Wrapping this portion up, I saw an article saying build games not engines. Which rings very true, but there is more to it and not everything is black and white.

I think it's worth linking to the article at this point for anyone unfamiliar with it: write games, not engines.  A lot of people simply read the title and maybe skim a little of the text and assume the article is telling you not to create an engine at all, but that's not actually the point it makes; rather, it suggests an approach to engine creation where you start by writing one or more games and refactoring out the useful functionality to form the basis of your engine -- and it's a very good approach for ensuring your engine meets the requirements of a real game without wasting your time on features that might not be needed.

 

Obviously not an approach for everyone, but valuable advice to read and consider.  Looking forward to seeing how your efforts progress! smile.png

 

(...and yes, if you really feel like you prefer to keep this as a forum topic rather than journal entries that's fine, although I'll move you from For Beginners to Game Programming. :))



#15 ShadowKGames   Members   -  Reputation: 335

Like
1Likes
Like

Posted 17 January 2014 - 10:15 PM

 


Wrapping this portion up, I saw an article saying build games not engines. Which rings very true, but there is more to it and not everything is black and white.

I think it's worth linking to the article at this point for anyone unfamiliar with it: write games, not engines.  A lot of people simply read the title and maybe skim a little of the text and assume the article is telling you not to create an engine at all, but that's not actually the point it makes; rather, it suggests an approach to engine creation where you start by writing one or more games and refactoring out the useful functionality to form the basis of your engine -- and it's a very good approach for ensuring your engine meets the requirements of a real game without wasting your time on features that might not be needed.

 

Obviously not an approach for everyone, but valuable advice to read and consider.  Looking forward to seeing how your efforts progress! smile.png

 

(...and yes, if you really feel like you prefer to keep this as a forum topic rather than journal entries that's fine, although I'll move you from For Beginners to Game Programming. smile.png)

 

 

Sounds great, thanks for the citation. That actually wasn't the one I originally read, the article linked shows you should make a game and have an engine built around it which is right on the money and my intentions..

 

Also thanks for the move and appreciate the input, from everyone.. I really do, I don't pretend to know everything and I love it when a community comes together to get a solution put together.

 

I'll just add, any information / point of view or opinion is not a reflection of the company I work at. This is my own personal take on things..



#16 ShadowKGames   Members   -  Reputation: 335

Like
0Likes
Like

Posted 19 January 2014 - 02:46 PM

Part 2: OpenGL and Math
 
I'll start off by covering some basic questions:
 
Do I need to be a Math Genius?
 
No! You have a massive calculator in front of you called a PC, if the prerequisite was a PHD in maths before you could make a game then many people wouldn't get very far. But you do need to have a basic understanding of how the math is put together, the reasons why we do it and how it relates to rendering.
Being straight up honest, the formulas made by math Guru's is freely available to copy and paste on the web. But we don't want to do that do we? No! smile.png Except perhaps the strange formula that is used for quaternion rotation.
 
End users and abstraction
 
One thing you need to decide is who are you doing this for? A commercial released engine for finance, or do you want to use it for a team / personal use? Why I mention this  here is you will have to automate a lot of the framework for end users who do not care or desire to know what you do about how an engine works. Abstraction can be the a world of pain unto itself and cause more headaches than building the engine itself. All the math will be publicly available, the renderer will be manipulated with a couple of clicks.
 
Sure automating tools to become very simple to use can also benefit yourself, but you may find the time to create is not worth the effort. But this tutorial / set of whining, will be based around a commercial engine. Use the concepts and choose your positions wisely..
 
OK!!
 
So I've covered the Q&A, now I think it's time to start learning about how the engine is built!
 
Coordinates
 
The shear basics are, we have a screen filled with pixels that are RGB (Red, Green and Blue) or (Cyan, Magenta, Yellow and black). The higher the resolution, the more pixels are displayed on the screen. We have CRT's which use a timing system, but as technology has moved on I'll not mention anything about them.
LCD screens represent pixels as dots / squares, usually manufactured on a 2D grid.
 
Just as pixels are displayed in a grid, we also are going to compute based on an invisible grid with locations called Vectors. We can have 2 vectors (X,Y) one to represent horizontal axis and another to represent the vertical axis. Whilst we do use 2D Vectors in a 3D game, we will mainly be focusing on a 3 dimensional vectors represented as (X,Y,Z) and a 4D representation (X,Y,Z,W) for rotations. What? 4D? It's a simple way to think about it and as it's not necessary to explain every detail so I'll move on from this one and return later :).
 
The claw GRRR!
 
Easiest way to think about coordinates in 3D space is to stick your hand up in the air and do a three finger claw, your middle finger pointing straight out, your index finger pointing up and finally your thumb pointing the side. Or make an L shape with your Index / thumb and have your middle finger pointing directly away from you.
We always start from (0,0,0), theres nothing stopping you from starting somewhere else.. But there's an issue I'll discuss later in the tutorials with how floating point precision far away from the default location of (0,0,0) can cause issues. By the way (0,0,0) = (0(X), (0(Y), (0(Z)) so in other words X,Y and Z are all set to zero in parenthesis.
So what's the point of all this?
 
It's how we do anything in games, drawing lines, spheres and even calculated positions of complex meshes. Then we transform them from one location to another based on coordinates. So now we have a basic understanding of vectors, let's move on to our next lovely set of maths.
 
Vertices and indices
 
In essence we play a massive game of dot to dot and GL will join the lines up and every line is classed as a vertex, I don't want to go into optimisation to early. But indices brings up a valid point, what if we have two conjoining triangles with overlapping lines? What will OpenGL do if we only displayed everything as Vertices? Well it would draw every Vertex, so we have two triangles and six lines (Vertices). This isn't ideal as I will show you later, drawing is one of the most resource hungry portions in regards to rendering. Welcome to indices, in which simply we just don't duplicate vertices and append as an indices. So we don't draw line, 1,2,3,4,5 and 6 and duplicate the intersection of line 3 (to the adjoining triangle) we just draw, 1,2,3 and 2,3,4.
 
Enter the Matrix
 
What we could do is, set all these positions manually. We could transform from one location to another in 3D space by setting every position in code line by line, but that would be silly and utter waste of time.
 
So what we use is a Matrix / matrices and take advantage of our giant calculator to perform transforms for us.. Cool huh?
The biggest question is what are we trying to achieve with these matrices?
 
Well basically we need to define where we are in relation to anything else, I'm sat in the lower left corner of my room. On top of a chair, in a certain part of the world.
So we have three important matrices to consider:
 
World, view and projection
 
World 
 
Where is the object / model / mesh located in the world? That's about it for world view:
 
View
 
Where exactly is the model so the camera can see it? 
 
Projection
 
This is what type of camera we are going to use, for our project there will be two. One as a main 3D projection and a secondary as a 2D orthographic projection for the user interface.
 
Finishing off
 
I hope you enjoyed this, next time we are going to look at how we apply these features and delve a little deeper into applying the math and how OpenGL responds.
 
I'm going to reveal what sort of game we're going to make as well :)!
 
An RPG WOOO! Wait Wait.. What about my FPS? What about my X game!??
 
It doesn't matter, once you have learn't all of this you will realise very quickly.. There's honestly not that much difference in it!! You might change where the camera is, or modify the physics slightly for your grenade launcher.
 
Why an RPG then?
 
Because, it is the biggest pain in the ass I could find and the toolset will be extensive and transformable to most other games!. Hence when someone pops onto a forum with no experience and says HEY! I want to make an MMORPG and we just have a giggle about it. If they knew what they were getting into, they'd run to the hills!

Edited by ShadowKGames, 19 January 2014 - 03:05 PM.





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS