• # Neural Networks Neural Networks 101

Artificial Intelligence

Intention

How Neural Networks Work

Neural networks work by using a system of receiving inputs, sending outputs, and performing self-corrections based on the difference between the output and expected output, also known as the cost.

Neural networks are composed of neurons, which in turn compose layers, or collections of neurons. For example, there is an input layer and an output layer. In between the these two layers, there are layers known as hidden layers. These layers allow for more complex and nuanced behavior by the neural network. A neural network can be thought of as a multi-tier cake: the first tier of the cake represents the input, the tiers in between, or lack thereof, represent the hidden layers, and the last tier represents the output.

The two mechanisms of learning are Forward Propagation and Backward Propagation. Forward Propagation uses linear algebra for calculating what the activation of each neuron of the next layer should be, and then pushing, or propagating, those values forward. Backward Propagation uses calculus to determine what values in the network need to be changed in order to bring the output closer to the expected output.

Forward Propagation

As can be seen from the gif above, each layer is composed of multiple neurons, and each neuron is connected to every other neuron of the following and previous layer, save for the input and output layers since they are not surrounding by layers from both sides.

To put it simply, a neural network represents a collection of activations, weights, and biases. They can be defined as:

• Activation: A value representing how strongly a neuron is firing.
• Weight: How strong the connection is between two neurons. Affects how much of the activation is propagated onto the next layer.
• Bias: A minimum threshold for whether or not the current neuron's activation and weight should affect the next neuron's activation.

Each neuron has an activation and a bias. Every connection to every neuron is represented as a weight. The activations, weights, biases, and connections can be represented using matrices. Activations are calculated using this formula:

After the inner portion of the function has been computed, the resulting matrix gets pumped into a special function known as the Sigmoid Function. The sigmoid is defined as:

The sigmoid function is handy since its output is locked between a range of zero and one. This process is repeated until the activations of the output neurons have been calculated.

Backward Propagation

The process of a neural network performing self-correction is referred to as Backward Propagation or backprop. This article will not go into detail about backprop since it can be a confusing topic. To summarize, the algorithm uses a technique in calculus known as Gradient Descent. Given a plane in an infinite number of dimensions, the direction of change that minimizes the error must be found. The goal of using gradient descent is to modify the weights and biases such that the error in the network approaches zero.

Furthermore, you can find the cost, or error, of a network using this formula:

Unlike forward propagation, which is done from input to output, backward propagation goes from output to input. For every activation, find the error in that neuron, how much of a role it played in the error of the output, and adjust accordingly. This technique uses concepts such as the chain rule, partial derivatives, and multi-variate calculus; therefore, it's a good idea to brush up on one's calculus skills.

High Level Algorithm

1. Initialize matrices for weights and biases for all layers to a random decimal number between -1 and 1.
2. Propagate input through the network.
3. Compare output with the expected output.
4. Backwards propagate the correction back into the network.
5. Repeat this for N number of training samples.

Source Code

If you're interested in looking into the guts of a neural network, check out AI Chan! It's a simple to integrate library for machine learning I wrote in C++. Feel free to learn from it and use it in your own projects.

Resources

Report Article

## User Feedback

There are no comments to display.

## Create an account

Register a new account

• ### What is your GameDev Story?

In 2019 we are celebrating 20 years of GameDev.net! Share your GameDev Story with us.

• 0
• 4
• 0
• 0
• 0

• 15
• 11
• 9
• 9
• 45
• ### Similar Content

• By MATov
It's a story on how to write a plugin for Unity Asset Store, take a crack at solving the well-known isometric problems in games, and make a little coffee money from that, and also to understand how expandable Unity editor is. Pictures, code, graphs and thoughts inside.
Prologue
So, it was one night when I found out I had pretty much nothing to do. The coming year wasn't really promising in my professional life (unlike personal one, though, but that's a whole nother story). Anyway, I got this idea to write something fun for old times sake, that would be quite personal, something on my own, but still having a little commercial advantage (I just like that warm feeling when your project is interesting for somebody else, except for your employer). And all this went hand in hand with the fact that I have long awaited to check out the possibilities of Unity editor extension and to see if there's any good in its platform for selling the engine's own extensions.
I devoted one day to studying the Asset Store: models, scripts, integrations with various services. And first, it seemed like everything has already been written and integrated, having even a number of options of different quality and detail levels, just as much as prices and support. So right away I've narrowed it down to:
code only (after all, I'm a programmer) 2D only (since I just love 2D and they've just made a decent out-of-the-box support for that in Unity) And then I remembered just how many cactuses I've ate and how many mice've died when we were making an isometric game before. You won't believe how much time we've killed on searching viable solutions and how many copies we've broken in attempts to sort out this isometry and draw it. So, struggling to keep my hands still, I searched by different key and not-so-much-key words and couldn't find anything except a huge pile of isometric art, until I finally decided to make an isometric plugin from scratch.
Setting the goals
The first I need was to describe in short what problems this plugin was supposed to solve and what use the isometric games developer would make of it. So, the isometry problems are as follows:
sorting objects by remoteness in order to draw them properly extension for creation, positioning and displacement of isometric objects in the editor Thus, with the main objectives for the first version formulated, I set myself  2-3 days deadline for the first draft version. Thus couldn't being deferred, you see, since enthusiasm is a fragile thing and if you don't have something ready in the first days, there's a great chance you ruin it. And New Year holidays are not so long as the might seem, even in Russia, and I wanted to release the first version within, like, ten days.
Sorting
To put it short, isometry is an attempt made by 2D sprites to look like 3D models. That, of course, results in dozens of problems. The main one is that the sprites have to be sorted in the order in which they were to be drawn to avoid troubles with mutual overlapping.

On the screenshot you can see how it's the green sprite that is drawn first (2,1), and then the blue one goes (1,1)

The screenshot shows the incorrect sorting when the blue sprite's drawn first
In this simple case sorting won't be such a problem, and there are going to be  options, for example:
- sorting by position of Y on the screen, which is (isoX + isoY) * 0.5 + isoZ
- drawing from the remotest isometric grid cell from left to right, from top to down [(3,3),(2,3),(3,2),(1,3),(2,2),(3,1),...]
- and a whole bunch of other interesting and not really interesting ways
They all are pretty good, fast and working, but only in case of such single-celled objects or columns extended in isoZ direction After all, I was interested in more common solution that would work for the objects extended in one coordinate's direction, or even the "fences" which have absolutely no width, but are extended in the same direction as the necessary height.

The screenshot shows the right way of sorting extended objects 3x1 and 1x3 with "fences" measuring 3x0 and 0x3
And that's where our troubles begin and put us in place where we have to decide on the way forward:
split "multi-celled" objects into "single-celled" ones, i.e. to cut it vertically and then sort the stripes emerged think about the new sorting method, more complicated and interesting I chose the second option, having no particular desire to get into tricky processing of every object, into cutting (even automatic), and special approach to logic. For the record, they used the first way in few famous games like Fallout 1 and Fallout 2. You can actually see those strips if you get into the games' data.
So, the second option doesn't imply any sorting criteria. It means that there is no pre-calculated value by which you could sort objects. If you don't believe me (and I guess many people who never worked with isometry don't), take a piece of paper and draw small objects measuring like 2x8 and, for example, 2x2. If you somehow manage to figure out a value for calculation its depth and sorting - just add a 8x2 object and try to sort them in different positions relative to one another.
So, there's no such value, but we still can use dependencies between them (roughly speaking, which one's overlapping which) for topological sorting. We can calculate the objects' dependencies by using projections of isometric coordinates on isometric axis.

Screenshot shows the blue cube having dependency on the red one

Screenshot shows the green cube having dependency on the blue one
A pseudocode for dependency determination for two axis (same works with Z-axis):
bool IsIsoObjectsDepends(IsoObject obj_a, IsoObject obj_b) { var obj_a_max_size = obj_a.position + obj_a.size; return obj_b.position.x < obj_a_max_size.x && obj_b.position.y < obj_a_max_size.y; } With such an approach we build dependencies between all the objects, passing among them recursively and marking the display Z coordinate. The method is quite universal, and, most importantly, it works. You can read detailed description of this algorithm, for example, here or here. Also they use this kind of approach in popular flash isometric library (as3isolib).
And everything was just great except that time complexity of this approach is O(N^2) since we've got to compare every object to every other one in order to create the dependencies. I've left optimization for later versions, having added only lazy re-sorting so that nothing would be sorted until something moves. So we're going to talk about optimization little bit later.
Editor extension
From now on, I had the following goals:
sorting of objects had to work in the editor (not only in a game) there had to be another kind of Gizmos-Arrow (arrows for moving objects) optionally, there would be an alignment with tiles when object's moved sizes of tiles would be applied and set in the isometric world inspector automatically AABB objects are drawn according to their isometric sizes output of isometric coordinates in the object inspector, by changing which we would change the object's position in the game world And all of these goals have been achieved. Unity really does allow to expand its editor considerably. You can add new tabs, windows, buttons, new fields in object inspector. If you want, you can even create a customized inspector for a component of the exact type you need.  You can also output additional information in the editor's window (in my case, on AABB objects), and replace standard move gizmos of objects, too. The problem of sorting inside the editor was solved via this magic ExecuteInEditMode tag, which allows to run components of the object in editor mode, that is to do it the same way as in a game.
All of these were done, of course, not without difficulties and tricks of all kinds, but there was no single problem that I'd spent more than a couple of hours on (Google, forums and communities sure helped me to resolve all the issues arisen which were not mentioned in documentation).

Screenshot shows my gizmos for movement objects within isometric world
Release
So, I got the first version ready, took the screenshot. I even drew an icon and wrote a description. It's time. So, I set a nominal price of \$5, upload the plugin in the store and wait for it to be approved by Unity. I didn't think over the price much, since I didn't really want to earn big money yet. My purpose was to find out if there is a general demand and if it was, I would like to estimate it. Also I wanted to help developers of isometric games who somehow ended up absolutely deprived of opportunities and additions.
In 5 rather painful days (I spent about the same time writing the first version, but I knew what I was doing, without further wondering and overthinking, that gave me the higher speed in comparison with people who'd just started working with isometry) I got a response from Unity saying that the plugin was approved and I could already see it in the store, just as well as its zero (so far) sales. It checked in on the local forum, built Google Analytics into the plugin's page in the store and prepared myself to wait the grass to grow.
It didn't take very long before first sales, just as feedbacks on the forum and the store came up. For the remaining days of January 12 copies of my plugin have been sold, which I considered as a sign of the public's interest and decided to continue.
Optimization
So, I was unhappy with two things:
Time complexity of sorting - O(N^2) Troubles with garbage collection and general performance Algorithm
Having 100 objects and O(N^2) I had 10,000 iterations to make just to find dependencies, and also I'd have to pass all of them and mark the display Z for sorting. There should've been some solution for that. So, I tried a huge number of options, could not sleep thinking about this problem. Anyway, I'm not going to tell you about all the methods I've tried, but I'll describe the one that I've found the best so far.
First thing first, of course, we sort only visible objects. What it means is that we constantly need to be know what's in our shot. If there is any new object, we got to add it in the sorting process, and if one of the old one's gone - ignore it. Now, Unity doesn't allow to determine the object's Bounding Box together with its  children in the scene tree. Pass over the children (every time, by the way, since they can be added and removed) wouldn't work - too slow. We also can't use OnBecameVisible and other events because these work only for parent objects. But we can get all Renderer components from the necessary object and its children. Of course, it doesn't sound like our best option, but I couldn't find another way, same universal and acceptable by performance.
List<Renderer> _tmpRenderers = new List<Renderer>(); bool IsIsoObjectVisible(IsoObject iso_object) { iso_object.GetComponentsInChildren<Renderer>(_tmpRenderers); for ( var i = 0; i < _tmpRenderers.Count; ++i ) { if ( _tmpRenderers[i].isVisible ) { return true; } } return false; } There is a little trick of using GetComponentsInChildren function that allows to get components without allocations in the necessary buffer, unlike another one that returns new array of components
Secondly, I still had to do something about O(N^2). I've tried a number of space splitting techniques before I stopped at a simple two-dimensional grid in the display space where I project my isometric objects. Every such sector contains a list of isometric objects that are crossing it. So, the idea is simple: if projections of the objects are not crossed, then there's no point in building dependencies between the objects at all. Then we pass over all visible objects and build dependencies only in the sectors where it's necessary, thereby lowering time complexity of the algorithm and increasing performance. We calculate the size of each sector as an average between the sizes of all objects. I found the result more than satisfying.
General performance
Of course, I could write a separate article on this... Okay, let's try to make this short. First, we're cashing the components (we use GetComponent to find them, which is not fast). I recommend everyone to be watch yourselves when working with anything that has to do with Update. You always have to bear in mind that it happens for every frame, so you've got to be really careful Also, remember about all interesting features like custom == operator. There are a lot to things to keep in mind, but in the end you get to know about every one of them in the built-in profiler. It makes it much easier to memorize and use them
Also you get to really understand the pain of garbage collector. Need higher performance? Then forget about anything that can allocate memory, which in C# (especially in old Mono compiler) can be done by anything, ranging from foreach(!) to emerging lambdas, let alone LINQ which is now prohibited for you even in the simplest cases. In the end instead of C# with its syntactic sugar you get a semblance of C with ridiculous capacities.
Here I'm gonna give some links on the topic you might find helpful: Part1, Part2, Part3.
Results
I've never known anybody using this optimization technique before, so I was particularly glad to see the results. And if in the first versions it took literally 50 moving objects for the game to turn it into a slideshow, now it works pretty well even when there're 800 objects in a frame: everything's spinning at top speed and re-sorting for just for 3-6 ms which is very good for this number of objects in isometry. Moreover, after initialization it almost haven't allocate memory for a frame
Further opportunities
After I read feedbacks and suggestions, there were a few features which I added in the past versions.
2D/3D Mixture
Mixing 2D and 3D in isometric games is an interesting opportunity allowing to minimize drawing of different movement and rotations options (for instance, 3D models of animated characters). It's not really hard thing to do, but requires integration within the sorting system. All you need is to get a Bounding Box of the model with all its children, and then to move the model along the display Z by the box's width.
Bounds IsoObject3DBounds(IsoObject iso_object) { var bounds = new Bounds(); iso_object.GetComponentsInChildren<Renderer>(_tmpRenderers); if ( _tmpRenderers.Count > 0 ) { bounds = _tmpRenderers[0].bounds; for ( var i = 1; i < _tmpRenderers.Count; ++i ) { bounds.Encapsulate(_tmpRenderers[i].bounds); } } return bounds; } that's an example of how you can get **Bounding Box** of the model with all its children

and that's what it looks like when it's done
Custom isometric settings
That is relatively simple. I was asked to make it possible to set the isometric angle, aspect ratio, tile height. After suffering some pain involved in maths, you get something like this:

Physics
And here it gets more interesting. Since isometry simulates 3D world, physics is supposed to be three-dimensional, too, with height and everything. I came up with this fascinating trick. I replicate all the components of physics, such as Rigidbody, Collider and so on, for isometric world. According to these descriptions and setups I make the copy of invisible physical three-dimensional world using the engine itself and built-in PhysX. After that I take the simulation data calculated and get those bacl in duplicating components for isometric world. Then I do the same to simulate bumping and trigger events.

The toolset physical demo GIF
Epilogue and conclusions
After I implemented all the suggestions from the forum, I decided to raise the price up to 40 dollars, so it wouldn't look like just another cheap plugin with five lines of code I will be very much delighted to answer questions and listen to your advices. I welcome all kinds of criticism, thank you!
Unity Asset Store page link: Isometric 2.5D Toolset
• By tromtrom
Hi there,
I've been absolutely fascinated by Noita trailers, and gameplay videos.
https://noitagame.com/
Their pitch is that every single pixel is simulated. How does it work exactly?
I'm trying to take another game as example, let's say Oxygen Not Included, tiles have properties such as material, temperature, quality of air, pressure, etc...
It's actually very easy to understand how to model these kind of data. It can just be a massive grid, each element of that grid is a tile, tiles are regrouped by chunks. A chunk getting a change would be set to dirty, then get reupdated, etc... Of course it's not easy to make, but it's easy to understand,
Now what about Noita? Could it be the same way to model the environment, but at the level of a pixel instead of a tile? Would that not be way too costly?
Just trying to wrap my head around the concept...

• By achiga
I am new to shader programming. I was learning how to detect edges using shaders these days. And I found the UnityChan toon shader project. But I found it difficulty to understand when I read its implementation of sobel filter.
Especially for these s few lines:(From Line166)
depthsDiag -= centerDepth;
depthsAxis /= centerDepth;
It seems to try to get the diagonal and axial depth values. But I don't know why we need to substract centerDepth from depthsDiag and divide depthsAxis by centerDepth?

Another confusion comes from these a few lines when it tries to return the final color from fragment shader:(From line 191)

float SobelX = dot(SobelH, float4(1,1,1,1));
float SobelY = dot(SobelV, float4(1,1,1,1));   float Sobel = sqrt(SobelX * SobelX + SobelY * SobelY);       Sobel = 1.0-pow(saturate(Sobel), _Exponent);   //NK   float4 Col = tex2D(_MainTex, i.uv[0].xy);   Col = _EdgesColor * Col * (1.0 - Sobel) + Sobel;   return Col * lerp(tex2D(_MainTex, i.uv[0].xy), _TestColor, _BgFade);
What does Sobel = 1.0-pow(saturate(Sobel), _Exponent) do? And what dose Col = _EdgesColor * Col * (1.0 - Sobel) + Sobel mean?
Sorry for my bad English and hope anyone could help me understand this.
• By NikiTo
I am having an argument about this with a buddy now, and we don't believe each other.

• Hello everyone! I'm writing research on topic - "The best game engine for a small team to create 2D indie games". I need a research consultant or any feedback in this topic because I can't do it on my own - I'm not a scientist or someone else in game engines and other things. I've chosen criteria and game engines for research. They are in image "table.png" below. As we can see, it contains 6 game engines and 10 criteria. I want to know - are this criteria and game engines suitable for my research? In "Notes" section I put quiestion to help respondents answering.
In the last line - Value of criteria(in different words, multiplier), I wrote my own estimation of each criterion. This method let us to get different order of engines for every game development team. For example, if I value "Learning Curve" and "Ease of use" I will get one order, but if I value "Genre" and "Extra Benefits" I will get another order. Thank you for your attention. I hope you will leave feedback. Have a nice day :3.
×

## Important Information

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!