• # Neural Networks Neural Networks 101

Artificial Intelligence

Intention

How Neural Networks Work

Neural networks work by using a system of receiving inputs, sending outputs, and performing self-corrections based on the difference between the output and expected output, also known as the cost.

Neural networks are composed of neurons, which in turn compose layers, or collections of neurons. For example, there is an input layer and an output layer. In between the these two layers, there are layers known as hidden layers. These layers allow for more complex and nuanced behavior by the neural network. A neural network can be thought of as a multi-tier cake: the first tier of the cake represents the input, the tiers in between, or lack thereof, represent the hidden layers, and the last tier represents the output.

The two mechanisms of learning are Forward Propagation and Backward Propagation. Forward Propagation uses linear algebra for calculating what the activation of each neuron of the next layer should be, and then pushing, or propagating, those values forward. Backward Propagation uses calculus to determine what values in the network need to be changed in order to bring the output closer to the expected output.

Forward Propagation

As can be seen from the gif above, each layer is composed of multiple neurons, and each neuron is connected to every other neuron of the following and previous layer, save for the input and output layers since they are not surrounding by layers from both sides.

To put it simply, a neural network represents a collection of activations, weights, and biases. They can be defined as:

• Activation: A value representing how strongly a neuron is firing.
• Weight: How strong the connection is between two neurons. Affects how much of the activation is propagated onto the next layer.
• Bias: A minimum threshold for whether or not the current neuron's activation and weight should affect the next neuron's activation.

Each neuron has an activation and a bias. Every connection to every neuron is represented as a weight. The activations, weights, biases, and connections can be represented using matrices. Activations are calculated using this formula:

After the inner portion of the function has been computed, the resulting matrix gets pumped into a special function known as the Sigmoid Function. The sigmoid is defined as:

The sigmoid function is handy since its output is locked between a range of zero and one. This process is repeated until the activations of the output neurons have been calculated.

Backward Propagation

The process of a neural network performing self-correction is referred to as Backward Propagation or backprop. This article will not go into detail about backprop since it can be a confusing topic. To summarize, the algorithm uses a technique in calculus known as Gradient Descent. Given a plane in an infinite number of dimensions, the direction of change that minimizes the error must be found. The goal of using gradient descent is to modify the weights and biases such that the error in the network approaches zero.

Furthermore, you can find the cost, or error, of a network using this formula:

Unlike forward propagation, which is done from input to output, backward propagation goes from output to input. For every activation, find the error in that neuron, how much of a role it played in the error of the output, and adjust accordingly. This technique uses concepts such as the chain rule, partial derivatives, and multi-variate calculus; therefore, it's a good idea to brush up on one's calculus skills.

High Level Algorithm

1. Initialize matrices for weights and biases for all layers to a random decimal number between -1 and 1.
2. Propagate input through the network.
3. Compare output with the expected output.
4. Backwards propagate the correction back into the network.
5. Repeat this for N number of training samples.

Source Code

If you're interested in looking into the guts of a neural network, check out AI Chan! It's a simple to integrate library for machine learning I wrote in C++. Feel free to learn from it and use it in your own projects.

Resources

Report Article

## User Feedback

You need to be a member in order to leave a review

## Create an account

Register a new account

3 of 3 members found this review helpful 3 / 3 members

Overall, this article is not very useful. The wikipedia page on artificial neural networks is a far better starting place. There are much better articles on neural networks elsewhere on the internet and actually have enough depth to give the reader the capability to create their own ANN. This article also completely misses several very important topics to ANN's which cannot be skipped:
-Deep learning
-How to train an ANN with a training set

Forward propagation:
The sigmoid function is mentioned, but there is no mention of the commonly used RelU function: return (X>0)?X:0;

Backpropagation:
The meat and potatoes of ANN is in the back propagation algorithm. This article just says, "It's confusing, so look it up elsewhere".

I also think that if you're going to do an introductory article on artificial neural networks, it might be more valuable to skip the technical details of implementation all together and just talk about all the different types of machine learning ANN's are out there, what they're useful for, and generally how they work. Maybe something about ANN's, reinforcement learning, convolutional neural networks, LSTM's, etc.

• 0
• 0
• 1
• 1
• 3

• 32
• 13
• 9
• 11
• 9
• ### Similar Content

• By Waaayoff
I'm looking for an algorithm that I can use to remove vertices that are close to each other within a margin of error in a triangular mesh. Pretty much similar to Blender's "Remove Doubles" feature, if anyone is familiar with it.
I think the issue isn't just removing the doubles, but also how would I handle the face indices once I remove "duplicate" vertices?
• By iGrfx
I've learned that the triangle clipping in the rasterization process usually using Sutherland–Hodgman algorithm. I also found an algorithm called "Guard-band". I'm writing a software raster so I want to know what technical the GPU use, I want to implement it for study. Thanks!
updated: what's the more proper triangulate algorithm?

• Welcome to the first part of multiple effect articles about soft shadows. In recent days I've been working on area light support in my own game engine, which is critical for one of the game concepts I'd like to eventually do (if time will allow me to do so). For each area light, it is crucial to have proper soft shadows with proper penumbra. For motivation, let's have the following screenshot with 3 area lights with various sizes:

Fig. 01 - PCSS variant that allows for perfectly smooth, large-area light shadows

Let's start the article by comparison of the following 2 screenshots - one with shadows and one without:

Fig. 02 - Scene from default viewpoint lit with light without any shadows (left) and with shadows (right)

This is the scene we're going to work with, and for the sake of simplicity, all of the comparison screenshots will be from this exact same viewpoint with 2 different scene configurations. Let's start with the definition of how shadows are created. Given a scene and light which we're viewing. Shadow umbra will be present at each position where there is no direct visibility between given position and any existing point on the light. Shadow penumbra will be present at each position where there is visibility of any point on the light, yet not all of them. No shadow is everywhere where there is full direct visibility between each point on the light and position.
Most of the games tend to simplify, instead of defining a light as area or volume, it gets defined as an infinitely small point, this gives us few advantages:
For single point, it is possible to define visibility in a binary way - either in shadow or not in shadow From single point, a projection of the scene can be easily constructed in such way, that definition of shadow becomes trivial (either position is occluded by other objects in the scene from lights point of view, or it isn't) From here, one can follow into the idea of shadow mapping - which is a basic technique for all others used here.

Trivial, yet should be mentioned here.
inline float ShadowMap(Texture2D<float2> shadowMap, SamplerState shadowSamplerState, float3 coord) { return shadowMap.SampleLevel(shadowSamplerState, coord.xy, 0.0f).x < coord.z ? 0.0f : 1.0f; } Fig. 03 - code snippet for standard shadow mapping, where depth map (stored 'distance' from lights point of view) is compared against calculated 'distance' between point we're computing right now and given light position. Word 'distance' may either mean actual distance, or more likely just value on z-axis for light point of view basis.

Which is well known to everyone here, giving us basic results, that we all well know, like:

Fig. 04 - Standard Shadow Mapping

This can be simply explained with the following image:

Fig. 05 - Each rendered pixel calculates whether its 'depth' from light point is greater than what is written in 'depth' map from light point (represented as yellow dot), white lines represent computation for each pixel.

Percentage-Close-Filtering (PCF)
To make shadow more visually appealing, adding soft-edge is a must. This is done by simply performing NxN tests with offsets. For the sake of improved visual quality I've used shadow mapping with bilinear filter (which requires resolving 4 samples), along with 5x5 PCF filtering:

Fig. 06 - Percentage close filtering (PCF) results in nice soft-edged shadows, sadly the shadow is uniformly soft everywhere

Clearly, none of the above techniques does any penumbra/umbra calculation, and therefore they're not really useful for area lights. For the sake of completeness, I'm adding basic PCF source code (for the sake of optimization, feel free to improve for your uses):
inline float ShadowMapPCF(Texture2D<float2> tex, SamplerState state, float3 projCoord, float resolution, float pixelSize, int filterSize) { float shadow = 0.0f; float2 grad = frac(projCoord.xy * resolution + 0.5f); for (int i = -filterSize; i <= filterSize; i++) { for (int j = -filterSize; j <= filterSize; j++) { float4 tmp = tex.Gather(state, projCoord.xy + float2(i, j) * float2(pixelSize, pixelSize)); tmp.x = tmp.x < projCoord.z ? 0.0f : 1.0f; tmp.y = tmp.y < projCoord.z ? 0.0f : 1.0f; tmp.z = tmp.z < projCoord.z ? 0.0f : 1.0f; tmp.w = tmp.w < projCoord.z ? 0.0f : 1.0f; shadow += lerp(lerp(tmp.w, tmp.z, grad.x), lerp(tmp.x, tmp.y, grad.x), grad.y); } } return shadow / (float)((2 * filterSize + 1) * (2 * filterSize + 1)); } Fig. 07 - PCF filtering source code

Representing this with image:

Fig. 08 - Image representing PCF, specifically a pixel with straight line and star in the end also calculates shadow in neighboring pixels (e.g. performing additional samples). The resulting shadow is then weighted sum of the results of all the samples for a given pixel.

While the idea is quite basic, it is clear that using larger kernels would end up in slow computation. There are ways how to perform separable filtering of shadow maps using different approach to resolve where the shadow is (Variance Shadow Mapping for example). They do introduce additional problems though.

To understand problem in both previous techniques let's replace point light with area light in our sketch image.

Fig. 09 - Using Area light introduces penumbra and umbra. The size of penumbra is dependent on multiple factors - distance between receiver and light, distance between blocker and light and light size (shape).

To calculate plausible shadows like in the schematic image, we need to calculate distance between receiver and blocker, and distance between receiver and light. PCSS is a 2-pass algorithm that does calculate average blocker distance as the first step - using this value to calculate penumbra size, and then performing some kind of filtering (often PCF, or jittered-PCF for example). In short, PCSS computation will look similar to this:
float ShadowMapPCSS(...) { float averageBlockerDistance = PCSS_BlockerDistance(...); // If there isn't any average blocker distance - it means that there is no blocker at all if (averageBlockerDistance < 1.0) { return 1.0f; } else { float penumbraSize = estimatePenumbraSize(averageBlockerDistance, ...) float shadow = ShadowPCF(..., penumbraSize); return shadow; } } Fig. 10 - Pseudo-code of PCSS shadow mapping

The first problem is to determine correct average blocker calculation - and as we want to limit search size for average blocker, we simply pass in additional parameter that determines search size. Actual average blocker is calculated by searching shadow map with depth value smaller than of receiver. In my case I used the following estimation of blocker distance:
// Input parameters are: // tex - Input shadow depth map // state - Sampler state for shadow depth map // projCoord - holds projection UV coordinates, and depth for receiver (~further compared against shadow depth map) // searchUV - input size for blocker search // rotationTrig - input parameter for random rotation of kernel samples inline float2 PCSS_BlockerDistance(Texture2D<float2> tex, SamplerState state, float3 projCoord, float searchUV, float2 rotationTrig) { // Perform N samples with pre-defined offset and random rotation, scale by input search size int blockers = 0; float avgBlocker = 0.0f; for (int i = 0; i < (int)PCSS_SampleCount; i++) { // Calculate sample offset (technically anything can be used here - standard NxN kernel, random samples with scale, etc.) float2 offset = PCSS_Samples[i] * searchUV; offset = PCSS_Rotate(offset, rotationTrig); // Compare given sample depth with receiver depth, if it puts receiver into shadow, this sample is a blocker float z = tex.SampleLevel(state, projCoord.xy + offset, 0.0f).x; if (z < projCoord.z) { blockers++; avgBlockerDistance += z; } } // Calculate average blocker depth avgBlocker /= blockers; // To solve cases where there are no blockers - we output 2 values - average blocker depth and no. of blockers return float2(avgBlocker, (float)blockers); } Fig. 11 - Average blocker estimation for PCSS shadow mapping

For penumbra size calculation - first - we assume that blocker and receiver are plannar and parallel. This makes actual penumbra size is then based on similar triangles. Determined as:
penmubraSize = lightSize * (receiverDepth - averageBlockerDepth) / averageBlockerDepth This size is then used as input kernel size for PCF (or similar) filter. In my case I again used rotated kernel samples. Note.: Depending on the samples positioning one can achieve different area light shapes. The result gives quite correct shadows, with the downside of requiring a lot of processing power to do noise-less shadows (a lot of samples) and large kernel sizes (which also requires large blocker search size). Generally this is very good technique for small to mid-sized area lights, yet large-sized area lights will cause problems.

Fig. 12 - PCSS shadow mapping in practice

As currently the article is quite large and describing 2 other techniques which I allow in my current game engine build (first of them is a variant of PCSS that utilizes mip maps and allows for slightly larger light size without impacting the performance that much, and second of them is sort of back-projection technique), I will leave those two for another article which may eventually come out. Anyways allow me to at least show a short video of the first technique in action:

Note: This article was originally published as a blog entry right here at GameDev.net, and has been reproduced here as a featured article with the kind permission of the author.
You might also be interested in our recently featured article on Contact-hardening Soft Shadows Made Fast.

• Intro
Due to my belief in learning through self-discovery and my ongoing creative evolution, I've long put off doing any tutorials. However, after making pixel art for over 3 years I've established many solid techniques worth laying out in a concrete fashion. While I'm excited by the prospect of helping others with my experience, I still urge artists to explore things their own way. The wonderful thing about art is the unlimited number of solutions to a problem. I offer you solutions that have worked for me and I hope they work for you, but I will be even more thrilled if you discover a better solution along the way.

When it comes to pixel art, it all starts with a good color palette. Creating a custom color palette can be a very satisfying and powerful way to establish your own unique look. I'll guide you through my method as I create a new palette. But first, let's go over some basic principals.

I find it easiest to understand and control color through HSB.
Hue - The actual color (0 - 360º)
Saturation - The intensity or purity of a color (0 - 100%)
Brightness - The amount of black or white mixed with a color (0 - 100%)
By understanding and adjusting these 3 fundamental properties you can create custom colors with precise control. I recommend this article by Steven Bradley for more detailed definitions of HSB.

Color Ramps
A color ramp is a specific range of colors that work well together, arranged according to brightness. Here is an example of what I consider a good color ramp.

Brightness steadily increases from left to right in this example. As the colors reach high brightness levels it's important to decrease saturation, or you'll end up with intense eye burning colors. Also, colors with very low brightness can become overly rich and weighty with high saturation. Saturation peaks in the middle swatch in this example.
A good color ramp should also apply hue-shifting, which is a transition in hue across the color ramp. In the previous example the hue is shifting by positive degrees as the brightness increases.
Many beginners overlook hue-shifting and end up with 'straight ramps' that only transition brightness and saturation. There is no law that says you can't do this but the resulting colors will lack interest and be difficult to harmonize with ramps of a different hue. This only makes sense to me if you want a monochromatic look and stick to one straight ramp.

The Palette
A color ramp is essentially a palette, but most palettes contain multiple ramps. I like to create large palettes with lots of ramps, which I can then pull smaller palettes from per assignment.
Mondo  - 128 colors

I took the opportunity to make a brand new palette for this tutorial. My intention was to create a general purpose palette that strikes a balance between vibrant colors and desaturated natural colors. So, how to make such a large palette?
First I decide how many swatches I want per ramp and how many degrees of hue shift. For this palette I want 9 swatches per ramp with 20 degrees of positive hue shift between each swatch. I like a lot of hue shift because it creates harmony between ramps and just looks neat, but 20 is about as high as I go.

The color picker panel in Photoshop. We only need to be concerned with adjusting HSB.

I use Photoshop, but a similar color picker panel should be accessible in just about any graphics software. To start I pick a color that will fit right in the the middle of a ramp. The hue is somewhat arbitrary, but the saturation and brightness is critical. I want the middle color to be be the most vibrant so I set the saturation and hue to the max combined number I'm willing to go.

After I've chosen my first color I can set the hue for the remaining swatches based on the positive 20 degree shift I wanted. I could reverse the direction of hue shift if I want but positive hue shift usually results in more natural colors, warming as they become brighter.
I still need to sort out the increments for S&B. Unlike hue, shifting the S&B in uniform increments doesn't necessarily produce satisfactory results. However, there are a few tendencies I follow. Brightness consistently increases from left to right and usually never starts at 0, unless I want black. Saturation peaks around the middle and never fully goes to 100, or 0. The goal in mind is to create even contrast between each color.

After some tuning and eyeballing these are my final values and resulting color ramp. The hue shift looks pretty strong but it will make sense when I add more ramps.

This version shows the difference in the increments. Pay attention to what the S&B are doing. You can see there is some consistency in the pattern. The saturation takes larger steps on the ends and smaller steps in the middle where it's the highest percentage. The brightness takes smaller steps as it gets closer to the end at full 100%.

Here's another visualization that clearly shows the flow of S&B as line graphs. You don't have to follow this general flow of S&B. It just depends what look you're going for. I've made ramps where the saturation continues to climb as the brightness decreases, creating an X pattern. This results in vivid dark colors. The biggest mistake is combining high saturation and brightness, unless you want to burn some eyeballs. I recommend a lot of experimentation with the HSB values of your ramp. I've tried to come up with mathematically precise formulas but it always seems to come down to trusting the eyeballs to some extent.
Now let's finish the palette.

Up to this point all I have been doing is picking colors and drawing them as single pixel dots on a tiny canvas. I haven't actually added any swatches into the swatch panel. With the first ramp established all I have to do to create more ramps for my palette is shift the entire set of hues.

I want 8 ramps total so I will shift the hues of each ramp by 45 degrees to complete the 360 degree cycle around the color wheel. I could do this in the color picker by adjusting the H value one color at a time, but In Photoshop I can save a lot of time by duplicating the ramp and changing the hue of the entire selection (Image-Adjustments-hue/saturation, or ⌘+U).

After adjusting the hue of all my color ramps my palette appears like this. It looks pretty nice but It's lacking more neutral desaturated colors.

To add desaturated colors I duplicate the whole middle section of the palette, omitting only the darkest and lightest colors on the ends, flip it over and desaturate them with the Hue/Saturation panel. I omit the light and dark columns because they appear nearly the same as the originals. I flip the colors because it makes for easy navigation, and it looks cool. The desaturated colors can provide a more natural look, and work well as grays in combination with the vibrant colors.

The final task is actually adding the colors into the swatch panel. With the color picker panel open I sample each color with the eyedropper and click the 'Add to Swatches' button. I add them from left to right, top to bottom so they will appear in the swatch panel in the correct order. This is quite tedious but the only way I know of to add the colors in the particular order I want.

Once I've added all the colors into the swatch panel I click on the panel options and make sure to save my palette. I can then easily load the palette as a .aco file into the swatch panel anytime. Also, by selecting 'Save Swatches for Exchange' you can create a .ase file, which can be loaded into several other Adobe programs. Save the image of your palette as a .png file and you can load it into Aseprite.
Well, that completes my 128 color palette - Mondo. Now let's look at how I use the palette with some examples.

Picking Colors

This example keeps it pretty simple, mostly relying on horizontal ramps of adjacent colors. You can also see how the warm desaturated colors work nicely with the vivid hues. I've added white into palette for extra contrast.

This example shows how ramps can move horizontally and diagonally. Because of the hue shift every color is surrounded by colors that can work together.

Harmony is everywhere, just pick and play!

This example uses complimentary color in combination with neutrals. The result captures an ominous yet hopeful feeling that perfectly fits the mood I wanted.
Picking colors for your art always requires some good sense, but a versatile palette with criss-crossing ramps like this makes it much easier. A little color goes a long way with pixel art, as you can see I never use a lot of colors for any one image.
Creating a palette with this method also works great for game art, and will ensure everything in your game has consistent colors. I used this method to create a 160 color palette for Thyrian Defenders. We've been able to depict an incredible range of environments and characters while maintaining a consistent look overall. Other aesthetic choices come into play, but color is the fundamental ingredient that ties everything together.

Final Word
Overall I'm quite happy with how this palette turned out. I think you'll be seeing more of my work in the Mondo palette from now on!
I hope this helps you come up with some palettes of your own. I know It can take a bit of time to get a feel for HSB, but even if you're a beginner I think making palettes like this is a great way to understand color. Go crazy with HSB and don't be afraid to experiment with formulas that look different than my example. Also, you don't have to make such a large palette. Start with trying to make a small ramp.

Raymond Schlitter (Slynyrd) is a former graphic designer who turned his creative passion to pixel art and game design in early 2015. Now he shares his knowledge with tutorials while he continues to make fantastic art and work on games. Support him on Patreon and get the inside scoop on his latest work.

Note: This post was originally published on Raymond's blog, and is reproduced here with kind permission from the author.  If you enjoyed this article please consider supporting Raymond on Patreon, where he provides backers with exclusive downloads such the Mondo palette as .aco, .ase, and .png files. Get Mondo!  You can also make a one time donation to the author if you prefer not to subscribe on Patreon.
[Wayback Machine Archive]
• By lilbump
Hi guys i'm new here, i really hope my question won't sound utterly stupid..
I'd like to know whether it's better to use a PRNG or a regular RNG, if you are trying to program your own video slot machine. Actually i don't even have clearly understood the difference between the two =D
2nd question is: which developer i should rely on? I'm following this guide, they talk about RNG but not which one or where to find it.
×

## Important Information

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!