Jump to content
  • entries
    5
  • comments
    11
  • views
    1314

Implementing Neural Evolution for generic species in Unity

AlphaSilverback

2944 views

I did some evaluations on the time I need to spend on AI. It is not that difficult to create AI. It’s not even difficult to create impossibly difficult AI. What is really hard, is to create interesting AI.  So – I started researching neural evolution. I spent a day reading through the Neural Evolution of Augmented Topologies (NEAT) by Kenneth O. Stanley a couple of times. A paper can be found here:

http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf

The algorithm is not that difficult to understand, but you need a bit of knowledge about neural networks and how they work to understand this paper. Luckily, I had already taken a course in Machine Learning by Andrew Ng and Stanford University. I would really recommend this to all interested in the field.

Concretely for my case, I wanted to see if implementing this for the Moose in this game (and potentially all other species) would be a viable solution to create some interesting AI. I could then run a learning simulation for some time to create some base for these creatures, then add them with some mediocre intelligence to the version I ship to the end-users, but keep the ability to evolve the genome for the creatures alive in the game.

NEAT in short

For those who don’t know much about the learning algorithms, I just want to explain a little about how they work and the purpose of them.   Neural networks are a computer model imitating the neurons, axons and dendrites that the brain consists of. Every neuron is a kind of computer that does a single calculation based on the input it gets. It then sends the calculated value to all the neurons it is connected to scaled by the connections.

Here’s a picture of a classical neural network:

tikz41.png

This network can be trained over many iterations of measuring how far off from the goal the network performs, (called the error value or cost value).  You then alter the connection values to try and better the performance for what you want.     This could be looking at an image of a letter or number and trying to classify which number or letter it is. Or it could be analyzing an image of a road and training a car to drive itself. Common for all the purposes is that you have a large list of training data that tells the system how far off from the actual case the system is. The pixels of a picture here is the input values on the left, and the prediction of what the hand-written number is comes out on the right.

For the case where you don’t have any training data, you can use something like neural evolution. The requirement here then is, that you supply the system with a Fitness function. The fitness function will supply the system by awarding “points” to the fitness of the performing genome every time it has randomly evolved to do something that has made it survive.

For a very “simple” example, I’d recommend you watch Seth Bling’s version on YouTube:

 

Here, he creates a system to teach a computer to play Mario. The fitness function is just a value of how far over time Mario has moved. This works fine for this case. The inputs are the fields of blocks on the screen which he has classified to be two different types (standable, and enemy).

It is a rather impressive implementation.

So - an evolution algorithm works by creating a genome, mutate it to add random neurons and random connections between them, then measure how well they do before they die (or some other kill function). We then take the best few of the saved genomes and fuse them together. The fusion compares the genes in the genome and adds the matching genes from the parents with a value from either parent. It then takes the genes, that doesn’t match up, from the best performing parent (the one with the highest fitness number) and puts it into the new child.

What is essential to get NEAT to work, is understanding how a genome is built up. Kenneth Stanley has some quite smart ideas on tracking where the different genes comes from.   A genome here is two lists describing the neurons in the system and the connections between the neurons in the brain.

image.png.84aadee256f8149491934ab732f7db9c.png

 

The connections have an extra feature. They have what he calls an Innovation number attached, which describes when in the simulation this connection has appeared. It is this value that is used to compare the genes when fusing the genomes.

As the simulation runs, and you breed the best performing creatures/genomes, you get better and better performing creatures.  How long this simulation will take to run depends on the complexity of how you want the system to perform.

Implementing the System

Running the Neural Network

I started by planning the interface for the system that should be run.

To make the evolution system generic, the “brainstem” must be defined for each creature. Analogous to the real brain, this system will need a coupling between the actual brain and the inputs/outputs that each creature can have.

350px-Blausen_0114_BrainstemAnatomy.png

 

The brainstem works as a relay, interpreting the signals from the body and other senses and forwards these signals to the brain. Simultaneously, these signal are dampened or enhanced depending on the severity of the signal and possibly also per specific signal.   The brainstem also converts signals from the brain to nerve impulses to the body and muscles.

It is this module, that should be defined for each creature in order to make a generic learning algorithm that all creatures can be run through.

So a general model for how the brain iteration could work is something like this:

image.png.977da3f821972f702270db52480e527c.png

Every iteration start with the creature “Sensing” what is around it. For the Moose, I wanted to include stuff like its hunger and thirst level, because I wanted to see if it could evolve to be smart enough to find food or water when it was almost dead. I had a long list of stuff as the sensory.

The Sensory is gathered and sent to an adapter that inputs these values to the neural network. The network is iterated through and the output values is again sent to an adapter that interprets each of the output signals. The output signals are translated into actions and sent to a Resonator module. The function of this is to save the last output from the neural network and perform the given actions until the next output from the neural network comes. The reason I created this module was that I wasn’t sure my little laptop could keep up with running the network every frame. This means I can turn down the speed to something like ten times per second, but the creature would still act in between iterations.

The Evolution System

When the creatures die or are terminated for some reason, the evolution system come into play. I started by making what you could call a cradle for the evolution to start in. This was just a terrain where I put up four walls to create and area of something like a square kilometer. To start with, I made 26 copies of the same creature that was “protected” by the evolution system – meaning that when they die, it’s body isn’t deleted, only its brain. This way, I made sure that no matter how much bad luck they would have, at least 26 entities would be simulating at any given moment.

When the creatures did nothing for a wanted amount of time, I terminated their brain and created/fused/mutated a new one and injected this brain into the creature along with resetting its vitals.

The Neural Network Manager attached to each creature records the performance of the performing creature and saves its fitness number to a file when it dies.  The Neural Evolution Manager is then responsible for finding the genomes for the specific creature and breeding the best of them, mutating them and instantiating the new brain and injecting it into the creature again.  The same algorithm is used when a natural birth occurs, only these individuals die for real when they run out of food or water.

image.png.c4e1acf4f141eef49c315c3fa61fc2a3.png

Presentation

For debugging any system like this, testing and unit tests are necessary. Although, with real-time simulations, you cannot necessarily test every scenario that the system will experience, mainly because it can be hard to imagine every scenario. So visualizing the actual neural network is also vitally necessary.

Here’s a video of the performing system from the beginning.

https://drive.google.com/open?id=0BxnLa_qsqQBoR3Q5S2IzS2dzRlU
And another session: https://drive.google.com/file/d/0BxnLa_qsqQBoVGpQd3ZEMTBsNzg/view?usp=sharing

What you can see is me starting by setting some parameters for the evolution system. These are values such as how long the creature can stand still and do nothing before termination or how much its fitness has to change to be recorded. There’s also a value that controls how many loop iterations the performing neural network can take. This value is needed, because two nodes can feed each other, creating an infinite loop that never returns.

The next thing that happens is… nothing. Nothing happens for the first 10-20 seconds. Eventually, a creature starts reproducing, running, turning or eating, which is awarded fitness points. So the next generation of creatures all inherit these features and half of them starts doing that.

The green orbs in the system are input values and are shown for the current creature being viewed on the left. The red orbs are the output values for the current creature being viewed.  The yellow orbs that begin to appear in between are “hidden neurons”. The lines between them are connections and its scaling value is representing by how much color it is shown in (black/grey means close to zero).

Eventually they develop some behavior that gets them forward. at generation 10, they start to be able to detect obstacles in front of them, turn and conserve energy based on the steepness of the surface they stand on. This is very interesting to me. I really feel like this has great potential, once I clean up the input values and normalize them. 

If any of you are interested in seeing this early build and reading the source code, I’d be happy to share it.

What's next?

I need to supply a hunter/predator to the training simulation. This is just going to be a placeholder – most likely an animated red box with the “Player” tag. This being a hunter following and killing the creature when it is close enough. This will eventually train the creatures to flee, turn or attack at the right moment so that the creature doesn’t get killed, thus survive and increase its fitness.  This requires the creatures to know the relative direction to the predator, and perhaps the distance.  Another critical optimization for this system is normalizing the input values to be between zero and one. Some of these values are vectors and requires vector normalization, and I fear these calculations may be hard on the system and require me to turn down the iterations per second.

I will optimize this further and begin some game tests when I have the time.




5 Comments


Recommended Comments

I apologize for the bad video quality. For some reason it gets worse when I upload it. I'll find another place to upload it to or try recording a better quality.

Share this comment


Link to comment

 

1 hour ago, Scouting Ninja said:

If you have a google drive account you can link directly from it.

That was what I tried, but the video is not showing in the res I recorded in or have on disc. I will attempt again. Thank you for your help!

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement
  • Advertisement
  • Blog Entries

  • Similar Content

    • By Rio Lloyd
      Hey all!
      we are a team of 3 looking for more members, 
      we are making an isometrical Survival RPG.
      we are looking For Members who can make low poly 3D artists who can do character models, environments, tools and more.
       
      if interested and want to know more email me at rioishere14@gmail.com
    • By nxrighthere
      BenchmarkNet is a console application for testing the reliable UDP networking solutions.
      Features:
      Asynchronous simulation of a large number of clients Stable under high loads Simple and flexible simulation setup Detailed session information Multi-process instances Supported networking libraries:
      ENet UNet LiteNetLib Lidgren MiniUDP Hazel Photon Neutrino DarkRift More information and source code on GitHub.
      You can find the latest benchmark results on the wiki page.
       
    • By trapazza
      I'm trying to add some details like grass, rocks, trees, etc. to my little procedurally-generated planet. The meshes for the terrain are created from a spherified cube which is split in chunks (chunked LOD).
      To do this I've wrote a geometry shader that takes a mesh as input and uses its vertex positions as locations where the patches of grass will be placed (as textured quads).
      For an infinite flat world (not spherical) I'd use the terrain mesh as input to the geometry shader, but I've found that this won't work well on a sphere, since the vertex density is not homogeneous across the surface.
      So the main question would be: How to create a point cloud for each terrain chunk whose points were equally distributed across the chunk?
      Note: I've seen some examples where these points are calculated from intersecting a massive rain of totally random perpendicular rays from above... but I found this solution overkill, to say the least.
      Another related question would be: Is there something better/faster than the geometry shader approach, maybe using compute shaders and instancing?
    • By FedGuard
      Hello all,
       
      I would like to start off with thanking you all for this community. Without fora like these to assist people the already hard journey to making an own game would be exponentially more difficult. Next I would like to apologize for the long post, in advance...
      I am contemplating making a game. There, now that's out of the way, maybe some further details might be handy.
      I am not some youngster (no offence) with dreams of breaking into the industry, I am 38, have a full-time job, a wife, kid and dog so I think I am not even considered indie? However I recently found myself with additional time on my hands and decided I would try my hand at making a game.Why? Well mostly because I would like to contribute something, also because I think I have a project worth making (and of course some extra income wouldn't hurt either to be honest). The first thing I realized was, I have absolutely no relevant skill or experience. Hmm; ok, never mind, we can overcome that, right?
      I have spent a few months "researching",meaning looking at YouTube channels, reading articles and fora. Needless to say, I am more confused now than when I started. I also bought some courses (Blender, Unity, C#) and set out to make my ideas more concrete.
      I quickly discovered, I am definitely not an artist... So I decided, though I do plan to continue learning the art side eventually, I would focus on the design and development phase first. The idea being, if it takes me a year or more solely learning stuff and taking courses without actually working on my game, I would become demoralized and the risk of quitting would increase.
      So I thought I would:
      1: Keep following the courses Unity and C# while starting on the actual game development as the courses and my knowledge progress.
      2: Acquire some artwork to help me get a connection with the game and main character, and have something to helm keep me motivated. (I already did some contacting and realized this will not be cheap...). Also try to have the main character model so I can use it to start testing the initial character and game mechanics. For this I have my first concrete question. I already learned that outsourcing this will easily run up in the high hundreds or thousands of dollars... (lowest offer so far being 220 USD) I am therefore playing with the idea of purchasing https://assetstore.unity.com/packages/3d/animations/medieval-animations-mega-pack-12141 with the intention of then have an artist alter and/or add to the animations (it is for a Roman character so some shield animations are not going to work the same way.). This way I could start  with the basic character mechanics. Is this a good idea, waste of money,...? Any suggestions? I then have a related but separate question. Is it a good idea to buy Playmaker (or some other similar software I haven't yet heard of like RPGAIO), and using this for initial build, then changing/adding code as the need arises?
      3.Get a playable initial level ready as a rough demo and then starting to look for artist for level design and character/prop creation.
      ...
       
      I would really appreciate some input from more experienced people, and especially answers to my questions. Of course any advice is extremely welcome.
    • By GameTop
      Dirt Bike Extreme - another game made with Unity. Took about 2 months to complete.
      Take part in extreme motorcycle races across the dangerous and challenging tracks. Dirt Bike Extreme is easy to pick up but hard to master. Race, jump and crash your way and other mad rivals through the amazing tracks as you master the skills and physics of motocross in this high-speed racing adventure. Conquer challenging routes on 23 different runs, discover new bikes and become the best of the best! Over 257K downloads already!
      Windows Version:
      https://www.gametop.com/download-free-games/dirt-bike-extreme/

      Mac Version:
      https://www.macstop.com/games/dirt-bike-extreme/
       

       


×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!