I have some basic experience with Unity. Specifically I used it to make 3D cellular automata. I also have limited, at least theoretical, knowledge of neural networks and I have used Encog a few times.

Now I would like to make a game, nothing complex, where several agents with ANN for brain, navigate an environment, interact with it and with each other. For start "eating" and moving is more than enough. Once fully finished I would like to use it for my thesis.

The thing is: I would like these agents to learn in real time, without predefined data set. That is, the network receives a stimulus, reacts and applies a learning rule while the game is running. Kind of like Norns in the Creatures game series.

Where should I start? Are there any libraries I should/could use? Are there any guides or tutorials that could help me?

I appreciate your input.

Since you don't seem to be getting help from anyone else, I'll give it a try.

A lot of people start projects similar to yours and try to train the neural networks using a genetic algorithm. But you can do much better than that.

I'll assume you have a finite set of possible actions (e.g., do nothing, advance, turn left, turn right). Here's what I would do:
* Make an ANN that uses sensor readings as input and has one output per allowed action (a standard feed-forward network with a couple of hidden layers will do).
* With probability p (say, p=0.99) you'll pick the action that has the largest output; with probability (1-p) you pick a random action.
* Keep a history of every situation you have encountered (or perhaps only the last N situations, if memory becomes a concern). Every time step store the inputs, the action that was taken, any rewards obtained at this time and the next set of inputs.
* Train the ANN using a minibatch formed with a random sample of situations from the history.

This is a version of Q-learning. You can find the details in this fascinating paper (Algorithm 1, at the top of page 5).

Let me know if you have a hard time getting started.

Thank you. The paper is indeed interesting, I've read about the work this team did with Atari games a while ago but it never occurred to me that I could use similar techniques for my project. Once I finish semesteral exams I will start working on it.

Thank you again.