• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
CryoGenesis

Neural Network Math Help ? :)

17 posts in this topic

Hey, I'm trying to make my first neural network program and its working kind of well at the moment I just need some help with the math.
In the program the value of a neuron is the value of the neuron connected to it with the highest weight.
so if neuron 1 has the value of 5 and a weight of 2 and neuron 2 has the value of 2 and the weight of 5 were connected to the neuron then the neurons value would be 2.
I would like to make it so that it would be more of an average but based on weight.
so the average of neuron 1 and 2 would be 3 due to the weight of neuron 2 being bigger than the weight of 1.
Would anyone know of a formula that would achieve this?
0

Share this post


Link to post
Share on other sites
In a standard neural network, all neurons influence the neurons they are connected to. When a neuron is activated, it receives a series of inputs that map to a series of weights. The neuron sums each input multiplied by that input's weight ([url="http://en.wikipedia.org/wiki/Dot_product"]http://en.wikipedia.org/wiki/Dot_product[/url]), and then applies its activation function to the result.

[url="http://en.wikipedia.org/wiki/Artificial_neuron"]http://en.wikipedia.org/wiki/Artificial_neuron[/url]
0

Share this post


Link to post
Share on other sites
Basically if you have 2 input, 2 hidden:
Each input has a connection to each hidden, for 4 connections.
Each connection has a weight. When you pass a value from the input and input node to a hidden node along a connection, you multiply the input nodes value by the weight and add it to the node.

Each hidden node receives input from both input nodes, and the "influence" the input node has on the hidden node is determined by the connections weight. A weight of 0 would mean "no influence".

To make the network do nifty things you will need to add an activation function to each node. The activation function takes the nodes accumulated value as input, and then spits out a new number that is then passed on to each node it connects to. Typically people will choose tanh or another sigmoid type function.
0

Share this post


Link to post
Share on other sites
Hello

Could you please explain what is the problem you try to solve with an ANN ?
In that way, it'll be possible to propose an ANN type, and a learning algorithm [img]http://public.gamedev.net//public/style_emoticons/default/rolleyes.gif[/img]

I wish I can help ...

Nico
1

Share this post


Link to post
Share on other sites
You pretty much answered your own question, you want a weighted average of the inputs.

So instead of output = (n1 + n2 + ... + nx) / x, you will want something like output = (n1 * w1 + n2 * w2 + ... + nx * wx) / x.
0

Share this post


Link to post
Share on other sites
Hey Guys, I've attempted to solve this 3 times and I think I've got this working to some extent apart from one thing.
I'm using a sigmoid activation function to convert the multiple inputs to a floating point number between 0 and 1.
the weight is a random floating point number between -1 and 1.
The output of the neural network is always between 0.4 - 0.6.
I think this is because I'm giving node's output as the activation function.
Should I only use the activation function for the Output Nodes and the Hidden Nodes output is just its activation number? (i1*W1 + i2 + W2 ... In * Wn)
I think that would work because the node's output wont be too high or too low.

P.S: the program I am making doesn't use a backpropagation function to train the networks it uses a genetic algorithm to train the network.
0

Share this post


Link to post
Share on other sites
[quote name='CryoGenesis' timestamp='1332095370' post='4923084']
[left]The output of the neural network is always between 0.4 - 0.6.[/left]

...
(i1*W1 + i2 + W2 ... In * Wn)
[/quote]

... or maybe is it because you didn't put a bias to your nodes ?
([b]W0[/b] + i1*W1 + i2*W2 ... In * Wn) , where W0 is a weight with the constant input 1.

A simple way to add a bias is to add a 1.0 component to the input vector
1

Share this post


Link to post
Share on other sites
[quote][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left]... or maybe is it because you didn't put a bias to your nodes ?[/left][/size][/font][/color]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left]([/left][/size][/font][/color][b]W0[/b][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left] + i1*W1 + i2*W2 ... In * Wn) , where W0 is a weight with the constant input 1.[/left][/size][/font][/color]


[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left]A simple way to add a bias is to add a 1.0 component to the input vector [/left][/size][/font][/color][/quote]
Whats the point of Bias?
Is it needed for the neural network to function?
because I haven't put it in :/
0

Share this post


Link to post
Share on other sites
[quote name='CryoGenesis' timestamp='1332101788' post='4923102']
[quote]
[left]... or maybe is it because you didn't put a bias to your nodes ?[/left]

[left][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3]([/size][/font][/color][/left]
[b]W0[/b]
[left][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3]+ i1*W1 + i2*W2 ... In * Wn) , where W0 is a weight with the constant input 1.[/size][/font][/color][/left]



[left][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3]A simple way to add a bias is to add a 1.0 component to the input vector [/size][/font][/color][/left]
[/quote]
Whats the point of Bias?
Is it needed for the neural network to function?
because I haven't put it in :/
[/quote]

Oh and does the bias have to be used for every node or just the input nodes?
0

Share this post


Link to post
Share on other sites
[quote name='CryoGenesis' timestamp='1332101788' post='4923102']
Whats the point of Bias?
Is it needed for the neural network to function?
[/quote]

The point of Bias is to shift the activation function along x axis (and it can be considered as a constant input for the implementation as I suggested)
It's needed for practical purpose : if you don't use a bias, the function you are approximating (ie the problem you are solving) must pass threw (0,f(0)) where f is the activation function you chose. Otherwise, the net won't converge. With a bias you don't have this limitation anymore.

[quote]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left]Oh and does the bias have to be used for every node or just the input nodes? [/left][/size][/font][/color]
[/quote]
The bias has to be used with any node that does signal integration, so typically all the nodes except input ones (since these are just 'slots' to provide input to the net).
1

Share this post


Link to post
Share on other sites
[quote name='NicoLaCrevette' timestamp='1332103075' post='4923107']
[quote name='CryoGenesis' timestamp='1332101788' post='4923102']
Whats the point of Bias?
Is it needed for the neural network to function?
[/quote]

The point of Bias is to shift the activation function along x axis (and it can be considered as a constant input for the implementation as I suggested)
It's needed for practical purpose : if you don't use a bias, the function you are approximating (ie the problem you are solving) must pass threw (0,f(0)) where f is the activation function you chose. Otherwise, the net won't converge. With a bias you don't have this limitation anymore.

[quote]
[left]Oh and does the bias have to be used for every node or just the input nodes?[/left]

[/quote]
The bias has to be used with any node that does signal integration, so typically all the nodes except input ones (since these are just 'slots' to provide input to the net).
[/quote]

Thanks for the info but now it always returns between 0.6 - 0.7...

Here is the source code just in case I've done something wrong -_-

[CODE]
package AI;
public class Node {
public float[] weight;
public float value;
public float activation;
public final float e = 2.7183f;
public final float p = 1;

public boolean in = false;

public Node(float[] weight){
this.weight = weight;
}

public static void main(String[] args){
NeuralNetwork net = new NeuralNetwork(1,1,2,2);
net.createNetwork();
float[] f = {100f};
net.input(f);
System.out.println(net.getOutput(0));
}
public float activationSigmoidMethod(float activation){
double a = -activation/p;
double b = e;

double c = Math.pow(e, a);
double e = 1 + c;
double f = 1/e;

return (float) f;

}

public void input(Node[] node, int num){
if(in = true){
activation += 1;
}
for(int i = 0; i < node.length; i++){
activation += (node[i].value * node[i].weight[num]);
}
value = activationSigmoidMethod(activation);
activation = 0;
}

public float getOutput(){
return value;
}



}

[/CODE][CODE]
package AI;
import java.util.Random;
public class NeuralNetwork {
public Node[] in;
public Node[] out;
public Node[][] node;

public NeuralNetwork(int ins, int outs, int layers, int num){
in = new Node[ins];
out = new Node[outs];
node = new Node[layers][num];
}

public float[][] returnInWeights(){
float[][] ini = new float[in.length][node[0].length];
for(int i = 0; i < in.length; i ++){
for(int b = 0; b < node[0].length; b++){
ini[i][b] = in[i].weight[b];
}
}
return ini;
}
public float[][][] returnNodeNormWeights(){
float[][][] weight = new float[node.length][node[0].length][node[0][0].weight.length];
for(int i = 0; i < node.length - 1; i ++){
for(int b = 0; b < node[i].length; b ++){
for(int a = 0; a < node[i][b].weight.length; a++){
weight[i][b][a] =node[i][b].weight[a];
}
}
}
return weight;
}
public float[][] returnOutNodeWeights(){
int length = node.length - 1;
float[][] nodes = new float[node[length].length][node[length][node[length].length].weight.length];
for(int i = 0; i < node[length].length; i ++){
for(int b = 0; b < node[length][node[length].length].weight.length; b++){
nodes[i][b] = node[length][i].weight[b];
}
}
return nodes;
}

public float[] returnRanWeights(int amount){
Random a = new Random();
float[] weight = new float[amount];
for(int i = 0; i < amount; i ++){
weight[i] = a.nextFloat() + a.nextFloat() - 1;
}
return weight;
}

public void createNetwork(){
for(int i = 0; i < in.length; i ++){
in[i] = new Node(returnRanWeights(node[0].length));
in[i].in = true;
}
for(int i = 0; i < node.length; i ++){
for(int b = 0; b < node[i].length; b ++){

if(i < node.length - 1){
node[i][b] = new Node(returnRanWeights(node[i + 1].length));
}else{
node[i][b] = new Node(returnRanWeights(out.length));
}

}
}
for(int i = 0; i < out.length; i ++){
out[i] = new Node(null);

}
}

public void input(float[] inp){
for(int i = 0; i < in.length; i++){
in[i].value = inp[i];

}

for(int i = 0; i < node.length; i ++){
for(int b = 0; b < node[i].length; b ++){

if(i == 0){
node[i][b].input(in, b);
}else{

node[i][b].input(node[i-1],b);

}


}
}

for(int i = 0; i < out.length; i++){
out[i].input(node[node.length - 1], i);
}

}

public float getOutput(int num){
return out[num].getOutput();
}
public float[] getOutput(){
float[] a = new float[out.length];
for(int i = 0; i < a.length; i++){
a[i] = getOutput(i);
}
return a;
}


}
[/CODE]
0

Share this post


Link to post
Share on other sites
(this is not activation+=1 but activation+=W0)

I think you can add a bias to your nodes without modifying too much your code.
Add an extra component to your input array and put 1.0 in it at the start of the program. Your input vectors are now : [[b]1.0,[/b]i1,i2,....,in].
Then W0, ie the weigth associated to your constant input value 1.0 will evoluate like any other weight.
1

Share this post


Link to post
Share on other sites
[quote name='NicoLaCrevette' timestamp='1332105337' post='4923120']
(this is not activation+=1 but activation+=W0)

I think you can add a bias to your nodes without modifying too much your code.
Add an extra component to your input array and put 1.0 in it at the start of the program. Your input vectors are now : [[b]1.0,[/b]i1,i2,....,in].
Then W0, ie the weigth associated to your constant input value 1.0 will evoluate like any other weight.

Finally adding a bias to a net is (just) equivalent to add a 1.0 extra input to it [img]http://public.gamedev.net//public/style_emoticons/default/smile.png[/img] [size=3][i](more precisely any constant value but [s]nobody cares[/s] that's irrelevant and 1.0 is the usual choice)[/i][/size]
[/quote]

Would the weight of the bias be 1.0 and the input be 1.0 or the weight a random float (0 - 1.0) and the input 1.0?
0

Share this post


Link to post
Share on other sites
An example maybe ? Let's say I want to approximate a 2-parameters (x and y) function with a single node.
So I have to add a third component to the input set to 1.0
Then the node has 2[b]+1[/b] inputs (so 3 weights too) and the input vector is [1.0,x,y]
So the integration is W0*1+W1*x+W2*y

[u]So the code of a node 'without bias' is perfect : just add an extra 1.0 to its input[/u] and it's done, you have a node with a bias.
1

Share this post


Link to post
Share on other sites
[quote name='NicoLaCrevette' timestamp='1332107161' post='4923130']
An example maybe ? Let's say I want to approximate a 2-parameters (x and y) function with a single node.
So I have to add a third component to the input set to 1.0
Then the node has 2[b]+1[/b] inputs (so 3 weights too) and the input vector is [1.0,x,y]
So the integration is W0*1+W1*x+W2*y

[u]So the code of a node 'without bias' is perfect : just add an extra 1.0 to its input[/u] and it's done, you have a node with a bias.
[/quote]

Thanks so much. I think it works now. Not sure though but when you input -10 it comes up with (on average) low outputs. When I put big inputs (100 for example) it usually returns relatively high outputs.
Thanks once again.
0

Share this post


Link to post
Share on other sites
Hello CryoGenesis

I'm glad I can help :)

NB : it's recommended to 'normalize' your input values so that abs(input)
0

Share this post


Link to post
Hello CryoGenesis
I'm glad I can help :)
NB : it's recommended to 'normalize' your input values so that abs(input)<1
Otherwise you will use huge values in the learning rule, and the weights will oscillate indefinitely instead of stabilize.

For example if you know the min and the max of the values you provide to the ANN, you can apply something like that to each input :
i'= (i-min)/(max-min)
and provide i' instead of i.

Bye !
Nico
1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0