Archived

This topic is now archived and is closed to further replies.

eNGIMa

Backpropagation Training

Recommended Posts

I''ve recently tried to write a multi layered perceptron network which simply calculates XOR values of two binary digits. After many hours of frustration and research I can''t find what''s wrong with my network. Im using 2-3-4-2 neurons for the layers. The problem appears to me that after a large number of training runs the weights in the last layer become a few orders of magnitude above the previous layers (eg -1000 compared to -10), while the outputs become increasingly smaller each training run gradually decreasing to zero. I read that large weights can be caused by overspecialisation, but i can''t see why only the output layer would be like this. I know this is fairly vague, but any suggestions would be greatly appreciated.

Share this post


Link to post
Share on other sites
First of all, make sure your network can learn simpler functions such as AND and OR. The results you are getting sound unusual, so the chances are that you have an error somewhere in your code.

Here''s a handy hint for debugging your code. Download JavaNNS and create a 2-3-4-2 network with identical bias and weight values. The JavaNNS allows you to step through the backpropagation learning process and observe the changes to the weights and bias values. Just compare these changes with the changes in your own network. Although this can be a tedious exercise, if you have an error in your code you should be able to spot the problem fairly quickly.

You should also be aware that to solve the XOR problem with a neural network you only need a 2-2-1 architecture. Using extra hidden layers will only slow down the process of learning in this situation.

If you still have problems, you could post some of your code for people to take a look at.

Share this post


Link to post
Share on other sites
I remember having a similar problem once.
The weights didnt seem to change and the debug output showed that the weight change was zero or very close to it. I am not 100% sure what caused the problem but i think it could have somthing to do with that my weights were initialized to zero sometimes. Perhaps this can help...



--Spencer

"Relax, this dragon is sleeping..."

Share this post


Link to post
Share on other sites
I _finally_ found the problem. Went through logically step by step and figured it had to be something where the delta for each layer wasn''t passed through properly. Turns out that i was passing the outputs through a sigmoid function twice. Must''ve been real sleepy when i wrote that ... I feel really stupid now.

Thanks you both heaps for your suggestions, was of great help

Share this post


Link to post
Share on other sites