• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
ROBERTREAD1

Neural networks: hidden Layer Bias

10 posts in this topic

How important is it to have a bias on your hidden layers? Can you have the same effect by just having a bias on the input layer? Is it worth having one (or more) fixed value(s) in your training (and output) patterns? i.e. for XOR input{A,B,fixed}output{result,fixed} {0,0,1}{0,1} {0,1,1}{1,1} {1,0,1}{1,1} {1,1,1}{0,1} Does anyone have an simple C(++) source I could try these out with multilayer EBP? Thanks, Rob.
0

Share this post


Link to post
Share on other sites
You need bias going into every hidden node and every output node.

There is no bias going into the input nodes.

Bias is always 1.0

Have the learning algorithm learn the weights for the connecting edges normally.

You will need a sigmoid and sigmoidPrime for the bias activation function.

Basically, a bias neuron helps seperate cases and allows you to have smaller neural nets than you would without bias.

A little fuzzy on the exact reasons as its been a while since I went through the theory but I just coded a NN at work and it works well.

Also, xor should look like this:

A - B - Output
0 - 0 - 0
0 - 1 - 1
1 - 0 - 1
1 - 1 - 0

The bias shouldn't even show up in the training data. It should just be handled automatically without you even having to see it.

Make you neural net data driven. 2 files, 1 config and 1 data.

That way you can much about with the internal structure of the nn by modding the config file rather than having to mod the code.
0

Share this post


Link to post
Share on other sites
Thanks, but can I get away with JUST biasing the input pattern? (to simplify implementation).

I last did NNs ages ago, and I remember mucking around with my own C NN "engine" which is lost on a very old disk (I recompiled it for GCC to use 32bit memory addressing, that's how old!).

I'm planning on writing a NN to drive round a race track, and I'm going to make the program use a technique I thought of to create it's own learning patterns on the fly.

The NN Inputs will consist of:
Direction of travel relative to heading.
Steering Wheel relative to heading.
De/Acceleration Force.
Accelerator pedal.
64 radial Time-To-Impact with left side of track Inputs surrounding the driver.
64 radial Time-To-Impact with right side of track Inputs surrounding the driver.
(1 implies Impact next frame; -1 no chance of impact)

The outputs are Steering Wheel, Accelerator.
0

Share this post


Link to post
Share on other sites
I don't have any mathematical proof, but intuitively, I would suspect that with any bias on the hidden layers, you will be limiting the ability of your net. The bias allows your fit curve to shift and have an arbitrary intercept. If you think about it in one dimension, consider an X-axis with a sigmoidal curve going through the origin. Without the bias, you're stuck with the curve going through the origin. If you add in a bias term, you can now move that interception point on the X-axis around. This would translate to the hidden layers as well. Without a bias, they can only have one possible point of intercept without shifting, which may limit the ability to model a given curve.

I should point out that an earlier post said that the bias was necessary to address linearly inseperable problems, but that is not correct. Linear inseperability is addressed by having hidden layers. Without hidden layers, you essentially have a linear function approximator - the perceptron. When you add in a hidden layer, you start allowing for nonlinear transformations of the input variables.

I hope that helps.

-Kirk

0

Share this post


Link to post
Share on other sites
Quote:
Original post by ROBERTREAD1
Thanks, but can I get away with JUST biasing the input pattern? (to simplify implementation).

I think the answer is yes. If the network really needs to have a bias further down the line, it can always use one of the neurons to just produce a constant, which can then be used as bias in the next layer.

0

Share this post


Link to post
Share on other sites
Quote:
Original post by kirkd
I don't have any mathematical proof, but intuitively, I would suspect that with any bias on the hidden layers, you will be limiting the ability of your net. The bias allows your fit curve to shift and have an arbitrary intercept. If you think about it in one dimension, consider an X-axis with a sigmoidal curve going through the origin. Without the bias, you're stuck with the curve going through the origin. If you add in a bias term, you can now move that interception point on the X-axis around. This would translate to the hidden layers as well. Without a bias, they can only have one possible point of intercept without shifting, which may limit the ability to model a given curve.

I should point out that an earlier post said that the bias was necessary to address linearly inseperable problems, but that is not correct. Linear inseperability is addressed by having hidden layers. Without hidden layers, you essentially have a linear function approximator - the perceptron. When you add in a hidden layer, you start allowing for nonlinear transformations of the input variables.

I hope that helps.

-Kirk



Ah so this is an actual way of doing a feed forward network?

Do you know if there are any articles about this that i can read up on?

THanks
0

Share this post


Link to post
Share on other sites
Yes, the bias nodes are part of a feed forward network. I assumed we were talking about feed forward networks from the context.

As for articles, I don't have any specific recommendations. A few books of interest include Neural Networks for Pattern Recognition by Bishop, Neural and Adaptive systems by Principe, The Elements of Statistical Learning by Hastie, and Pattern Classification by Duda. The only one of those that includes a disucssion of bias units is Bishop, and there is no discussion of their necessity in the hidden layer.

There are tons of tutorials on the web. Try Google-ing for "neural net tutorial" and you'll get a whole weekend's worth of reading material. I search for quite a while, but didn't find anything related to sacrificing the bias on the hidden layers.

-Kirk

0

Share this post


Link to post
Share on other sites
This is from a classifier perspective. Bias is needed when you need separation boundaries that don't all intersect at the origin.

Example:

weight1*x + weight2*y + weight3*z + .. + (0 bias term) = 0 defines a hyperplane going through the origin.

weight1*x + weight2*y + weight3*z + .. + weightbias*(bias=1) = 0 defines a hyperplane that does not intersect the origin.

Think of these hyperplanes, lines, etc. as decision boundaries on what output a neural network should have.

Edit: err what kirkd said ... :)
0

Share this post


Link to post
Share on other sites
Nick,

What are your thoughts on the need for bias nodes in hidden layers? Intuitively, I'm certain that they should be there for the exact same reason we want them in the input layer, but I can't come up with any supporting evidence. Everywhere I've looked and found discussions on bias nodes, they are included in all the hidden layers, but I've never seen a discussion of what would happen if they weren't there. Again, I can intuitively see that they will provide the same flexibility as they do in the input nodes - move the hyperplane away from the origin. Is it possible, however, that without a bias node in the hidden layer, the transform that occurs in the hidden layer will merely need to adjust itslef through the input-hidden layer weights?? Does a bias node in the hidden layer merely allow us to find "a" solution rather than "a specifc" solution???

Hmmmm.....


-Kirk


0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0