Sign in to follow this  

Training Error vs Testing Error

This topic is 4224 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi all, I have trained three different kinds of networks for a Time series problem, Real Time Recurrent Learning (RTRL), 1 hidden layer Feedforward Network, 1 Hidden layer Elman Network. Upon training these nets I find that after the training process, the MSE comes out to be in the range of 1e-5~1e-4. But no matter how much I train, and irrespective of the order of presentation of the inputs (it is randomized), the error on the test data is kind of stagnant at abt 1e-3. Sometimes when I train my RTRL longer, the error on the test data actually increases (as vs lesser training). Is this because of the weights getting stuck at some local minima? I may also add that the data I am using (currency exchange rate data) doesnt really seem to have any underlying pattern (this is what I gathered from a plot of the data). Also, the data has *not* been preprocessed. My particular investigation is really abt Neural net structure, and preprocessing the data might bias my findings. So I havent done anything abt it. What do you recommend? thanks Sidhant

Share this post


Link to post
Share on other sites
If you're noticing performance degrade as the length of training increases, then that suggests your network is over-fitting the training data.

As for the RMSE of your test data, you should expect it to be worse than your training data as a function of the cumulative distance between each training set datum and test set datum. That is, you don't expect the network to represent the data perfectly (particularly in your domain where there is a significant element of noise in the data), so you expect that it will not perform as well on data it has not seen before.

Since you're dealing with time series data, you might consider a hybrid approach whereby you pre-filter the data and the filter parameters are chosen so as to minimise the RMS of the test set.

Cheers,

Timkin

Share this post


Link to post
Share on other sites

This topic is 4224 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this