|
I have been working with a Convolutional Deep Belief Network. I find that if i try to increase the number of bases in the first layer (bases:number of convolution kernels to be used at a layer) to say a 100, I start getting infinities when sampling from the system. By sampling I mean that I take some initialization of the (convolution responses in) the hidden layer, and calculate the reconstruction in the visible layer. It is noticed that while training the norm of weights of the network connections, ||w||, starts growing exponentially, from say order 0.1 --> 1-->10-->100... This seems to cause samples from the system to go to infinity. Is there some suggested technique for accomplishing this, or are there limitations on the number of bases? I am already using weight decay, so should I be increasing the rate of decay? Is this an issue endemic to all convolutional neural networks? |
|
You can normalize the weights after each training step. Or increase the weight decay. Your problems sounds much like the exploding gradient problem. Maybe try to use dropout. Or a smaller learning rate. Or a bigger batch size. Hi, thank you for your answer. My issue, embarrassingly enough, was due to a coding error. I modified someone else's code incorrectly.
(Jan 28 '14 at 10:05)
AS1
|
