|
I'm working on a RBM (for a DBN) for image classification. I' working with two RBM layers. The first has Gaussian visible units and binary hidden units and the second has binary visible units and softmax hidden units. It works quite well. I now want to try to use Noise Rectified Linear Unit as the hidden layer, but I fail to understand how to implement them. Everything I've tried has just led to terrible results. Now, if I understand correctly: The activation probability of a ReLU is simply p = max(0, x + N(0, 1)), but then how do I sample the values to activate the visible units ? Should the noise only be used in sampling and not in activation probabilities ? Another thing: in some papers I saw that the noise is to be N(0,1) and some others use N(0,sigmoid(x)). So, what should be the activation function and how the values should be sampled ? |
|
I use max(0, x) for activation max(0, x + N(0, Sigmoid(x)) for sampling There must be something else wrong in my implementation because I have terrible results with these functions.
(Jun 16 '14 at 11:38)
Baptiste Wicht
Apparently, the RBM works, but my Conjugate Gradient fine-tuning does not work with NRLU units :(
(Jun 16 '14 at 14:02)
Baptiste Wicht
|
|
Make sure you sample directly with max(0, x + N(0, Sigmoid(x)) without applying max(0, x) first |
