|
I'm currently working on Convolutional RBM and I fail to train it so that it learns sparse features. With "normal" RBM, I was using the technique of Nair and Hinton with a sparsity target and a penalty computed for each hidden unit and applied to each its input weight. It was working quite well. However, with CRBM, it does not work the same way because an element of the weight matrix cannot be linked to an hidden unit since all the weights are shared. Honglak Lee uses its own method from (Lee et al., 2008) to train via sparsity regularization. But I fail to see how to compute the "gradient of the regularization term". Another technique is the one from (Goh et al., 2010), but I don't see how to adapt it to CRBM. So, how does one train a sparse CRBM ? |
|
I finally found the answer in the Matlab code. We can apply a penalty on the hidden biases: pbias_lambda is the sparsity learning rate and pbias is the target sparsity. |
