Rotation and scale invariance RBM

Thanks Andreas! Yeah third order RBMs are cooler :). But I havent found any code to do discriminative classification with them. Im not feeling confident that I will be able to implement them quickly just to try them out. Are you aware of any code on the net?

I don't know of any code for third order RBMs. I know there has been a paper about classification with third order on nips last year (or maybe the one before?) but that is not what you want I think. There has been something about occlusion modeling with third order which I think does finetuning, not sure though. Do you know any papers using third order "at the bottom" and doing finetuning? My lab has Python/CUDA code which aims to be very modular (and fast). It doesn't do 3rd order but I think it's not to hard to implement it. If you want, you can have a look here: https://github.com/deeplearningais/CUV or at my blog (with feature list): http://peekaboo-vision.blogspot.com/2010/11/restricted-boltzmann-machine-on-cuda.html If you want to do RBMs, even on MNIST, I recommend you get a GeForce ;)

I believe Andreas is thinking of the "Gated Softmax Classifier" which has code (and errata for the paper) at http://www.cs.toronto.edu/~rfm/gatedsoftmax/index.html Be warned, from what I recall when I re-implemented it myself the code was a bit confusing in some places.

Marc'Aurelio Ranzato and his coauthors have used certain types of 3-way models as generative models of images and as first layers for all sorts of deep models with a good deal of success. There have also been many other related models to deal with some of the difficulties of the mean-covariance RBM. Roland Memisevic's work on image transformations uses conditional 3-way models which leads to very different challenges than Ranzato et al.'s work and has different applications. V. Mnih et al. describe some of the difficulties training conditional RBMs can pose and some possible solutions in http://www.cs.toronto.edu/~vmnih/docs/uai_crbms.pdf That being said, I don't think it is obvious how to use 3-way RBMs to simply make a model deal better with rotated and scaled input.

Do you know of any conditional 3-way models - like the one I mentioned in my answer - that are used for classification?

Take a look at:

3-d object recognition with deep belief nets - Nair, V. and Hinton, G.

An Efficient Learning Procedure for Deep Boltzmann Machines. - Ruslan Salakhutdinov

I dont think they are trained discriminatively, but here they do seem to learn certain invariance using third order RBMs.

Hm I somewhat forgot about that paper. The measure introduced there is quite interesting. Though I would rather call the features "slow" than scale and rotation invariant. I wonder how sift with Harris Laplace would do in this test...

Well I want to use RBMS to sidestep feature selection. What maybe does make sense is to have a whole image representation that is invariant to rotation. A polar representation would help in converting rotations into translations. Do you know if RBMS can be easily made invariant to translations?

@Roderick - The only architecture I've seen that offers any amount of translation invariance is convolutional, but it only nets you a small amount of actual translation invariance. Take a look at Yann LeCunn's "LeNet". It uses many convolutional layers stacked on-top of each-other -- the net effect is it becomes translation invariant.

I agree that expanding the training set is the most promising approach when you are trying to do classification. However, I am not sure whether it helps to generate invariant features when doing unsupervised training. Do you know about any insights on that? Transforming autoencoders seem like one possiblity to learn invariant features. But I think work on these is still in a very early stage. Do you know of any paper that uses features from transforming auto-encoders to do classification (or use the features in any other way)?

@Andreas - I agree with your concerns. On the one hand, other research (eg, Yann LeCunn's LeNet) clearly demonstrates that it's possible to build something capable of dealing with rotational and (at least to a degree) scale invariance. But, what is really being learned here and how well does it generalize? It seems to me it's still not going to truly generalize the concepts. Supposing for example you trained it on all possible 3s at the same scale/orientation, then trained it on many scales/rotations for many of the 3s, my guess is it would have trouble with the ones that were left out. At least in terms of the human visual system, the brain has to be doing something other than just imagining every possible permutation of something it sees in order to 'learn' that concept. My daughter, for example, is pretty good at reading but I don't regularly try to get her to read upside down. Nevertheless, she can do it albeit slowly because she's not used to it. By contrast, if all rotations near 180 degrees were left out of the training set a DBN would probably have trouble with 180 degree rotated images even though it was exposed to other orientations.

I started working in this direction. I am using RBMLIB to train a discriminative RBM or a Deep Belief Network. However, I have troubles setting the parameters. Is there a rule of thumb to set the amount of hidden units and layers? I have the impression my network is overfitting to the training data.

Why do you think you are overfitting? Did you compare likelihood ratios? I haven't really paid much attention to it but I have never actually heard of a case of overfitting in RBMs.

@Roderick - Check out Hinton's paper 'A Practical Guide To Training Restricted Boltzmann Machines': http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf . In addition, re: your question, in a very real sense the hidden units will adhere to basic principles of information storage. The system will try to come up with weights that will reproduce everything in the training set with high probability; the more bits you give it, the better it can do (not necessarily will do), but past some point you'll see diminishing returns unless you use a sparse regularizer. Furthermore, are you adding noise during each gibbs step? If not, that may be the problem.