Role of Sparsity Penalty in training Stacked Autoencoders

1

I am trying to learn hierarchical features on natural images using stacked autoencoders. I am a bit new to this area and have some basic questions:

  1. Will SAEs be able to learn features on very low resolution images? [say 28x28] By features I mean not just the edges learnt in the first layer, but better abstractions over the succeeding deeper layers.

  2. There are a whole lot of algorithms to learn features in an unsupervised way as described in this paper - Denoising AE, different types of RBMs, ConvNets and so on. Now, when targeting a specific application, what would be the criteria based on which a proper model can be chosen? In my case, I am dealing with high resolution natural images, but I do not yet have a GPU to work on. So I am looking for some alternative which can work on workstation class machines (with lot of RAM on CPU!).

  3. While training Stacked AEs, is it a good practise to change the parameters of learning (sparsity penalty, weight decay etc) for each layer of the network? If so, how to determine what value would be ideal for best learning?

I apologise if some of the questions are way too trivial. Thanks in advance for your help.

asked Jan 27 '14 at 20:37

Sharath%20Chandra's gravatar image

Sharath Chandra
311●13●16●21

edited Jan 28 '14 at 22:47

One Answer:
oldest newest most voted
2

  1. It will learn some pretty nice stuff, thats what people achieved on the CIFAR datasets (basically small images) http://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf They are just harder to visualize and make sense once you deeper on your network.

  2. There is no easy answer here. So, welcome to Neural Networks World. Have your seat and be prepared to play with hyper parameters (cost functions, learning rates, number of layer, number of neurons on each layer). Just to beg your patience. Yan LeCunn said on G+ that the wienner of ImageNet 2013 was playing with his deep net for a year before coming up with his final results. But talking about the proper framework to work with, you can try Theano which compiles your code to C/C++ (or CUDA when you get your GPU) and that should make whatever you have to try faster (maybe not that much on CPUs, depending on your data).

  3. Once again welcome to neural nets. You should always expect good results from well advised regularizations, but remember that your time is important. Make it simple first and fine tune your model later if you have to.

You shouldn't apologies for asking and learning!!!

answered Jan 27 '14 at 22:51

eder's gravatar image

eder
216●2●5●11

edited Jan 27 '14 at 22:52

Thanks for your answer.

I am now working with Convolutional RBM. I am curious if we can learn features specific to a task at hand. Will fine tuning make a very big difference, to put it bluntly.

The thing is I am trying to learn features in an unsupervised manner and then use them for different domains. So will this work?

Also, can you give me some input about what these parameters in CRBM mean: learning rate for sparsity udpate target sparsity

(Feb 17 '14 at 22:20) Sharath Chandra
Your answer
toggle preview

learn more | privacy | contact

powered by OSQA

User submitted content is under Creative Commons: Attribution - Share Alike; Other things copyright (C) 2010, MetaOptimize LLC.