|
Hi, I just started to learn deep learning recently by starting with Prof. Bengio's paper "Learning Deep Architectures for AI". In this paper, there is one paragraph about auto-encoder that puzzles me. I've search for quite some places on the Internet, but found no luck. Hope I can find some help here. Below is quoted from Section 4.6 in the paper:
Assume that have the input data
Can someone explain the above quoted paragraph? Thanks! |
first encoded as 
in the hidden layer and the recovered 
:
|
A continuous autoencoder typically does not have a sigmoid in the reconstruction step (otherwise it could not reconstruct inputs outside of the [0,1] interval). So it should be:
If the weights in W1 are very small, then:
because the sigmoid function is approximately linear when the input is close to zero (and sigmoid(0) = 0.5, hence the additional offset). Then:
We can get rid of the constant term by setting b_1 = -0.5 and b_2 = 0. Then we can just choose W_1 and W_2 so the product of their transposes is the identity matrix:
where epsilon is a small constant. Now we get Of course this is undesirable because you want the hidden units to capture useful structure in the data, rather than copy them. Note that the same reasoning applies to an autoencoder with binary inputs (in that case there is a sigmoid in the reconstruction step as well), you just need even bigger weights W_2 to drive the sigmoid into its saturation regions. Now I got it. Thanks for your nice explanation, Sander!
(Apr 03 '14 at 11:28)
mintaka
1
There's actually a small mistake: we can't set b_1 = -0.5 because then the sigmoid is no longer in its linear region. It needs to be b_1 = 0, and then we can set b_2 = - 0.5 W_2^T e (where e is a vector of all ones) to compensate for it. Apologies for the confusion.
(Apr 04 '14 at 04:38)
Sander Dieleman
|
. The smaller epsilon, the better the approximation.
|
I have another question, could you help me? Thanks in advance. As for autoencoder, why the function of intermediate layer is not sigmoid ? |
