Deriving RNN backprop equations

Question

I'm working on constructing an RNN along the lines of what James Martens describes in his paper Learning Recurrent Neural Networks with Hessian-Free Optimization, and I'm a little confused on how backprop works with RNNs.

When calculating the derivatives, it looks as though some of the errors propagate forward through time.

Here are the forward equations (quoted from the paper):

t_i    = Whx*x_i + Whh*h_(i-1) + bh
h_i    = e(t_i)
s_i    = Wyh h_i + by
yhat_i = g(s_i)

... where h are hidden units, yhat are the output units, and t/s are their net inputs.

When calculating the derivative for Whx, since h_(i-1) makes use of that weight then the derivative of the error at the ith time-step has a dependency on derivatives from a prior time-step.

Basically, my intuition tells me I'm doing this wrong, but the math seems to suggest otherwise. Comments? Suggestions?

edit Adding an equation:

For just two time-steps with one output, where j indexes the input vector and k indexes the hidden-layer, this is the expression I came up with for dEdWhx:

dEdWhx_jk =    (yhat_1 - y1)            // dE/dyhat
            .* g'(s_1)                  // dyhat/ds
            .* Wyh_k                    // ds/dh_{1,k}
            .* e'(t_{1,k})              // dh_{1,k}/dt_{1,k}
            .* [                        // dt_{1,k}/dWhx_{j,k}
                  x_{1,j}               // the part that only depends on external inputs
                + d/dWhx_jk( Whh*h_0 )  // the part depending on a previous iteration
               ]
            + (...)                     // Derivative of the error at the 0th step output

Accepted Answer

I'm not 100% sure what you mean, but part of the derivative does indeed come from previous time steps due to the derivative of the activation function e if that is what you mean. The derivative of tanh is for example 1 - h_i^2. Because h_i depends on previous timesteps, the final gradient also does.

In the end, backprop in an RNN is the same as in a feedforward network after unfolding it over time except for the difference that the weights are tight/shared so you have to sum their individual gradients in the end. When you are using matrix algebra to compute the gradients, this leads to matrix multiplications for the final quantities of interest that automatically include this summation over time. In matrix notation the backprop algorithm should be something like:

dtanh = 1 - h^2
pre_dh = Wyh^T * dy (so this is a matrix now)
dh[:, i-1] = pre_dh[:, i-1] + Whh^T * dh[:, i] * dtanh[:, i-1]
dh[:, end] += pre_dh[:, end] * dtanh[:, end]
dWyh = dy * h.T
dWhh = dh[:, 1:] * h[:, :end]^T
dWhx = dh * x^T

Sorry for mixing matlab and numpy notation a bit...

Deriving RNN backprop equations

Subscription:

Related questions