Penalizing non-orthogonal weights in an autoencoder

I'm wondering if someone could point me in the right direction, here. I've been working on a stacked denoising autoencoder for the purpose of generic object recognition for a while, now, and I keep butting my head up against the same problem. Essentially, I'm finding that the weights for my neurons, despite being initialized randomly, tend to converge to very similar results during pretraining. My cost function is the half mean squared error, plus a weight decay term.

My first response when I saw this was to introduce another penalty determined by the degree of non-orthogonality. To that end, I used the formula developed in http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.45.3455 (download the cached version). The problem I experienced with this is that random noise is orthogonal, so adding this term tended to reinforce the noise used to initialize the weights. What other approaches could I use to solve this? I've been testing using caltech 256, which admittedly isn't a very good dataset, but there's next to no variation whatsoever in my pretrained weights.

Edit: If this non-orthogonal cost function still has merit, are there any more recommended papers?

Possibly related: http://metaoptimize.com/qa/questions/13401/deeplearningtutorials-using-the-stacked-autoencoder-results-in-almost-identical-semantic-hashes