Anyone recommend a good, easy to use, hyperparameter-free adaptive learning rate for SGD?

I am using AdaGrad and AdaDelta in my current project for training deep neural network. AdaGrad is very sensitive to its initial learning rate. If initial learning rate is inappropriate, AdaGrad has very poor performance. Moreover, the learning rate of AdaGrad always decay over time, which is not good for non-stationary task. In particular, I think different layers of a deep neural network should have different learning rate schedules, and the learning rate of one parameter should be adaptive w.r.t. the current state of the neural network.

The performance of AdaDelta is often comparable or even better than the best turned AdaGrad, and it's performance is insensitive to hyperparameters. I don't need to tune it's hyperparameters for different learning tasks and different datasets. However, the learning rate of AdaDelta never decay over time, so it never converges to any local optimum. In my project (a classification learning task), an important prediction quality measure is the difference between the empirical positive rate and predicted positive rate (averaged over the entire training data or test data). If the model converges, the two rates should be the same (at least on the training data). But I found that for AdaDelta trained models the two rates differ a lot, which is problematic. AdaDelta has a window size hyperparameter, and it computes the approximate expected squared gradient and expected squared parameter update (Delta x) over the data window. I don't know whether increasing the window size helps converging.

I have also considered the algorithms presented in the paper "No More Pesky Learning Rates" and "Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients". However, these algorithms require either computing the diagonal hessian or doing two steps of feed-forwarding/back-propagation for each mini-batch, so they don't fit into my current learning framework. Moreover, since I'm using relu activation function, the loss function is non-smooth, so the algorithm presented in "No More Pesky Learning Rates", which requires diagonal hessian, does not work. On the other hand, the algorithm presented in "Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients" has been experimented only on extremely simple one-dimensional learning tasks, and is not compared with other adaptive learning rate algorithm.

Anyone recommend a better adaptive learning rate algorithm? Some properties are desired:

1. The performance is insensitive to hyperparameter. So we don't need to tune the hyperparameter too frequently.

2. Adaptive. Each parameter should have its own learning rate.

3. Guarantee of convergence for stationary task.

4. Suitable for mini-batch SGD (the size of mini-batch may be large).

5. Only need first-order gradient estimated on each mini-batch. For each mini-batch and each parameter, only one gradient is needed (so only one feed-forwarding/back-propagation is performed for each mini-batch).

6. Performance comparable to AdaGrad and AdaDelta.
   

Any ideas? Thanks.