Let's talk about ICML 2011 and noteworthy papers

Bayesian Learning via Stochastic Gradient Langevin Dynamics

Abstract: In this paper we propose a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize. This seamless transition between optimization and Bayesian posterior sampling provides an in-built protection against overfitting. We also propose a practical method for Monte Carlo estimates of posterior statistics which monitors a ``sampling threshold'' and collects samples after it has been surpassed. We apply the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients.

(I don't try to summarize the paper, the paper as a whole is well written and astonishing)

There is of course the usage of Martens' hessian free optimizer on recurrent neural networks to generate text.

"Generating Text with Recurrent Neural Networks" and "Learning Recurrent Neural Networks with Hessian-Free Optimization". They beat the longstanding state of the art for learning long term dependencies in time series, LSTM. While the achievement is big, I wonder how HF+LSTM would work.

Anyway, here is what that RNN of them produced, character by character.

The meaning of life is the tradition of the ancient human reproduction: it is less favorable to the good boy for when to remove her bigger. In the show’s agreement unanimously resurfaced. The wild pasteured with consistent street forests were incorporated by the 15th century BE. In 1996 the primary rapford undergoes an effort that the reserve conditioning, written into Jewish cities, sleepers to incorporate the .St Eurasia that activates the population. Mar??a Nationale, Kelli, Zedlat-Dukastoe, Florendon, Ptu’s thought is. To adapt in most parts of North America, the dynamic fairy Dan please believes, the free speech are much related to the

"On RandomWeights and Unsupervised Feature Learning"

I wanted to highlight this paper again because I noticed that it makes this comment about pre-training convolutional architectures:

"However, we find that the performance improvement can be modest and sometimes smaller than the performance differences due to architectural parameters."

This result is both surprising as there is a growing body of work these days about pre-training neural networks. Isn't LeCun moving in this direction of applying pretraining to his CNNs? (Hinton mentioned it in one of his video lectures)

Active Learning from Crowds

Abstract: Obtaining labels is expensive or time-consuming, but unlabeled data is often abundant and easy to obtain. Many learning task can profit from intelligently choosing unlabeled instances to be labeled by an oracle also known as active learning, instead of simply labeling all the data or randomly selecting data to be labeled. Supervised learning traditionally relies on an oracle playing the role of a teacher. In the multiple annotator paradigm, an oracle, who knows the ground truth, no longer exists; instead, multiple labelers, with varying expertise, are available for querying. This paradigm posits new challenges to the active learning scenario. We can ask which data sample should be labeled next and which annotator should we query to benefit our learning model the most. In this paper, we develop a probabilistic model for learning from multiple annotators that can also learn the annotator expertise even when their expertise may not be consistently accurate (or inaccurate) across the task domain. In addition, we provide an optimization formulation that allows us to simultaneously learn the most uncertain sample and the annotator/s to query the labels from for active learning. Our active learning approach combines both intelligently selecting samples to label and learning from expertise among multiple labelers to improve learning performance.

I feel like this is a problem many of us tackle often while trying to do what we really care about. Having somebody take the time to solve this in a reasonable fashion is very nice.