|
David Rumelhart, who just died, modeled linguistic phenomena (e.g., morphology and phonology) using neural networks, already back in the 50s. (edited: oops, it was the 80s) Is he a predecessor, or early pioneer, of modern NLP and machine learning? Or are there fundamental differences in how he modeled things, compared to modern approaches that, say, also use perceptron learning, etc.? I'm not trying to elicit hymns on how great he was; I'm just interested in the question if what we do today can be seen as roughly similar to what he and his connectionist colleagues did (rejecting rule-based systems, purely learning from data, etc.) in the 50s, or if the apparent similarities are only superficial. One difference I can see is that no empirical NLP researcher who uses machine learning today would make any (strong) claims about being able to model what happens in the brain, simulate thinking, etc. (Or, am I wrong?) |
|
Certainly the ideas pioneered by Rumelhart resonated widely and formed a basis for much of the work on neural networks in the 1980s and 1990s, which was more or less the beginning of machine learning as we know it. The mid-to-late 90s saw the rise in popularity of kernel methods and what Yann LeCun once called "convexivitis", where neural networks and related methods were largely abandoned by the research community because their loss functions weren't convex, making them difficult to analyze theoretically. However Rumelhart's impact on the discipline of machine learning persisted, and connectionist-style models are undergoing something of a renaissance since 2006 when Geoff Hinton, a coauthor with Rumelhart on the seminal paper Learning representations by backpropagating errors, made the first significant strides toward the problem of training very deep networks that automatically learn multiple levels of representations. This, perhaps, is a key difference between connectionism then-and-now; there's less interest today in hand-engineering models of "how the brain might do it" with specific structure built-in (though there certainly are still researchers who use connectionist models in neuroscientific investigation, Sue Becker and Dave Plaut come to mind). Amongst circles where neurally inspired machine learning is investigated (the so-called "deep learning" movement) there is more interest in developing clever learning algorithms for learning representations of the input automatically, whether or not they help us understand the way a biological brain might work. As for NLP, the vast majority of practical NLP done today is statistical/learned from data, so I would say his influence is felt, though the specifics of his research are less relevant. Lot of the most exciting work in the field today has a connectionist flavour to it, see for example the work of Ronan Collobert and colleagues over the past few years. |
