|
Does anyone know of sources for negative results in machine learning? With this I mean articles or reports that report on the failures of ideas and methods that should have worked. Most journals focus of course on things that DO work but it would be great if there was an exception to this rule. Knowing more about this would probably be a very good way to prevent people from trying things that have been tried already and failed. It would probably also help to be better able to scrutinize one's own ideas and filter out the bad ones. Of course some of this is documented in textbooks but this happens almost exclusively when some other method is introduced that addresses the limitations of another. On google I found two places that were intended for gathering this type of information but both seemed to be sort of dead. |
|
the forthcoming issue of sigkdd explorations is on negative results: http://www.sigkdd.org/explorations/upcomingissue.php This was exactly what I had in mind. Thanks for the link.
(Dec 16 '10 at 11:23)
Philemon Brakel
|
|
I agree that this is lacking, but it's a hard thing to manage. If negative results count in any way towards advancing one's career then there's an incentive for people to just try stuff and give up perhaps too soon, before any deeper insight that would solve the problem is obtained. On the other hand, there's nothing stopping people from publishing their negative results on the arXiv apart from the fact that no one wants to waste time documenting carefully and redoing experiments of something that is clearly a bad idea. I think a good sweet spot is for people to put in a final paper at least a reference to other techniques that didn't work, mostly to warn readers trying to extend the work. Another solution is to use places like mlcomp to run experiments more often, and then it's just a matter of making negative results public. I agree that it makes more sense to motivate people to publish good results. The reason I started thinking about this was that I stumbled upon the page of the workshop on negative results from NIPS 2002
(Dec 15 '10 at 13:31)
Philemon Brakel
I think maybe an informal forum, like this one, can be good for negative results, as other people can validate them and it gets easy for a newcomer to pick experimental/modeling mistakes that might be to blame for the negativity.
(Dec 15 '10 at 13:32)
Alexandre Passos ♦
I like the idea. It would probably work well as long as it didn't end up like so many support forums where someone says "Oh, I fixed it" and stops participating in the discussion. -Brian
(Dec 15 '10 at 14:55)
Brian Vandenberg
|
|
You might see some in the relevant categories in PLoS ONE. They publish everything that's fit to print, reviewing only for technical accuracy and not "notability". Thanks! I actually didn't know about that site and it looks very interesting.
(Dec 15 '10 at 13:28)
Philemon Brakel
|
|
The trouble with this line of thinking is the tendency to stop looking in a direction because 'someone else' failed. What [I think] you're referring to is not so much giving up on "the big idea", but giving up on a particular approach to solving the problem at hand, but I believe the same applies. I might have failed to derive a solution to PDEs using an infinite series of functions (Fourier series), but that doesn't mean it's not possible. -Brian
This answer is marked "community wiki".
I agree with your example, but for the more empirical cases I think it might be fruitful to at least know that something is very hard. For example that the convergence of certain training algorithms doesn't scale up to high dimensional spaces or that they have bad generalization properties. Of course you still need to be sure that the people who did the research actually did their best...
(Dec 15 '10 at 13:36)
Philemon Brakel
@Philemon: I think for the precise scenario we can trust either of these two things: (1) that authors will be clear about the limitations of their methods and (2) that someone else should probably reproduce any significant result and report that method X is worse than some baseline if this is the case.
(Dec 15 '10 at 13:38)
Alexandre Passos ♦
|
|
There just happens to be a journal titled "journal of interesting negative results in natural language processing and machine learning." the website is http://jinr.site.uottawa.ca/ . i remember hearing about it awhile ago, but had to look it up because i couldn't remember if i had heard it in a dream... yeah, just looked closely and it seems dead...
(Dec 15 '10 at 18:03)
Joseph Austerweil
|
Good negative results are published in JMLR, ICML, etc. NIPS 2010 had a paper about failure of resistance distance for similarity-based learning. There are so few such papers because most negative results aren't that interesting. For instance, one researcher told me about his failure to get Generalized BP to work well for anything except grids. At that point, the title of paper would've been "Generalized BP performs poorly, possibly due to bugs"
Those are nice example of useful and useless negative results. I guess the more interesting results are indeed published in the conventional journals and conferences anyway...