I'm looking at the best platform to build a web application that would have a strong natural language search (question answering) component.

As the rest of the application isn't all that complicated and most web frameworks can handle that part, I'm thinking that having a mature, well supported NLP/NLS libraries would be a key point of interest. At this point I'm just collecting information from people who have actually used the libraries.

So, if you were looking to do natural language search, what libraries would you reach for? Why? Can you give pro and con for your choice(s)? I know this isn't a question that has a right or wrong answer... Just a quest for more information/background so I can narrow down my research some.

asked Aug 04 '10 at 19:05

Sean%20T%20Allen's gravatar image

Sean T Allen

edited Aug 09 '10 at 12:42

Joseph%20Turian's gravatar image

Joseph Turian ♦♦

What do you mean by natural language search? A full-text search engine (like google)? A question-and-answer engine? Something like powerset?

(Aug 04 '10 at 19:40) Alexandre Passos ♦

yes along the lines of powerset. additionally there would probably be something to collate similar content together ( find more like this, similar to this, etc )

(Aug 04 '10 at 21:38) Sean T Allen

2 Answers:

I would look at the most recent work by Hoifung Poon, which includes code:

  • Poon and Domingos (2010). "Unsupervised Ontological Induction from Text." [paper]
  • Poon and Domingos (2009). "Unsupervised Semantic Parsing." [paper | slides | code]

They have a completely unsupervised technique for semantic parsing, and they test it on question-answering, getting twice the recall of competing approaches. The Java code for the 2009 approach (USP) is available at the link above. I am asking Hoifung about the 2010 (LOFT) code.

When I read the 2009 work, it completely blew my mind. You read this paper and you're like: "Whoa, you really just did that. You attempted to actually solve NLP. I forgot that we're allowed (supposed) to do that." And the technique is so elegant, small wonder it got best paper award.

answered Aug 09 '10 at 12:53

Joseph%20Turian's gravatar image

Joseph Turian ♦♦

edited Aug 26 '10 at 16:16

Solr, with some plugins should do the job.

answered Sep 24 '10 at 22:28

DirectedGraph's gravatar image


Your answer
toggle preview

powered by OSQA

User submitted content is under Creative Commons: Attribution - Share Alike; Other things copyright (C) 2010, MetaOptimize LLC.