I am wondering if the correct term for Stop words is Function words. It seems function words is the proper term and has been used by linguists. what do you think?

asked Jan 18 '11 at 04:12

Mark%20Alen's gravatar image

Mark Alen
1323234146


One Answer:

While most of the time all words that are considered Stop words are indeed Function words, I think it can still be useful to discriminate between the actual linguistic term and the term used for words you are not interested in for some task. Some less frequent function words are perhaps still interesting. 'versus' or 'aboard' for example might not be in a Stop word list because they are still highly correlated with certain semantic content.

answered Jan 18 '11 at 05:01

Philemon%20Brakel's gravatar image

Philemon Brakel
244593560

1

I couldn't agree more. The term "function words" has a specific linguistic meaning -- they are words that carry little or no referential value, but contribute to the syntactic interpretation of language. "Stop words" are just things you'd like to ignore for a particular task (say, low tf*idf terms). Usually these two groups have a lot of overlap, but I don't think we should forget the distinction.

(Jan 21 '11 at 07:28) Andrew Rosenberg

Also, sometimes function words are not stopwords. In the Nigam, Mccallum, Thrun, and Mitchell paper that introduced EM for semi-supervised naive bayes ( http://www.cs.uu.nl/docs/vakken/ll/Labeled_unlabeled.pdf ), they note that the word "my" (which is in many stoplists) is actually a very good feature for one of the tasks (student homepage classification).

(Jan 21 '11 at 08:30) Alexandre Passos ♦
Your answer
toggle preview

powered by OSQA

User submitted content is under Creative Commons: Attribution - Share Alike; Other things copyright (C) 2010, MetaOptimize LLC.