Text classifiers using accidental features

This is an interesting question which surfaces daily in all sorts of organizations. On the one hand, we don't want to trust the computer "too much", as mechanical analysis can, as you note, latch onto things which are completely nonsensical. Hence, many organizations review the output of such analytics, from the point of view of "sanity checks". On the other hand, the whole point behind using computer-based analysis is that humans, too, are fallible, and likewise draw mistaken connections.

There is an old story about a large computer system which would fail regularly on a particular day of the week. None of its human handlers could think of a reason for this pattern. There were not especially high loads on the computer that day of the week, and other obvious causes were ruled out. As it turned out, the reason for these periodic failures was that the air intake for the computer's cooling apparatus was jammed by grass clippings kicked up during the weekly mowing of the lawn. I have wondered whether a data mining system, supplied with all the workings of this business, including the computer failure reports and the lawn care schedule might not have found this correlation? Perhaps more importantly, would anyone have believed it, if it had?

I don't know offhand about any works talking about this particular example, but it seems that the issue is that the training domain is not representative of the test data.

One technique for alleviating this would be to gather a large, representative, unlabelled corpus, and do semi-supervised learning. A hyperparameter could control the amount by which the unlabelled data is used to regularize the model.