Summary

We release code written by Ali Afshar, which turns the KEA keyphrase extractor into an XML-RPC service. This allows you to use KEA as a service, calling it from a variety of different programming languages. The code is released under the New BSD License.

Background

Keyphrase extraction (AKA terminology mining, term extraction, term recognition, or glossary extraction) is the process of extracting multi-word phrases that summarize the meaning of a text passage.

For example, in this document entitled “The Growing Global Obesity Problem: Some Policy Options to Address It”, the keyphrases might be: [“developing countries”, “food consumption”, “overweight”, “taxes”, “prices”, “price policies”, “fiscal policies”, “feeding habits”, “nutritional requirements”, “diet”, “nutrition policies”, and “food intake”.]

These keyphrases are useful for summarizing the topic of the text. Also, these keyphrases are useful in later NLP processing steps, and sometimes more informative and disambiguating than just the individual word tokens in the text.

KEA is a great keyphrase extraction implementation. It is useful because it is open-source, backed by solid research, comes with some annotated training data, and because it can extract keyphrases over unrestricted text, without needing a vocabulary of possible keyphrases.

Other implementations of keyphrase extraction include:

• Maui, a topic extractor from the same people that wrote KEA.
• topia.termextract is a Python term extractor, which is relatively noisy, and proposes many bogus keywords, but it simple to use. This is my recommendation for quick-and-dirty but works immediately out-of-the-box.

API implementations include:

• Termine by NacTem, a permissive term extractor I’ve used in the past. They will give you bulk access for research purposes. It is a UK webservice that also is relatively noisy, and proposes many bogus keywords. However, it appears to me to be slightly more accurate than topia.termextract. YMMV.
• Alchemy’s term extractor.
• The Yahoo term extraction API, which is now only available through YQL. It is low recall but high precision. In other words, it gives you a small number of high quality terms, but misses many of the terms in your documents.
• Five Filters, a web service version of topia’s term extractor (see above).
• Maui on Appspot.

Peter Turney has done a lot of research on keyphrase extraction, and licenses his implementation.

There is a wide academic literature on term extraction, which I won’t summarize here. The best introductory techniques are written by Park, who is now at IBM:
“Automatic glossary extraction: beyond terminology identification” and
“Glossary extraction and utilization in the information search and delivery system for IBM technical support”. You can read more about how to roll your own termex implementation here.

More information about the topic is available on the Maui blog.

Code

When running KEA, instead of a standalone program which reads input from disk, for speed one might want a resident service that keeps the model in memory. Additionally, one might want to call this service from non-Java languages. XML-RPC is a widely supported standard for implementing remote services.

We hereby release KEA service written by Ali Afshar, which turns the KEA keyphrase extractor into an XML-RPC service. This allows you to use KEA as a service, calling it from a variety of different programming languages. The code is released under the New BSD License.

Also included in the documentation is a description of how to this Java program was converted into a XML-RPC service.

Summary:

If you have text data (like a web scrape) stored in a MySQL database, and you want to share the data, mysqldump to XML using the –xml flag.

When fields are unlikely to contain tabs, an even simpler format is a tab-separated file, created using the –tab=path flag to mysqldump. path must be owned by the MySQL database user.

The Problem with the standard MySQL dump format

The standard MySQL dump looks as follows

INSERT INTO `sources` VALUES (1,'2009-03-07 22:06:36','"You\'ve got to be kidding me"', ...

The problem is that the standard dump format is difficult to interact with programmatically.

It is difficult to parse using regular expressions because you cannot merely search for single quotes. You have to search for single quotes that are not preceded by a backslash (unless, perhaps, that backslash is preceded by a backslash).

Also, there are no libraries for reading the standard dump format, nor scripts for converting it into a standard format like JSON or XML. I asked the oracle as well as stackoverflow.

So if you receive a MySQL dump in the standard format, you might have to install MySQL and import the dump to get at your data.

The tabbed MySQL dump format

You can create a directory with one file per table, and the table will be one-row-per-line, with tab-separated values:

mysqldump --tab=path database

Here is some example output:

1	2009-03-07 22:06:36	"You've got to be kidding me"

If you get an error of the following form when you issue the mysqldump command:

mysqldump: Got error: 1: Can't create/write to file 'path/database.txt' (Errcode: 13) when executing 'SELECT INTO OUTFILE'

You can resolve this complaint by making sure that /tmp/path is owned by the mysql user (and also writeable by the current Unix user). Thanks JinRong Ye!

This format is convenient if none of your data contains tabs. In NLP, however, it is quite possible that your text will contain tabs.

The XML MySQL dump format

Enter the XML MySQL dump format:

        <table_data name="sources">
        <row>
                <field name="id">1</field>
                <field name="created_at">2009-03-07 22:06:36</field>
                <field name="text">&quot;You've got to be kidding me&quot;</field>

Ah… pure bliss. You can get the XML dump format as follows:

mysqldump --xml database