Email me your pitch and how you need help monetizing data.
If I like your pitch, I’ll give you a free consultation on data strategy (NLP, ML, business intelligence, etc.)
Afterwards, if we both think that I can add value to your business, we can talk about a longer-term relationship.

You should forward this blog post to any friend who could use this information.

What is data strategy?

Do you know how to monetize the data you have? How can you improve monetization using other data available to you? How do you transform your data into actionable business intelligence?

I can help you shape your data strategy, your long-term plan for how your business will capture, process, and monetize data. For example, data strategy can help you in the following circumstances:

• You don’t know who your individual users are or what they want, so you can’t effectively target ads.
• You don’t know what user behavior on your site to track.
• You don’t know what information you should start scraping from the web, information which you could use months or years down the line.

Besides working backwards from your business goals and business assets to a viable data strategy, I can also help you with more concrete challenges in NLP and machine learning:

• How do I improve my search engine so that users don’t miss out on relevant results?
• How do I add or improve recommendation, to connect users with what they want?
• How do I scale this ML algorithm to billions of examples with millions of features?
• How do I improve the accuracy of this NLP or ML tool?

Who am I?

My name is Joseph Turian, and I head MetaOptimize LLC. We consult on NLP, ML, and data strategy. We also run the MetaOptimize Q&A site, where ML and NLP experts share their knowledge.

• I am a data expert, holding a Ph.D. in natural language processing and machine learning. I have a decade of experience in these topics. I specialize in large data sets.
• I’m business-minded, so I focus on business goals and the most direct path of execution to achieve these goals.
• I am also a technology generalist who has been hacking since age 10 and has programmed competitively at a world-class level.

References from clients past and present available upon request.

What is the offer?

You send me information about what you’re doing and why you think I can help you.
Bonus points if you send me your deck, so I can understand your entire business picture. You are asking me to invest valuable expertise and potentially IP in your company, so appeal to me as a potential investor.
Demerits if you send me an NDA prematurely. Uptight companies who think what they are doing isn’t protected by good execution are a turn-off. But if you must be all James Bond about it, I’ll still consider you.

If I like what you’re doing and I can budget time, we schedule a meeting (in person or over Skype) and I’ll give you a free consultation on what you’re doing.

If the initial meeting goes well, and we both see how I can add value to your business, we can decide to continue working together. I can continue to help you either:

• Advising you periodically about your data strategy.
• Building you new tools to use in your product.
• Licensing to you existing tools I’ve already built.
• Training your smart tech geeks on NLP and ML technology for you to build in-house.

Compensation accepted in the form of cash or equity or a mix of both. Pro-bono if you’re an awesome non-profit.

Why am I doing this?

• More deals is always good.
• I am a social hacker, and enjoy connecting and sharing with other entrepreneurs. I want to meet some more excellent people.
• I would like to improve my understanding of widespread challenges and pain points in data strategy. That way, I can build a product that is useful for many people.
• This is an interesting social business experiment.

Who is this offer for?

• Open-source projects looking to use NLP + ML to improve their users’ experience.
• Unfunded startups with a promising team, product, and market.
• Funded startups.
• Established companies.

What are you waiting for?

Email me your pitch and how you need help monetizing data.
Or forward this blog post to a friend who could use this information.

We release code written by Ali Afshar, which turns the KEA keyphrase extractor into an XML-RPC service. This allows you to use KEA as a service, calling it from a variety of different programming languages. The code is released under the New BSD License.

Background

Keyphrase extraction (AKA terminology mining, term extraction, term recognition, or glossary extraction) is the process of extracting multi-word phrases that summarize the meaning of a text passage.

For example, in this document entitled “The Growing Global Obesity Problem: Some Policy Options to Address It”, the keyphrases might be: [“developing countries”, “food consumption”, “overweight”, “taxes”, “prices”, “price policies”, “fiscal policies”, “feeding habits”, “nutritional requirements”, “diet”, “nutrition policies”, and “food intake”.]

These keyphrases are useful for summarizing the topic of the text. Also, these keyphrases are useful in later NLP processing steps, and sometimes more informative and disambiguating than just the individual word tokens in the text.

KEA is a great keyphrase extraction implementation. It is useful because it is open-source, backed by solid research, comes with some annotated training data, and because it can extract keyphrases over unrestricted text, without needing a vocabulary of possible keyphrases.

Other implementations of keyphrase extraction include:

• Maui, a topic extractor from the same people that wrote KEA.
• topia.termextract is a Python term extractor, which is relatively noisy, and proposes many bogus keywords, but it simple to use. This is my recommendation for quick-and-dirty but works immediately out-of-the-box.

API implementations include:

• Termine by NacTem, a permissive term extractor I’ve used in the past. They will give you bulk access for research purposes. It is a UK webservice that also is relatively noisy, and proposes many bogus keywords. However, it appears to me to be slightly more accurate than topia.termextract. YMMV.
• Alchemy’s term extractor.
• The Yahoo term extraction API, which is now only available through YQL. It is low recall but high precision. In other words, it gives you a small number of high quality terms, but misses many of the terms in your documents.
• Five Filters, a web service version of topia’s term extractor (see above).
• Maui on Appspot.

Peter Turney has done a lot of research on keyphrase extraction, and licenses his implementation.

There is a wide academic literature on term extraction, which I won’t summarize here. The best introductory techniques are written by Park, who is now at IBM:
“Automatic glossary extraction: beyond terminology identification” and
“Glossary extraction and utilization in the information search and delivery system for IBM technical support”. You can read more about how to roll your own termex implementation here.

More information about the topic is available on the Maui blog.

Code

When running KEA, instead of a standalone program which reads input from disk, for speed one might want a resident service that keeps the model in memory. Additionally, one might want to call this service from non-Java languages. XML-RPC is a widely supported standard for implementing remote services.

We hereby release KEA service written by Ali Afshar, which turns the KEA keyphrase extractor into an XML-RPC service. This allows you to use KEA as a service, calling it from a variety of different programming languages. The code is released under the New BSD License.

Also included in the documentation is a description of how to this Java program was converted into a XML-RPC service.

I provide a basic indexing and retrieval code using the PyLucene 3.0 API. Lucene In Action (2nd Ed) covers Lucene 3.0, but the PyLucene code samples for have not been updated for the 3.0 API, only the Java ones. Unfortunately, there is currently little (no?) example PyLucene code in blogosphere. If you have links to more Lucene 3.0 tutorials and samples, please share them in the comments.

Update 20100810: In light of discussions with other, this post has been substantially rewritten since it was first posted.

Background

Historically, I have found it easy to write basic PyLucene 2.4 (or 2.9?) code. PyLucene includes Lucene In Action code samples ported from Java to Python, and these code samples are correct and easy to adapt. I recently was developing a new project based upon Lucene (biased-text-sample), and I decided to try PyLucene 3.0.2–1. I was surprised to find that PyLucene code samples in samples/LuceneInAction/ are out-of-date, and use the 2.x API. (Note: The code in samples/*.py appears to be updated to the 3.0 API.)

I was able to find no Lucene 3.0 tutorials or code samples on the web, except for this one article:

• Lucene In-Memory Search Example: Now updated for Lucene 3.0.1

If you have links to more Lucene 3.0 tutorials and samples, please share them in the comments.

Sample PyLucene 3.0 code

In the spirit of Lingpipe’s Lucene 2.4 in 60 seconds, here are relevant PyLucene 3.0 code snippets from my biased-text-sample project, for indexing and retrieval.

Indexing

import lucene
from lucene import \
    SimpleFSDirectory, System, File, \
    Document, Field, StandardAnalyzer, IndexWriter, Version

if __name__ == "__main__":
    lucene.initVM()
    indexDir = "/Tmp/REMOVEME.index-dir"
    dir = SimpleFSDirectory(File(indexDir))
    analyzer = StandardAnalyzer(Version.LUCENE_30)
    writer = IndexWriter(dir, analyzer, True, IndexWriter.MaxFieldLength(512))

    print >> sys.stderr, "Currently there are %d documents in the index..." % writer.numDocs()

    print >> sys.stderr, "Reading lines from sys.stdin..."
    for l in sys.stdin:
        doc = Document()
        doc.add(Field("text", l, Field.Store.YES, Field.Index.ANALYZED))
        writer.addDocument(doc)

    print >> sys.stderr, "Indexed lines from stdin (%d documents in index)" % (writer.numDocs())
    print >> sys.stderr, "About to optimize index of %d documents..." % writer.numDocs()
    writer.optimize()
    print >> sys.stderr, "...done optimizing index of %d documents" % writer.numDocs()
    print >> sys.stderr, "Closing index of %d documents..." % writer.numDocs()
    writer.close()
    print >> sys.stderr, "...done closing index of %d documents" % writer.numDocs()

Retrieval

import lucene
from lucene import \
    SimpleFSDirectory, System, File, \
    Document, Field, StandardAnalyzer, IndexSearcher, Version, QueryParser

if __name__ == "__main__":
    lucene.initVM()
    indexDir = "/Tmp/REMOVEME.index-dir"
    dir = SimpleFSDirectory(File(indexDir))
    analyzer = StandardAnalyzer(Version.LUCENE_30)
    searcher = IndexSearcher(dir)

    query = QueryParser(Version.LUCENE_30, "text", analyzer).parse("Find this sentence please")
    MAX = 1000
    hits = searcher.search(query, MAX)

    print "Found %d document(s) that matched query '%s':" % (hits.totalHits, query)

    for hit in hits.scoreDocs:
        print hit.score, hit.doc, hit.toString()
        doc = searcher.doc(hit.doc)
        print doc.get("text").encode("utf-8")

I speculate that job hopping, if it becomes a widespread phenomenon, might actually lead to improved business efficiency. In this way, the “Gen Y” job hopping phenomenon could ultimately prove beneficial.

Background

• Mark Suster begins the debate by writing: “[Job Hoppers] Make Terrible Employees”.
• Paul Dix responds that job hopping is not correlated with employee quality and there are many better ways to assess the value of an individual employee than the length of their previous jobs.
• Penelope Trunk thinks that job hoppers make the best employees, because they are more qualified and loyal.
• Mark Suster replies to Paul Dix, clarifying and defending his original arguments.
• Jason Calacanis looks at the overall trend of job hopping, and argues that it is a negative trait of Gen Y.
• Andrew Warner argues that startup employers might simply be mismanaging expectations.

I was discussing these articles today with Chris Kenton of SocialRep.

Issues with Jason Calacanis’s piece

Jason’s piece seems to have a very crochity tone, with a lot of: “The kids these days are driving society to hell in a handbasket” sort of feel. To wit:

• “the majority of them seem to lack killer instinct but have excel at entitlement“
• “It’s so obvious to me why our country is spiraling like a regional jet piloted by a $9 an hour, 20 year-old pilot with under 1,000 hours of flight time.“

These all sound like the sort of criticisms every older generation lobs at younger ones, which make me immediately skeptical.

Obviously, Jason has had negative experiences with employees who leave after one year. I’m not saying these employees were good. But I think he draws the wrong generalizations and I suspect that the trend of job hopping might ultimately lead to societal and economic good.

Could Job Hopping be beneficial?

I think there is definitely a social shift that is occurring, but I think this concept of discrete “generations” is a red herring, since the shift is occurring gradually, not as a step function.

Here I’m just going to speculate a bit: If Jason’s prediction is true, and ten years down the road it is not uncommon that most people job hop every year until they find a good relationship, it might not be as grim as the old guard predicts. In fact, it could ultimately have beneficial effects. I can understand how this idea is scary to conventional businesses, but since I don’t have extensive industry experience, I have the luxury of having little enough bias to use my imagination about how this might ultimately be beneficial.

The real problem with job hopping is the initial expensive startup cost to integrating a new employee into your organization. Job hopping would not really be so problematic if businesses (and new employees) were set up for people to contribute value immediately. Perhaps employers and employees alike would benefit from businesses restructuring their processes to be more modular and self-contained. This is similar to how it seems initially expensive to design your code so that components are loosely coupled, but ultimately this discipline leads to greater flexibility and easier maintainability. Similarly, structuring your organization and processes in such a way that you can easily add (or remove!) talent can ultimately lead to efficiency. (I make similar comments about outsourcing your code.)

As I said, this idea on my part is purely creative speculation, and I can’t claim I have enough experience to know whether this is true or not. So when it comes to whether job hopping is good (as Paul Dix says) or bad (as Mark Suster and Jason Calcanis say), I have to abstain.

The idea of blind loyalty is an artifact of situations in which the party to which you are loyal (a large corporation, an Army, etc.) is far too large to have a relationship with you. When an actual relationship is possible, that is far preferable to some impersonal loyalty.

Alignment of interests and clear communication is the best way to make any sort of relationship work.

According to common wisdom, the best code is developed in-house. I am beginning to believe this is only true when the code must be tightly coupled, or there are realistic security concerns. These scenarios are less common than managers like to believe.

For run-of-the-mill development projects, outsourcing might have advantages above-and-beyond cost savings. If your code effort can be outsourced, you should try it. Not only will it be cheaper, but the final code will be easier to maintain.

Background

KSplice recently wrote about the best way to manage interns. The takehome point is: “Divide tasks to be as loosely-coupled as possible.”

Recently, a commentator on thefunded.com asked:

I’ve been working on a deal in which a larger software company would give me a platform they developed so we can build a business around it. The larger company has given up on it.

The key developer of the platform was to be included in the deal. But he’s apparently disgruntled and, literally, has gone postal. (There are serious issues; getting him back isn’t really an option now.)

So we have a platform, without documentation, and without the guy who built it. But it has been launched in public applications and is perfectly functional. Basically we would just be reskinning it and adding in a few new features when we relaunch under the new business.

My advice? Try outsourcing.

Try outsourcing

Here was my advice to this person:

Your goal is to improve the maintainability of your code, so that you can easily find new developers to jump in on your project. Your goal is also to have the code at a point that you are no longer beholden to any developers, and you can easily fire a developer without feeling like you are locked in to them.

My advice is that you find a good project manager to document the code and, more importantly, refactor the codebase to make the components more loosely coupled. This project manager should break the code into pieces and delegate to a handful of independent remote subcontractors who don’t communicate with each other. If independent remote workers can refactor and clean up the code, without communicating with each other, then it means the final code will be easy to maintain. It then follows that an in-house development team should be able to easily jump into the codebase. Or, you could outsource further improvments. Your choice.

Consider that the approach of independent remote developers with little communication is the same approach taken by many open-source projects.

If the project is hard to break into pieces, this is why you need a good project manager. He or she will understand the overall architecture, and see along what lines it is best to create division of responsibilities in the code.

You could choose a single tightly-knit dev team who are in constant communication, but the risk is that they will understand aspects of the code that they don’t document, and that there will be communal wisdom passed around by oral communication. In this case, you are bound to these developers.

What you want is everything written down and easy to pick up by the next guy. So you should force that to be the case in your refactoring process.

Although it might take independent remote developers more time to refactor the code-base than a single tightly-knit development team, if you go with the independent remote coders then the final product will be easier to maintain in the long run. And even though the independent remote coders will incur two or three times as many billable hours as the tightly knit team, if you use foreign programmers then their hourly rate is four to five times less than domestic programmers. So I think it’s a win in terms of cost and final results.

Even though I am a hardcore developer myself, I have recently been dabbling in subcontracting to independent developers in Eastern Europe, and have been amazed with the results. It allows me to develop much faster, and it makes my code easier to maintain, because it is impossible to subcontract work unless your code has good separation of concerns and is loosely coupled. I now have built some good relationships with sharp coders who I trust to understand my directions and deliver clean code on time.

I sense that I am going to get push back on this by defensive domestic coders, because it goes against the common wisdom, but I think it is an option worth considering.

Would you share your experiences, positive and negative, with outsourcing?

After reading a handful of articles making tenuous connections between entrepreneurship and music, including :

• The Notorious CEO: Ten Startup Commandments from Biggie Smalls
• Being like The Sex Pistols can help your startup?

I’ve decided to come out and share my favorite startup music.

Dirt, by The Stooges, is a proto-punk cut that sprawls for seven-minutes, brooding and smoldering. It never climaxes or burns out, it just persists and drives forward.

Anyway, I believe this song should be the mantra for boostrappers, in particular those that practice the lean startup methodology.

Ooh, I been dirt / And I don’t care / Cause I’m burning inside / I’m just a yearning inside / And I’m the fire o’ life.

Without further ado, DIRT:

I propose a default “Constitution for Governance of Open-Source Projects”.

Background

I recently got involved in the OSQA project, which is a fork of CNPROG, which in turn is a clone of the StackExchange Q&A forum software.

Note that the OSQA project has no formal “homepage”, or instructions on how to get involved. I only discovered by chance that there is a mailing-list (unarchived) and developer chat room. Nor was it immediately clear which OSQA github fork should one use.

This is because OSQA grew organically from one contributor to a handful, and developer involvement was an afterthought in this project. Not that there is anything wrong with that.
However, now that a handful of people are involved in the project, and more people are trying to get involved, we have begun discussing governance and decision-making policies on the mailing list. In fact,
Evgeny Fadeev poses this very question on StackOverflow, and proposes some potential answers.

I believe that, by default, there are some simple but clear principles that should be enunciated. I hereby propose my

Constitution for Governance of Open-Source Projects (v20100227)

Let it be affirmed that the primary goal in instituting governance of an open-source project be to ensure the long-term health of the project.

Accordingly, the default bias should be towards openness and inclusiveness.
However, policy should be changed as issues present themselves, in order to maintain the long-term health of the project.

For the model of decision making, we favor a “do-ocracy”.
The people who contribute the most generally command the respect of the community.
Alienating them is the best way to derail the project.

The repository should be open the committers, given that commits can easily be reverted and commit-access easily revoked. This is preferable to alienating potential committers.

To ensure transparency for developers new and old, and allow them to decide their involvement in a project based upon the history of the project, their should be transparency and openess in the inner working of the project. For example, the email archive should be public.

Lastly, let us remember that too much red-tape gets in the way of progress. So red-tape and other barriers to contribution should be avoided, and only added as issues present themselves.

This Constitution can and should be amended as issues present themselves.

Therefore be it resolved.

A pattern for persisting generators is to turn them into pickle-able class objects. This is useful when you use generators for streaming training examples.

I would also try generator_tools, which might be a more convenient alternative to the pattern I describe. I haven’t used it yet.

Generators for streaming training examples

For machine learning, python generators are a simple idiom that make it easy to generate a stream of training examples. Moreover, you can nest generators:

• The inner generator can be used to read one example at a time.
• The outer generator can be used to read examples from the inner generator until you have a full minibatch, and then yield this minibatch.

Here is some example code:

[Update: The example holds without the ALL CAPS magic variable names, “HYPERPARAMETERS”. However, I include HYPERPARAMETERS because I am including the actual code I am using. Hyperparameters are global, read-only variables that specify the particular experimental condition being tested. I can’t say that I have the best solution to this particular aspect of experimental control (hyperparameters). I might write a blog post about it in the future, to solicit feedback on improved methods. However, I have refined my current approach over several years, and I can assure you that it is far less painful than a handful of more “clean” approaches.]

def get_train_example():
    HYPERPARAMETERS = common.hyperparameters.read("language-model")

    from vocabulary import wordmap
    for l in myopen(HYPERPARAMETERS["TRAIN_SENTENCES"]):
        prevwords = []
        for w in string.split(l):
            w = string.strip(w)
            id = None
            if wordmap.exists(w):
                prevwords.append(wordmap.id(w))
                if len(prevwords) >= HYPERPARAMETERS["WINDOW_SIZE"]:
                    yield prevwords[-HYPERPARAMETERS["WINDOW_SIZE"]:]
            else:
                prevwords = []

def get_train_minibatch():
    HYPERPARAMETERS = common.hyperparameters.read("language-model")
    minibatch = []
    for e in get_train_example():
        minibatch.append(e)
        if len(minibatch) >= HYPERPARAMETERS["MINIBATCH SIZE"]:
            assert len(minibatch) == HYPERPARAMETERS["MINIBATCH SIZE"]
            yield minibatch
            minibatch = []

You can’t persist training state by pickling your generators

However, generators become problematic when you want to persist your experiment’s state in order to later restart training at the same place. Unfortunately, you can’t pickle generators in Python. And it can be a bit of a PITA to workaround this, in order to save the training state.

Pattern to workaround this annoyance

Following useful discussion on pylearn-dev and stackoverflow [1] [2], I propose the following pattern for converting generators to pickle-able class objects:

1. Convert the generator to a class in which the generator code is the __iter__ method
2. Add __getstate__ and __setstate__ methods to the class, to handling pickling. Remember that you can’t pickle file objects. So __setstate__ will have to re-open files, as necessary.

Here is the updated code, after applying this pattern:

class TrainingExampleStream(object):
    def __init__(self):
        # Set the state variables, in case pickling happens before __iter__ is called.
        self.filename = None
        self.count = 0
        pass

    def __iter__(self):
        HYPERPARAMETERS = common.hyperparameters.read("language-model")
        from vocabulary import wordmap
        self.filename = HYPERPARAMETERS["TRAIN_SENTENCES"]
        self.count = 0
        for l in myopen(self.filename):
            prevwords = []
            for w in string.split(l):
                w = string.strip(w)
                id = None
                if wordmap.exists(w):
                    prevwords.append(wordmap.id(w))
                    if len(prevwords) >= HYPERPARAMETERS["WINDOW_SIZE"]:
                        self.count += 1
                        yield prevwords[-HYPERPARAMETERS["WINDOW_SIZE"]:]
                else:
                    prevwords = []

    def __getstate__(self):
        return self.filename, self.count

    def __setstate__(self, state):
        """
        @warning: We ignore the filename.  If we wanted
        to be really fastidious, we would assume that
        HYPERPARAMETERS["TRAIN_SENTENCES"] might change.  The only
        problem is that if we change filesystems, the filename
        might change just because the base file is in a different
        path. So we issue a warning if the filename is different from what is expected.
        """
        filename, count = state
        print >> sys.stderr, ("__setstate__(%s)..." % `state`)
        iter = self.__iter__()
        while count != self.count:
#            print count, self.count
            iter.next()
        if self.filename != filename:
            assert self.filename == HYPERPARAMETERS["TRAIN_SENTENCES"]
            print >> sys.stderr, ("self.filename %s != filename given to __setstate__ %s" % (self.filename, filename))
        print >> sys.stderr, ("...__setstate__(%s)" % `state`)

class TrainingMinibatchStream(object):
    def __init__(self):
        pass

    def __iter__(self):
        HYPERPARAMETERS = common.hyperparameters.read("language-model")
        minibatch = []
        self.get_train_example = TrainingExampleStream()
        for e in self.get_train_example:
            minibatch.append(e)
            if len(minibatch) >= HYPERPARAMETERS["MINIBATCH SIZE"]:
                assert len(minibatch) == HYPERPARAMETERS["MINIBATCH SIZE"]
                yield minibatch
                minibatch = []

    def __getstate__(self):
        return (self.get_train_example.__getstate__(),)

    def __setstate__(self, state):
        """
        @warning: We ignore the filename.
        """
        self.get_train_example = TrainingExampleStream()
        self.get_train_example.__setstate__(state[0])

If you have text data (like a web scrape) stored in a MySQL database, and you want to share the data, mysqldump to XML using the –xml flag.

When fields are unlikely to contain tabs, an even simpler format is a tab-separated file, created using the –tab=path flag to mysqldump. path must be owned by the MySQL database user.

The Problem with the standard MySQL dump format

The standard MySQL dump looks as follows

INSERT INTO `sources` VALUES (1,'2009-03-07 22:06:36','"You\'ve got to be kidding me"', ...

The problem is that the standard dump format is difficult to interact with programmatically.

It is difficult to parse using regular expressions because you cannot merely search for single quotes. You have to search for single quotes that are not preceded by a backslash (unless, perhaps, that backslash is preceded by a backslash).

Also, there are no libraries for reading the standard dump format, nor scripts for converting it into a standard format like JSON or XML. I asked the oracle as well as stackoverflow.

So if you receive a MySQL dump in the standard format, you might have to install MySQL and import the dump to get at your data.

The tabbed MySQL dump format

You can create a directory with one file per table, and the table will be one-row-per-line, with tab-separated values:

mysqldump --tab=path database

Here is some example output:

1	2009-03-07 22:06:36	"You've got to be kidding me"

If you get an error of the following form when you issue the mysqldump command:

mysqldump: Got error: 1: Can't create/write to file 'path/database.txt' (Errcode: 13) when executing 'SELECT INTO OUTFILE'

You can resolve this complaint by making sure that /tmp/path is owned by the mysql user (and also writeable by the current Unix user). Thanks JinRong Ye!

This format is convenient if none of your data contains tabs. In NLP, however, it is quite possible that your text will contain tabs.

The XML MySQL dump format

Enter the XML MySQL dump format:

        <table_data name="sources">
        <row>
                <field name="id">1</field>
                <field name="created_at">2009-03-07 22:06:36</field>
                <field name="text">&quot;You've got to be kidding me&quot;</field>

Ah… pure bliss. You can get the XML dump format as follows:

mysqldump --xml database

A script for automatically sorting graph curves, e.g. for gnuplot.

Problem

When you have a bunch of curves, and you plot them in an arbitrary order, you might get the following:

Typically, you want to sort the graphs in what appears to be visually descending order, as follows:

Sorting the curves is usually done manually, by eyeballing the curves. However, manual sorting of graph curves can become tedious. And when some curves don’t go out as far on the x-axis, it can be even trickier to place these short curves. (Some curves might be short if this experimental run trains more slowly.)

Heuristic approach

An automatic heuristic sorting approach is as follows:

• We maintain a sorted list of curves, from highest to lowest. The sorted list is initialized to empty.
• At each iteration, we find the curve that goes the furthest out on the x-axis, but is not yet in the sorted list. We then will choose where to insert it into the sorted list.
  • For this curve and all curves in the sorted list, we want an estimate of the curve value at the current curve’s furthest x-value. We compute this estimate using a moving average. (For this reason, all curves should have aligned x-axis steps, and should have equidistant x-axis steps.)
  • We place this curve into the sorted list, to minimize the number of rank errors of curve estimates at this x-value.

And that’s it!

Example output

Here is the sorted output of a larger, more difficult example, sorted using the above heuristic. Click on this image to get a larger version you can inspect:

A few of the decisions aren’t good. For example, why is curve 15 placed about curve 6? But most of the decisions are reasonable. For example, curve 13 is placed at the bottom, because it is very low compared to the other curves for the short duration that curve 13 is present.

Code

I have written a script implementing the heuristic above.

Here is the latest version of sort-curves.py.
You will also need movingaverage.py from my Python common library.

USAGE:

./sort-curves.py *.dat

where every *.dat is in standard (gnuplot) two-column-per-line format:

xvalue yvalue

Overall, I find this script a useful timesaver.