STRUCTURING DECISIONS FROM UNSTRUCTURED DATA (8:40 AM), with Seth Grimes, Ron Avnur, Paul Speciale and Staffan Truve.

The first long session of the conference is about the general problem inducing structure in data. Athough the topic is quite broad, I hope to see Seth Grimes leads the discuss to non-obvious and forward-thinking business applications, particularly of text mining.

MACHINE LEARNING’S IMPACT ON BUSINESS MODELS AND INDUSTRY STRUCTURES (9:10 AM), with George Gilbert, Currie Boyle, Alexander Gray, Mok Oh, and Amarnath Thombre.

Chris Dixon has written on the struggle for developing effective machine learning business models, arguing that ML is “too hot” to be marketed in a B2B setting. I would like to see speaker insight into ML services as a B2B business model, as opposed to internal use of ML.

PUZZLING (12:05 PM), with Jeff Jonas.

I’ve been meaning to see Jeff Jonas for a while, ever since my friend Todd Huffman (@odd) spoke glowingly of him. Jeff’s talk appears to extend an idea I’ve mentioned in a recent talk: The next step in predictive analytics is using joins on machine extracted data sets to extract higher-level information.

UNDERWRITING FOR THE UNDERBANKED THROUGH DATA MINING (3:00 PM), with Mathew Ingram and Douglass Merrill.

I’ve been interested in the use of ML for assessing credit more accurately since reading Pando Daily’s taxonomy of lending and learning about startups in that space. Niche areas in lending are growing; consider, for example, in vitro loans, and the fact that credit scores were historically difficult to estimate in Brazil.

Disclosure: MetaOptimize is a media partner for GigaOm Structure:Data, which means that I get a free pass in exchanging for covering the event. It also means you get a discount of 20% if you buy a ticket through this link.

The following are the core problems with current discussion systems:

1. Trolls, acrimonious people, and low quality commentary can drown out thoughtful discussion and destroy a good community.
2. Bias towards seniority: Deep insight is penalized if it comes from a new, unknown, or anonymous voice. For example, on Quora, answering a question one month faster than someone else can lead to a rich-get-richer phenomenon where the old answer gets more upvotes because it is always shown as the top answer, and hence has more visibility merely because it is older. Nepotism creates artificial friction and a barrier-entry, because it is an effective technique for enforcing community standards. But it has the downside that it discriminates—gently or extremely—-against insightful new commentators and anonymity.
3. Voting systems can be gamed by voting rings and ballot stuffing.
4. Voting systems can lead to “mob rule”.

How do we address all these core problems in traditional commenting systems?
How can we create an engaging system that most effectively promotes discussion? Can we avoid nepotism and bias against new, unknown, and anonymous commentators? How can we defend against basic trolling and voting rings?
A next generation discussion system must address these core problems.

The core value of a discussion system is to encourage stimulating and engaging discussion. We want a system that is frictionless to participate in: You can lurk for years and then jump in when you have something great and insightful to say, and your voice is heard loud and clear. This is true democratization of discussion.

The solution is personalization. A next generation discussion system is personalized. Personalization makes discussion more stimulating and engaging. Each user that reads and participates gets a comment thread that is sorted by personal relevancy. Irrelevant comments are hidden by default, but can optionally be viewed. Personalization is tuned to promote discussion that the user finds stimulating and engaging, and hiding discussion that the user finds off-topic, spammy, excessively or insufficiently detailed, etc.

The beauty of personalization is its flexibility. It does not force a particular style of discussion. If the user enjoys:

• heated discussion back-and-forth discussion,
• calm discussion with well-reasoned but concise arguments,
• in-depth academic discourse,
• tabloid-like ad-hominem, or
• trolling and hate speech

then the user gets what they want.

Additionally, personalization is adapted on a per-topic basis. One particular user might enjoy a heated discussion about abortion, a calm well-reasoned discussion about NoSQL databases, and low-brow discussion about celebrity romance. Per-topic personalization can satify all these user needs.

I can discuss more details about this approach, including how to:

• capture personalization information through user interaction with the discussion board.
• incorporate atomic commenting.
• federate discussion across multiple sites and liberate discussion from a single site.

I can also discuss possible objections to personalization, and my response to them.
Due to space limitations (500 words), I omit these details for now, and focus on personalization, which I believe addresses the core problems of traditional discussion systems.

I am not a Rails hacker, so getting Fat Free CRM installed and running is non-trivial for me.

fatfreecrm-ec2 will automatically deploy Fat Free CRM on a fresh Amazon EC2 micro instance. I have also tested it on a fresh Ubuntu Linode slice.

Caveat: The five minutes will probably be spent spinning up the EC2 instance. The script should only take about thirty seconds.

You can also try Fat Free CRM on Heroku. The last time I tried this recipe, I had problems I couldn’t resolve, which I discuss in the comment thread on that blog post. However, it looks like the author has recently updated his code, so his script might be a good alternative.

Enjoy!

In the spirit of shared tasks and NLP “bake offs”, I hereby announce the first MetaOptimize Challenge. It’s an open problem, and I am interested in involving practitioners who want to demo their style, as well as people who want to learn some large-scale IR/NLP. Hopefully, we’ll all learn something about various real-world approaches.

Join the announcement list to hear about any developments or important announcements.

Join the discuss list to chat about techniques and approaches.

I also have an ulterior motive.

The Problem

Let’s say I have several ten or hundred million documents, which are very short (only a few words). There are several million word types in the vocabulary. What is the fastest way to find the top-k (say k=10) semantically related words for each word in the vocabulary?

“Semantically related” is purposefully left vague.

When I say fastest, I mean that it should take under a week of computation time, and as little human time as possible. So use of existing implementations is encouraged. Single machine or righteously parallel solutions will both be considered, as long as your approach works and you demo it, preferably in the next two weeks.

Background

I brought up this question on MetaOptimize Q+A: Find semantically related terms over a large vocabulary (>1M)? I had some ideas in mind. But I wanted to hear about other ideas.

Olivier Grisel and Andrew Rosenberg commented on my question, suggesting I post this as a public challenge. So here goes. I hope people participate.

Why this is cool?

Here is one potential application:
Increased insight into emerging topics, trends, and new products. Run this on social media updates (Facebook posts, Tweets) after collecting sufficient mentions of a topic, trend or product, and have insight more insight into what is being discussed.

Coming up with other applications is an left as exercise for the reader.

Problem Details

Here is a sample dataset, for development (6.7 million documents, 40 MB gzipped). There is one document per line. Each word is separated by a space:

abbey seal
abbey seekers
abbey series

Here is the sample vocabulary file, in decreasing order of frequency (1.1 million word types, 4.5 MB gzipped).
The first column is the frequency and the second column is the word.
There might be words in the dataset that are not in the vocabulary:

  32972 group
  31998 research
  30820 information
  30090 uk
  29721 10
  29665 london

I will soon post a larger dataset and vocabulary file, and announce it on the –announce mailing list.

The desired output you produce is a file with eleven columns.
The first column should be identical to the second column of the vocabulary file. There will be as many lines as there are in the vocabulary file. The next ten columns should be the ten most related words, in descending order of relevance:

group groups working research support pm steering ltd pvc advisory age
research researchers centre group researcher project | programme institute unit council

The challenge is, within two week, to post a full output file for the larger dataset. By “full” I mean there is one line for every vocabulary word.

How will you evaluate it?

You should explain why your solution is correct. There is no “right” answer. Specifying evaluation pretty much determines the solution, as Alexandre Passos says (p.c.).

Honestly, being able to define the problem and justify your answer is half the puzzle.

Edit: For any submission, I will post for a random subset of vocab words each entry’s 10 related terms. I’ll then ask people to vote blind. This is a reasonable technique for quantitative evaluation.

Why is it hard?

First you need to take each word, and define a similarity measure between words, depending upon their usage. You need to define this similarity measure over an appropriate feature vector for each word, and choosing a good feature vector is not necessarily obvious.

Second of all, you need to do fast retrieval of the ten most similar words. But if you look at all 1M * 1M pairs, that’s 1 trillion comparisons.

Challenge Details

In a few days, I’m going to write a small post discussing the problem, and possible approaches. I will also point to existing open-source code that can perhaps solve the problem, so that skilled engineers have enough information to put together a working implementation, even if they have no background in NLP/IR. If you write up a good solution on your own blog or on the MetaOptimize Q+A forum, I’ll mention it in my blog post.

If you have a solution, please share it within, say, two weeks (Friday, November 19th). Share your full result file, or send it to me and I’ll put it on s3.

Join the announcment list to hear about any developments or important announcements.

Join the discuss list to chat about techniques and approaches.

Data Set

The data set is unique terms that occur in a crawl of .uk.

I took the UKWAC web-as-corpus crawl (2 billion words, crawled in 2008), ran it through the splitta sentence splitter, removed all funny characters, ran the Penn treebank word tokenizer, and perform term extraction with topia.termextract, discarding terms that are single words:

./sentencesplit.py | remove-nonascii-characters.pl | ~/dev/common-scripts/tokenizer.sed | ./topiaterms.py | gzip –c > ukwac-allmultiwordterms.txt.gz

I then lowercased the terms, sorted them, and uniqued them, to give the dataset:

zcat ukwac-allmultiwordterms.txt.gz | remove-nonascii-characters.pl | perl –ne ‘print lc($_);’ | sort | uniq | gzip –c > ukwac-uniqmultiwordterms.txt.gz

Finally, I constructed the vocabulary from the unique terms, to give the vocabulary:

zcat ukwac-uniqmultiwordterms.txt.gz | perl –ne ‘s/ /\n/g; print’ | sort | uniq –c | sort –rn | gzip –c > ukwac-vocabulary.txt.gz

Ulterior motive

If you do this, I have more exciting project work for you, and can pay. This is very similar to the style of interview question I ask, and it’s also very similar to the sort of work I do. So if you can hack it, you’re basically my ideal choice for a collaborator right now.

I introduce “information organization”, an approach which I have been exploring for several years. As a case study, music recommendations should be organized, but existing applications currently organize music recommendations poorly. I discuss issues with current applications, and discuss features that address these issues.

Background

Information organization is basically a family of patterns for collecting, presenting, and navigating information that is structured and/or textual. These patterns draw upon ideas from NLP, ML, IR, UX, viz, and in general the patterns can be implemented as loosely-coupled components. The combination of the patterns leads to potent cross-interactions that add up more to the sum of the individual patterns. This will be more clear when we dive into the case study. These patterns are non-trivial to implement, and require some savvy in NLP or IR. For people knowledgeable in these arts, there is an opportunity to create differentiate your offering and creating unique value by implementing these patterns.

As a family of patterns, information organization is not tied to any particular application. Rather, it admits a variety of related applications that can build upon implemented patterns. In this case study, I’ll be talking about an application to organization music recommendations.

I have been for several years been developing this concept of information organization. Some of it has been implemented, but much of it has been designed but not yet implemented yet.

The problem

People enjoy sharing music recommendations, but lack an effective platform for sharing these recommendations. We are discussing here about sharing music preferences in numerical form (scores) and textual form (reviews). I ignore the question of sharing the audio itself.

Here are some current approaches:

• Directly make a recommendation to a friend, online or off. Problem: Inherently transient and non-archival form of sharing. Also, there is no way for me to get recommendations from new people.
• Listen to the radio, online or off. Problem: Not social. Inherently transient and non-archival form of sharing.
• I write a blog article for myself or a larger publisher (e.g. Pitchfork) with my review of the music. Problem: Social features are limited. Discussion of particular songs is fragmented across publishers, which means that recommendations are not being effectively shared.
• I share a Youtube link on Facebook, and discussion ensues. Problem: The discussion is circumscribed purely by my social circle, and I don’t have a mechanism for connecting with people outside my circle with whom I would nonetheless like to share music recommendations. Also, the historical archive is not accessible, and previous recommendations are not searchable.

So what we’re getting at is a music recommendation system that has social, archival, and recommendation features.

Approach

Here’s an application that could solve these problems. Click on the mockup image for a larger view of it.

Note that I am going to talk about many possible features for this application. If you are going to implement something, you should focus on core features. I talk about to variety of possible features to illustrate information organization patterns that are technically feasible but not yet commonplace.

At its core, there are two kinds of user activity:

• Navigating recommendations.
• Adding recommendations.

I’ll focus on navigation, since navigation suggests many of the most important features.

When navigating information, consider viewing the recommendations for a particular song. This page will contain different recommendations for the song. These recommendations will be summarized as expandable text snippets, and are presented in a ranked order. For example, if you have reviewed this song, your recommendation will rank at the top. Recommendations from your friends have higher rank, as do recommendations from people with similar taste to you. (Some users care more about the taste of their friends, and the ranking should reflect that. Some users care more about the taste of people with similar tastes, and for them the ranking should reflect that.) Less important, but nonetheless useful, is the objective “authority” of the source. For example, recommendations by well-respected critics like Pitchfork have higher rank than recommendations by unknown critics, if there is not enough social or personal information to rank the recommendations.

Another aspect of navigating is search. Search should implement auto-complete and auto-suggest. Content should be auto-tagged based upon existing music meta-data as well as reviews, so that searching for “bounce” will find all bounce tracks, even if the term “bounce” is not explicitly mentioned in any review of a particular track. Auto-tagging can be smoothed across different tracks by the same artist, as well as different tracks that have reviews that contain the same keywords.

Another aspect of navigating is finding related entities (entity = song, musician, genre, tag, etc.). Besides seeing popular songs by the same musician, it is also useful to see popular songs in the same genre, related tags, etc. Auto-tagging helps again here to figure out how related two entities are.

There is also the issue that there is no portable open data format for recording numerical preferences about some entity (AFAIK). Simply formalizing the exchange of preference information (not just for music, but any type of entity) would be a big deal.

Recap

We have discussed a handful of different components (ranking based upon social graph, navigating based upon related music, etc.). These features are non-trivial to implement, and require some NLP or IR savvy. The challenge in implementing these features poses an opportunity to those who can. The more features implemented, the more value is created based upon their interactions, so the application can phase shift to a higher echelon of quality. But there is clearly value in a music recommendation system that has only a partial feature set. Which features are the easiest to implement that create the most value upfront? I believe that social features, and integration with Facebook and/or Twitter can add a lot of value in terms of creating engagement.

What is the minimum viable product for this task?
Possible answers:

• You log in with twitter, and type a review of a song or a band. This will be auto-tweeted, but also added to the recommendation page for this song or band.
• Scrape web reviews and recommendations. Extract a summary text snippet for each. (Note that this is a non-trivial feature.) Aggregate these snippets on the site.

A benefit of the first approach is that it is inherently social. A benefit of the second approach is that it combats the “cold start” problem, i.e. it immediately populates the database with useful information.

I am curious what you think is a good minimum viable product. Is personalization of the view part of the core feature set?

About the author

I consult on data strategy (NLP, ML, business intelligence, etc.)
If you are interested in building out any of these ideas, get in touch with me and I can help. In particular, I can advise on how to build the ML + NLP components. I’ll help you get a practical prototype up-and-running really quick, and show you how to refine and improve the components as necessary. Part of the art in implementing information organization is identifying the components that add the most immediate value, and quickly implementing a solid baseline.

Email me your pitch and how you need help monetizing data.
If I like your pitch, I’ll give you a free consultation on data strategy (NLP, ML, business intelligence, etc.)
Afterwards, if we both think that I can add value to your business, we can talk about a longer-term relationship.

You should forward this blog post to any friend who could use this information.

What is data strategy?

Do you know how to monetize the data you have? How can you improve monetization using other data available to you? How do you transform your data into actionable business intelligence?

I can help you shape your data strategy, your long-term plan for how your business will capture, process, and monetize data. For example, data strategy can help you in the following circumstances:

• You don’t know who your individual users are or what they want, so you can’t effectively target ads.
• You don’t know what user behavior on your site to track.
• You don’t know what information you should start scraping from the web, information which you could use months or years down the line.

Besides working backwards from your business goals and business assets to a viable data strategy, I can also help you with more concrete challenges in NLP and machine learning:

• How do I improve my search engine so that users don’t miss out on relevant results?
• How do I add or improve recommendation, to connect users with what they want?
• How do I scale this ML algorithm to billions of examples with millions of features?
• How do I improve the accuracy of this NLP or ML tool?

My name is Joseph Turian, and I head MetaOptimize LLC. We consult on NLP, ML, and data strategy. We also run the MetaOptimize Q&A site, where ML and NLP experts share their knowledge.

• I am a data expert, holding a Ph.D. in natural language processing and machine learning. I have a decade of experience in these topics. I specialize in large data sets.
• I’m business-minded, so I focus on business goals and the most direct path of execution to achieve these goals.
• I am also a technology generalist who has been hacking since age 10 and has programmed competitively at a world-class level.

References from clients past and present available upon request.

What is the offer?

You send me information about what you’re doing and why you think I can help you.
Bonus points if you send me your deck, so I can understand your entire business picture. You are asking me to invest valuable expertise and potentially IP in your company, so appeal to me as a potential investor.
Demerits if you send me an NDA prematurely. Uptight companies who think what they are doing isn’t protected by good execution are a turn-off. But if you must be all James Bond about it, I’ll still consider you.

If I like what you’re doing and I can budget time, we schedule a meeting (in person or over Skype) and I’ll give you a free consultation on what you’re doing.

If the initial meeting goes well, and we both see how I can add value to your business, we can decide to continue working together. I can continue to help you either:

• Advising you periodically about your data strategy.
• Building you new tools to use in your product.
• Licensing to you existing tools I’ve already built.
• Training your smart tech geeks on NLP and ML technology for you to build in-house.

Compensation accepted in the form of cash or equity or a mix of both. Pro-bono if you’re an awesome non-profit.

Why am I doing this?

• More deals is always good.
• I am a social hacker, and enjoy connecting and sharing with other entrepreneurs. I want to meet some more excellent people.
• I would like to improve my understanding of widespread challenges and pain points in data strategy. That way, I can build a product that is useful for many people.
• This is an interesting social business experiment.

Who is this offer for?

• Open-source projects looking to use NLP + ML to improve their users’ experience.
• Unfunded startups with a promising team, product, and market.
• Funded startups.
• Established companies.

What are you waiting for?

Email me your pitch and how you need help monetizing data.
Or forward this blog post to a friend who could use this information.

We release code written by Ali Afshar, which turns the KEA keyphrase extractor into an XML-RPC service. This allows you to use KEA as a service, calling it from a variety of different programming languages. The code is released under the New BSD License.

Background

Keyphrase extraction (AKA terminology mining, term extraction, term recognition, or glossary extraction) is the process of extracting multi-word phrases that summarize the meaning of a text passage.

For example, in this document entitled “The Growing Global Obesity Problem: Some Policy Options to Address It”, the keyphrases might be: [“developing countries”, “food consumption”, “overweight”, “taxes”, “prices”, “price policies”, “fiscal policies”, “feeding habits”, “nutritional requirements”, “diet”, “nutrition policies”, and “food intake”.]

These keyphrases are useful for summarizing the topic of the text. Also, these keyphrases are useful in later NLP processing steps, and sometimes more informative and disambiguating than just the individual word tokens in the text.

KEA is a great keyphrase extraction implementation. It is useful because it is open-source, backed by solid research, comes with some annotated training data, and because it can extract keyphrases over unrestricted text, without needing a vocabulary of possible keyphrases.

Other implementations of keyphrase extraction include:

• Maui, a topic extractor from the same people that wrote KEA.
• topia.termextract is a Python term extractor, which is relatively noisy, and proposes many bogus keywords, but it simple to use. This is my recommendation for quick-and-dirty but works immediately out-of-the-box.

API implementations include:

• Termine by NacTem, a permissive term extractor I’ve used in the past. They will give you bulk access for research purposes. It is a UK webservice that also is relatively noisy, and proposes many bogus keywords. However, it appears to me to be slightly more accurate than topia.termextract. YMMV.
• Alchemy’s term extractor.
• The Yahoo term extraction API, which is now only available through YQL. It is low recall but high precision. In other words, it gives you a small number of high quality terms, but misses many of the terms in your documents.
• Five Filters, a web service version of topia’s term extractor (see above).
• Maui on Appspot.

Peter Turney has done a lot of research on keyphrase extraction, and licenses his implementation.

There is a wide academic literature on term extraction, which I won’t summarize here. The best introductory techniques are written by Park, who is now at IBM:
“Automatic glossary extraction: beyond terminology identification” and
“Glossary extraction and utilization in the information search and delivery system for IBM technical support”. You can read more about how to roll your own termex implementation here.

More information about the topic is available on the Maui blog.

Code

When running KEA, instead of a standalone program which reads input from disk, for speed one might want a resident service that keeps the model in memory. Additionally, one might want to call this service from non-Java languages. XML-RPC is a widely supported standard for implementing remote services.

We hereby release KEA service written by Ali Afshar, which turns the KEA keyphrase extractor into an XML-RPC service. This allows you to use KEA as a service, calling it from a variety of different programming languages. The code is released under the New BSD License.

Also included in the documentation is a description of how to this Java program was converted into a XML-RPC service.

I provide a basic indexing and retrieval code using the PyLucene 3.0 API. Lucene In Action (2nd Ed) covers Lucene 3.0, but the PyLucene code samples for have not been updated for the 3.0 API, only the Java ones. Unfortunately, there is currently little (no?) example PyLucene code in blogosphere. If you have links to more Lucene 3.0 tutorials and samples, please share them in the comments.

Update 20100810: In light of discussions with other, this post has been substantially rewritten since it was first posted.

Background

Historically, I have found it easy to write basic PyLucene 2.4 (or 2.9?) code. PyLucene includes Lucene In Action code samples ported from Java to Python, and these code samples are correct and easy to adapt. I recently was developing a new project based upon Lucene (biased-text-sample), and I decided to try PyLucene 3.0.2–1. I was surprised to find that PyLucene code samples in samples/LuceneInAction/ are out-of-date, and use the 2.x API. (Note: The code in samples/*.py appears to be updated to the 3.0 API.)

I was able to find no Lucene 3.0 tutorials or code samples on the web, except for this one article:

• Lucene In-Memory Search Example: Now updated for Lucene 3.0.1

If you have links to more Lucene 3.0 tutorials and samples, please share them in the comments.

Sample PyLucene 3.0 code

In the spirit of Lingpipe’s Lucene 2.4 in 60 seconds, here are relevant PyLucene 3.0 code snippets from my biased-text-sample project, for indexing and retrieval.

Indexing

import lucene
from lucene import \
    SimpleFSDirectory, System, File, \
    Document, Field, StandardAnalyzer, IndexWriter, Version

if __name__ == "__main__":
    lucene.initVM()
    indexDir = "/Tmp/REMOVEME.index-dir"
    dir = SimpleFSDirectory(File(indexDir))
    analyzer = StandardAnalyzer(Version.LUCENE_30)
    writer = IndexWriter(dir, analyzer, True, IndexWriter.MaxFieldLength(512))

    print >> sys.stderr, "Currently there are %d documents in the index..." % writer.numDocs()

    print >> sys.stderr, "Reading lines from sys.stdin..."
    for l in sys.stdin:
        doc = Document()
        doc.add(Field("text", l, Field.Store.YES, Field.Index.ANALYZED))
        writer.addDocument(doc)

    print >> sys.stderr, "Indexed lines from stdin (%d documents in index)" % (writer.numDocs())
    print >> sys.stderr, "About to optimize index of %d documents..." % writer.numDocs()
    writer.optimize()
    print >> sys.stderr, "...done optimizing index of %d documents" % writer.numDocs()
    print >> sys.stderr, "Closing index of %d documents..." % writer.numDocs()
    writer.close()
    print >> sys.stderr, "...done closing index of %d documents" % writer.numDocs()

Retrieval

import lucene
from lucene import \
    SimpleFSDirectory, System, File, \
    Document, Field, StandardAnalyzer, IndexSearcher, Version, QueryParser

if __name__ == "__main__":
    lucene.initVM()
    indexDir = "/Tmp/REMOVEME.index-dir"
    dir = SimpleFSDirectory(File(indexDir))
    analyzer = StandardAnalyzer(Version.LUCENE_30)
    searcher = IndexSearcher(dir)

    query = QueryParser(Version.LUCENE_30, "text", analyzer).parse("Find this sentence please")
    MAX = 1000
    hits = searcher.search(query, MAX)

    print "Found %d document(s) that matched query '%s':" % (hits.totalHits, query)

    for hit in hits.scoreDocs:
        print hit.score, hit.doc, hit.toString()
        doc = searcher.doc(hit.doc)
        print doc.get("text").encode("utf-8")

I speculate that job hopping, if it becomes a widespread phenomenon, might actually lead to improved business efficiency. In this way, the “Gen Y” job hopping phenomenon could ultimately prove beneficial.

Background

• Mark Suster begins the debate by writing: “[Job Hoppers] Make Terrible Employees”.
• Paul Dix responds that job hopping is not correlated with employee quality and there are many better ways to assess the value of an individual employee than the length of their previous jobs.
• Penelope Trunk thinks that job hoppers make the best employees, because they are more qualified and loyal.
• Mark Suster replies to Paul Dix, clarifying and defending his original arguments.
• Jason Calacanis looks at the overall trend of job hopping, and argues that it is a negative trait of Gen Y.
• Andrew Warner argues that startup employers might simply be mismanaging expectations.

I was discussing these articles today with Chris Kenton of SocialRep.

Issues with Jason Calacanis’s piece

Jason’s piece seems to have a very crochity tone, with a lot of: “The kids these days are driving society to hell in a handbasket” sort of feel. To wit:

• “the majority of them seem to lack killer instinct but have excel at entitlement“
• “It’s so obvious to me why our country is spiraling like a regional jet piloted by a $9 an hour, 20 year-old pilot with under 1,000 hours of flight time.“

These all sound like the sort of criticisms every older generation lobs at younger ones, which make me immediately skeptical.

Obviously, Jason has had negative experiences with employees who leave after one year. I’m not saying these employees were good. But I think he draws the wrong generalizations and I suspect that the trend of job hopping might ultimately lead to societal and economic good.

Could Job Hopping be beneficial?

I think there is definitely a social shift that is occurring, but I think this concept of discrete “generations” is a red herring, since the shift is occurring gradually, not as a step function.

Here I’m just going to speculate a bit: If Jason’s prediction is true, and ten years down the road it is not uncommon that most people job hop every year until they find a good relationship, it might not be as grim as the old guard predicts. In fact, it could ultimately have beneficial effects. I can understand how this idea is scary to conventional businesses, but since I don’t have extensive industry experience, I have the luxury of having little enough bias to use my imagination about how this might ultimately be beneficial.

The real problem with job hopping is the initial expensive startup cost to integrating a new employee into your organization. Job hopping would not really be so problematic if businesses (and new employees) were set up for people to contribute value immediately. Perhaps employers and employees alike would benefit from businesses restructuring their processes to be more modular and self-contained. This is similar to how it seems initially expensive to design your code so that components are loosely coupled, but ultimately this discipline leads to greater flexibility and easier maintainability. Similarly, structuring your organization and processes in such a way that you can easily add (or remove!) talent can ultimately lead to efficiency. (I make similar comments about outsourcing your code.)

As I said, this idea on my part is purely creative speculation, and I can’t claim I have enough experience to know whether this is true or not. So when it comes to whether job hopping is good (as Paul Dix says) or bad (as Mark Suster and Jason Calcanis say), I have to abstain.

The idea of blind loyalty is an artifact of situations in which the party to which you are loyal (a large corporation, an Army, etc.) is far too large to have a relationship with you. When an actual relationship is possible, that is far preferable to some impersonal loyalty.

Alignment of interests and clear communication is the best way to make any sort of relationship work.

According to common wisdom, the best code is developed in-house. I am beginning to believe this is only true when the code must be tightly coupled, or there are realistic security concerns. These scenarios are less common than managers like to believe.

For run-of-the-mill development projects, outsourcing might have advantages above-and-beyond cost savings. If your code effort can be outsourced, you should try it. Not only will it be cheaper, but the final code will be easier to maintain.

Background

KSplice recently wrote about the best way to manage interns. The takehome point is: “Divide tasks to be as loosely-coupled as possible.”

Recently, a commentator on thefunded.com asked:

I’ve been working on a deal in which a larger software company would give me a platform they developed so we can build a business around it. The larger company has given up on it.

The key developer of the platform was to be included in the deal. But he’s apparently disgruntled and, literally, has gone postal. (There are serious issues; getting him back isn’t really an option now.)

So we have a platform, without documentation, and without the guy who built it. But it has been launched in public applications and is perfectly functional. Basically we would just be reskinning it and adding in a few new features when we relaunch under the new business.

My advice? Try outsourcing.

Try outsourcing

Here was my advice to this person:

Your goal is to improve the maintainability of your code, so that you can easily find new developers to jump in on your project. Your goal is also to have the code at a point that you are no longer beholden to any developers, and you can easily fire a developer without feeling like you are locked in to them.

My advice is that you find a good project manager to document the code and, more importantly, refactor the codebase to make the components more loosely coupled. This project manager should break the code into pieces and delegate to a handful of independent remote subcontractors who don’t communicate with each other. If independent remote workers can refactor and clean up the code, without communicating with each other, then it means the final code will be easy to maintain. It then follows that an in-house development team should be able to easily jump into the codebase. Or, you could outsource further improvments. Your choice.

Consider that the approach of independent remote developers with little communication is the same approach taken by many open-source projects.

If the project is hard to break into pieces, this is why you need a good project manager. He or she will understand the overall architecture, and see along what lines it is best to create division of responsibilities in the code.

You could choose a single tightly-knit dev team who are in constant communication, but the risk is that they will understand aspects of the code that they don’t document, and that there will be communal wisdom passed around by oral communication. In this case, you are bound to these developers.

What you want is everything written down and easy to pick up by the next guy. So you should force that to be the case in your refactoring process.

Although it might take independent remote developers more time to refactor the code-base than a single tightly-knit development team, if you go with the independent remote coders then the final product will be easier to maintain in the long run. And even though the independent remote coders will incur two or three times as many billable hours as the tightly knit team, if you use foreign programmers then their hourly rate is four to five times less than domestic programmers. So I think it’s a win in terms of cost and final results.

Even though I am a hardcore developer myself, I have recently been dabbling in subcontracting to independent developers in Eastern Europe, and have been amazed with the results. It allows me to develop much faster, and it makes my code easier to maintain, because it is impossible to subcontract work unless your code has good separation of concerns and is loosely coupled. I now have built some good relationships with sharp coders who I trust to understand my directions and deliver clean code on time.

I sense that I am going to get push back on this by defensive domestic coders, because it goes against the common wisdom, but I think it is an option worth considering.

Would you share your experiences, positive and negative, with outsourcing?