Appropriate error-measure for de-duping?

One important thing to do is to avoid all-pairs comparisons using canopies whenever possible. The idea is to find some small high-recall low-precision features you can index by, and only compare records which match in at least one canopy.

Apart from that, as far as I know, state-of-the-art approaches involve mostly scoring "similarity" in the different fields with manually set weights and then manually setting thresholds for merging records that keep precision / recall in a small manually deduped set of records at acceptable levels. I think one of the old coref examples in factorie for doing bibtex record matching might be helpful, but the code is not very easy to read (and the new, cleaner, version is spread over many files).

One other important thing is to do deduping incrementally: it's often somewhat easy to find "consensus" values for dedupped records, and comparing other records against the consensus value might make it easier to see matches than comparing against other noisy samples.