Dedupe Examples

Example scripts for the dedupe, a library that uses machine learning to perform de-duplication and entity resolution quickly on structured data.

Part of the Dedupe.io cloud service and open source toolset for de-duplicating and finding fuzzy matches in your data. For more details, see the differences between Dedupe.io and the dedupe library.

To get these examples:

git clone https://github.com/dedupeio/dedupe-examples.git
cd dedupe-examples

or download this repository

cd /path/to/downloaded/file
unzip master.zip
cd dedupe-examples

Setup

We recommend using virtualenv and virtualenvwrapper for working in a virtualized development environment. Read how to set up virtualenv.

Once you have virtualenvwrapper set up,

mkvirtualenv dedupe-examples
pip install -r requirements.txt

Afterwards, whenever you want to work on dedupe-examples,

workon dedupe-examples

CSV example - early childhood locations

This example works with a list of early childhood education sites in Chicago from 10 different sources.

cd csv_example
pip install unidecode
python csv_example.py

(use 'y', 'n' and 'u' keys to flag duplicates for active learning, 'f' when you are finished)

To see how you might use dedupe with smallish data, see the annotated source code for csv_example.py.

Patent example - patent holders

This example works with Dutch inventors from the PATSTAT international patent data file

cd patent_example
pip install unidecode
python patent_example.py

(use 'y', 'n' and 'u' keys to flag duplicates for active learning, 'f' when you are finished)

Record Linkage example - electronics products

This example links two spreadsheets of electronics products and links up the matching entries. Each dataset individually has no duplicates.

cd record_linkage_example
python record_linkage_example.py

To see how you might use dedupe for linking datasets, see the annotated source code for record_linkage_example.py.

Gazetteer example - electronics products

This example links two spreadsheets of electronics products and links up the matching entries using the Gazetteer class

cd gazetteer_example.py
python gazetteer_example.py

MySQL example - IL campaign contributions

See mysql_example/README.md for details

To see how you might use dedupe with bigish data, see the annotated source code for mysql_example.

PostgreSQL big dedupe example - PostgreSQL example on large dataset

See pgsql_big_dedupe_example/README.md for details

This is the same example as the MySQL IL campaign contributions dataset above, but ported to run on PostgreSQL.

Athena example - IL campaign contributions

See athena_example/README.md for details

This is the same example as the MySQL IL campaign contributions dataset above, but ported to run on Athena.

Training

The secret sauce of dedupe is human input. In order to figure out the best rules to deduplicate a set of data, you must give it a set of labeled examples to learn from.

The more labeled examples you give it, the better the deduplication results will be. At minimum, you should try to provide 10 positive matches and 10 negative matches.

The results of your training will be saved in a JSON file for future runs of dedupe.

Here's an example labeling operation:

Phone :  2850617
Address :  3801 s. wabash
Zip :
Site name :  ada s. mckinley st. thomas cdc

Phone :  2850617
Address :  3801 s wabash ave
Zip :
Site name :  ada s. mckinley community services - mckinley - st. thomas

Do these records refer to the same thing?
(y)es / (n)o / (u)nsure / (f)inished