Apache Tika File Similarity based on Jaccard distance, Edit distance & Cosine distance

This project demonstrates the usage of the Tika-Python package (Python port of Apache Tika) to compute file similarity based on metadata features.

The script can iterate over all files in the current directory, or specific files by command line, derive their metadata features, and compute the union of all features. The union of all features becomes the "golden feature set" that all document features are compared to via intersect. The length of that intersect per file divided by the length of the unioned set becomes the similarity score.

Scores are sorted in reverse (descending) order which can be shown in three different Data-Driven document visualizaions. A companion project to this effort is Auto Extractor which uses Apache Spark and Apache Nutch to take web crawl data, and produce D3-visualizations and clusters of similar pages.

Pre-requisite

• Install Tika-Python
• Install Java Development Kit

Installation

git clone https://github.com/chrismattmann/tika-similarity
pip install tika editdistance

You can also check out ETLlib

How to use

Optional: Compute similarity only on specific IANA MIME Type(s) inside a directory using --accept

Key-based comparison

This compares metadata feature names as a golden feature set. Scripts now live under the package layout.

#!/usr/bin/env python
python tikasimilarity/distance/similarity.py -f [directory of files] [--accept [jpeg pdf etc...]]
or 
python tikasimilarity/distance/similarity.py -c [file1 file2 file3 ...]

Value-based comparison

This compares metadata feature names together with its value as a golden feature set

#!/usr/bin/env python
python tikasimilarity/distance/value-similarity.py -f [directory of files] [--accept [jpeg pdf etc...]]
or 
python tikasimilarity/distance/value-similarity.py -c [file1 file2 file3 ...]

Edit Distance comparison on Metadata Values

• This computes pairwise similarity scores based on Edit Distance Similarity.
• Similarity Score of 1 implies an identical pair of documents.

#!/usr/bin/env python
python tikasimilarity/distance/edit-value-similarity.py [-h] --inputDir INPUTDIR --outCSV OUTCSV [--accept [png pdf etc...]] [--allKeys]

--inputDir INPUTDIR  path to directory containing files

--outCSV OUTCSV      path to directory for storing the output CSV File, containing pair-wise Similarity Scores based on Edit distance

--accept [ACCEPT]    Optional: compute similarity only on specified IANA MIME Type(s)

--allKeys            Optional: compute edit distance across all metadata keys of 2 documents, else default to only intersection of metadata keys

Eg: python tikasimilarity/distance/edit-value-similarity.py --inputDir /path/to/files --outCSV /path/to/output.csv --accept png pdf gif

Cosine Distance comparison on Metadata Values

• This computes pairwise similarity scores based on Cosine Distance Similarity.
• Similarity Score of 1 implies an identical pair of documents.

#!/usr/bin/env python
python tikasimilarity/distance/cosine_similarity.py [-h] --inputDir INPUTDIR --outCSV OUTCSV [--accept [png pdf etc...]]

--inputDir INPUTDIR  path to directory containing files

--outCSV OUTCSV      path to directory for storing the output CSV File, containing pair-wise Similarity Scores based on Cosine distance

--accept [ACCEPT]    Optional: compute similarity only on specified IANA MIME Type(s)

Robustness / Tika server handling

The distance scripts (jaccard, edit-value, cosine) include resilience logic (ported from PR #107 / issue #106) to detect Tika server failures (e.g. during very large batches) and attempt to kill + restart the server process via killserver(). See the implementations in tikasimilarity/distance/. Dedicated unit tests (using mocks) live in tests/test_killserver.py.

D3 visualization

Cluster viz

• Jaccard Similarity

* python tikasimilarity/cluster/cluster-scores.py [-t threshold_value] (for generating cluster viz)
* open cluster-d3.html(or dynamic-cluster.html for interactive viz) in your browser

• Edit Distance & Cosine Similarity

* python tikasimilarity/cluster/edit-cosine-cluster.py --inputCSV <PATH TO CSV FILE> --cluster <INTEGER OPTION> (for generating cluster viz)

  <PATH TO CSV FILE> - Path to CSV file generated by running edit-value-similarity.py or cosine_similarity.py
  <INTEGER OPTION> - Pass 0 to cluster based on x-coordinate, 1 to cluster based on y-coordinate, 2 to cluster based on similarity score

* open cluster-d3.html(or dynamic-cluster.html for interactive viz) in your browser

Default threshold value is 0.01.

Circlepacking viz

• Jaccard Similarity

* python tikasimilarity/cluster/circle-packing.py (for generating circlepacking viz)
* open circlepacking.html(or dynamic-circlepacking.html for interactive viz) in your browser

• Edit Distance & Cosine Similarity

* python tikasimilarity/cluster/edit-cosine-circle-packing.py --inputCSV <PATH TO CSV FILE> --cluster <INTEGER OPTION> (for generating circlepacking viz)

  <PATH TO CSV FILE> - Path to CSV file generated by running edit-value-similarity.py or cosine_similarity.py
  <INTEGER OPTION> - Pass 0 to cluster based on x-coordinate, 1 to cluster based on y-coordinate, 2 to cluster based on similarity score


* open circlepacking.html(or dynamic-circlepacking.html for interactive viz) in your browser

Composite viz

This is a combination of cluster viz and circle packing viz. The deeper color, the more the same attributes in the cluster.

* open compositeViz.html in your browser

Sunburst viz

Visualization of document clusters from pairwise similarity (e.g. Jaccard, edit, or cosine).

* python tikasimilarity/cluster/sunburst-cluster.py --input <CSV from sim> --output clusters.json --clusters 5
* open html/sunburst.html (uses clusters.json)

Big data way

if you are dealing with big data, you can use it this way:

* python tikasimilarity/cluster/generateLevelCluster.py (for generating level cluster viz)
* open levelCluster-d3.html in your browser

You can set max number for each node _maxNumNode(default _maxNumNode = 10) in generateLevelCluster.py

Questions, comments?

Send them to Chris A. Mattmann.

Contributors

• Chris A. Mattmann, JPL
• Dongni Zhao, USC
• Harshavardhan Manjunatha, USC
• Thamme Gowda, USC
• Ayberk Yılmaz, USC
• Aravind Ram, USC
• Aishwarya Parameshwaran, USC
• Rashmi Nalwad, USC
• Asitang Mishra, JPL
• Suzanne Stathatos, JPL

Running tests

The project includes unit tests (using only the Python standard library) for:

• The killserver() robustness logic (from PR #107) that was added to the three distance scripts.
• The sunburst-cluster.py script and its create_sunburst_json() function (from PR #110).

To run all tests:

python -m unittest discover -s tests -v

Or run specific modules:

python -m unittest tests.test_killserver tests.test_sunburst_cluster -v

The killserver tests use mocks so they never actually terminate any processes. The sunburst tests exercise happy paths, error conditions, and clustering logic using temporary CSV/JSON files.

License

This project is licensed under the Apache License, version 2.0.