Search⌘ K
AI Features
Home
Courses
An Introduction to Entity Resolution in Python

Similarity Features

Explore how to apply similarity functions in Python for entity resolution using the RecordLinkage package. Understand indexing, configuring similarity scores for attributes, and evaluating match thresholds to improve duplicate detection in data records.

We'll cover the following...

RecordLinkage follows the following two main steps:

  1. Indexing: Select which pairs of records are duplicate candidates and therefore should be compared.
  2. Scoring: Configure and compute a vector of similarity functions for every pair in the index.

All-in indexing

We keep it simple here and add every possible pair to the index—a “full” index in the RecordLinkage terminology.

Python 3.8
import pandas as pd
import recordlinkage as rl
restaurants = pd.read_csv('solvers_kitchen/restaurants.csv')
indexer = rl.Index()
indexer.full()  # add all possible pairs to the index
candidate_links = indexer.index(restaurants.set_index('customer_id'))
print(candidate_links[:3])

Every element in the index is a pair of the customer_id values. The recordlinkage API warns us from using a full index, which can get very expensive computationally. That’s nothing we need to worry about now because of the small size of the data. The size of the full index is a simple function of the sample size.

Python 3.8
n = restaurants.shape[0]
print('Number of pairs in index: ', n * (n-1) / 2)

That’s roughly 373k pairs, which we will process in just a few seconds.

Measuring similarity

Our data below contains seven preprocessed attributes—clean and phonetic versions of the original data’s customer names, cities, and streets, and just clean phone numbers. We configure one similarity function per attribute.

Python 3.8
comparer = rl.Compare(n_jobs=-1)
print('Configuring one similarity function per attribute...')
for attribute in ['customer_name_c', 'customer_name_p', 'city_c', 'city_p']:
    comparer.string(left_on=attribute, right_on=attribute, method='jarowinkler', label=attribute + '_score')
for attribute in ['street_c', 'street_p']:
    comparer.string(left_on=attribute, right_on=attribute, method='damerau_levenshtein', label=attribute + '_score')
comparer.exact(left_on='phone_c', right_on='phone_c', label='phone_c_score')

The recordlinkage API has several built-in similarity functions. We have good reasons to choose different methods for different attributes.

The Jaro-WinklerIt is a similarity function counting nearby matching characters, transpositions, and emphasizing the start of strings. similarity emphasizes the start of strings, which makes sense for names and cities. It is based on the Jaro similarity defined as sj(x1,x2)=13(mlen(x1)+mlen(x2)+m−tm)s_j(x_1,x_2)=\frac{1}{3}(\frac{m}{len(x_1)}+\frac{m}{len(x_2)}+\frac{m-t}{m}) ...