Hands-on Entity Resolution with Python-v3.png

dockerfile.tar.gz

conda-main

A typical business stores data across multiple systems, including ERPs for operations, a CRM for marketing, files, notebooks, and BI apps for other purposes. Records of the same customer (entity) exist in multiple places, likely not in sync across nor unique within sources. This inconsistent situation generates an opportunity for us to drive business value by cross-referencing and deduplicating records with entity resolution.

This course covers business acumen and hands-on coding. It starts with several business cases and a quick introduction to entity resolution in Python. Then, it explores semantic-preserving preprocessing, similarity feature engineering, graph clustering, weak supervision, confident learning, and integration.

As a developer, you’ll increase your company’s business value by developing and deploying entity resolution pipelines. As a decision-maker, you’ll know which solution best suits your business cases and how to negotiate the best value for your money.

An Introduction to Entity Resolution in Python

It will always improve the precision using collective evidence.

Recall will always either increase or stay constant.

Precision and recall increase, or both decrease, depending on the clustering quality.

Precision or recall is not guaranteed to improve, but their harmonic mean, the $F_1$ score, will always benefit.

What is the impact of connected components clustering on the matching quality compared to pairwise prediction?

What is the impact of connected components clustering on the matching quality compared to pairwise prediction?

$n_{TC}$ is strictly smaller than $n_{CC}$ and $n_{MCC}$.

There is no guaranteed order in numbers of clusters among these algorithms.

Let $n_{TC}$, $n_{CC}$, and $n_{MCC}$ denote the total number of predicted clusters after transitive clustering, CC, and MCC. How do these numbers relate?

Let <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>n</mi><mrow><mi>T</mi><mi>C</mi></mrow></msub></mrow><annotation encoding="application/x-tex">n_{TC}</annotation></semantics></math>nTC​, <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>n</mi><mrow><mi>C</mi><mi>C</mi></mrow></msub></mrow><annotation encoding="application/x-tex">n_{CC}</annotation></semantics></math>nCC​, and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>n</mi><mrow><mi>M</mi><mi>C</mi><mi>C</mi></mrow></msub></mrow><annotation encoding="application/x-tex">n_{MCC}</annotation></semantics></math>nMCC​ denote the total number of predicted clusters after transitive clustering, CC, and MCC. How do these numbers relate?

We have multiple conflicts in our predictions—for example, the first and third records must match if all our positive predictions are correct.

We have two clusters after CC and a single after MCC.

We have a single cluster after CC and MCC.

We can still use transitive clustering if we do not have predicted probabilities. We would assign all four records to a single cluster in this case.

Let $s_{i,j}$ denote the predicted probability of a match between the $i$-th and $j$-th record. Further, let $s_{2,1}=0.95$, $s_{3,2}=0.85$, and $s_{4,3}=0.9$ denote the predicted probabilities of all pairs that are classified as matches. All other pairs are predicted not to match.

We implement CC so that the first component in the tuple is always assigned as the center and the other as the noncenter when both records are not yet set.

Which statement below is not true?

Let <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>j</mi></mrow></msub></mrow><annotation encoding="application/x-tex">s_{i,j}</annotation></semantics></math>si,j​ denote the predicted probability of a match between the <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math>i-th and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi></mrow><annotation encoding="application/x-tex">j</annotation></semantics></math>j-th record. Further, let <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mn>2</mn><mo separator="true">,</mo><mn>1</mn></mrow></msub><mo>=</mo><mn>0.95</mn></mrow><annotation encoding="application/x-tex">s_{2,1}=0.95</annotation></semantics></math>s2,1​=0.95, <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mn>3</mn><mo separator="true">,</mo><mn>2</mn></mrow></msub><mo>=</mo><mn>0.85</mn></mrow><annotation encoding="application/x-tex">s_{3,2}=0.85</annotation></semantics></math>s3,2​=0.85, and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mn>4</mn><mo separator="true">,</mo><mn>3</mn></mrow></msub><mo>=</mo><mn>0.9</mn></mrow><annotation encoding="application/x-tex">s_{4,3}=0.9</annotation></semantics></math>s4,3​=0.9 denote the predicted probabilities of all pairs that are classified as matches. All other pairs are predicted not to match.
We implement CC so that the first component in the tuple is always assigned as the center and the other as the noncenter when both records are not yet set.
Which statement below is not true?

Markov clustering is a deterministic procedure that creates the same clusters after each run.

Markov clustering will never create fewer clusters than transitive clustering.

Markov clustering will never create fewer clusters than MCC.

Markov clustering will never create more clusters than CC.

Which statement about Markov clustering is true?

Which statement about Markov clustering is true?

Manual review is a supervision source, which can help increase the quality of the training data we use to fit pairwise matching algorithms.

There is no reason to review predictions after pairwise matching. We should always wait for predictions after clustering.

Manual review does not help in selecting a clustering algorithm.

Manual review always improves the overall quality of the entity resolution outcome, even without refitting models.

Which statement about the role of manual review in entity resolution is true?

Which statement about the role of manual review in entity resolution is true?

Evaluate your learnings about clustering in entity resolution.

Introduction to Entity Resolution and Applications

A Quickstart Guide Using the RecordLinkage Package

Preprocessing

Indexing

Feature Engineering

Pairwise Matching

Clustering

Integration

Entity Resolution Fundamentals

Matching Products Across Two Online Shops

Conclusion

Appendix

Quiz on Clustering