Hands-on Entity Resolution with Python-v3.png

dockerfile.tar.gz

conda-main

A typical business stores data across multiple systems, including ERPs for operations, a CRM for marketing, files, notebooks, and BI apps for other purposes. Records of the same customer (entity) exist in multiple places, likely not in sync across nor unique within sources. This inconsistent situation generates an opportunity for us to drive business value by cross-referencing and deduplicating records with entity resolution.

This course covers business acumen and hands-on coding. It starts with several business cases and a quick introduction to entity resolution in Python. Then, it explores semantic-preserving preprocessing, similarity feature engineering, graph clustering, weak supervision, confident learning, and integration.

As a developer, you’ll increase your company’s business value by developing and deploying entity resolution pipelines. As a decision-maker, you’ll know which solution best suits your business cases and how to negotiate the best value for your money.

An Introduction to Entity Resolution in Python

We engineer a C++ library for entity resolution and create a Python interface as a binary file. We define a Python environment on Snowflake by installing the package using the file and uploading it to Snowflake. Finally, we author the entity resolution pipeline as a dbt model.

We use RecordLinkage and other Python dependencies on Snowflake’s Anaconda channel. We author the entity resolution pipeline as a dbt model and specify all the dependencies in the dbt configuration dictionary.

We write our entity resolution pipeline in Snowflake’s SQL dialect and author the entity resolution pipeline as a dbt model.

We use Splink and other Python dependencies on Snowflake’s Anaconda channel. We author the entity resolution pipeline as a dbt model and specify all the dependencies in the dbt configuration dictionary.

Our team uses Snowflake’s SaaS platform for analytics workloads and authors transformations with dbt. Which integration option below does not play well with the current setup?

Our team uses Snowflake’s SaaS platform for analytics workloads and authors transformations with dbt. Which integration option below does not play well with the current setup?

DuckDB is an embedded database engine optimized for vector search, making it an excellent option for storing and processing text, image, and audio embeddings.

DuckDB scales horizontally, adding more machines to the computing pool to overcome resource bottlenecks.

DuckDB can be embedded in memory (aka serverless), which is how it is usually used. However, it also natively supports a client-server architecture, making it a solid alternative to Oracle and other OLTP alternatives.

DuckDB is an embedded OLAP database engine optimized to execute expensive data transformation jobs in SQL utilizing multiple cores of a single machine.

Which statement about DuckDB is true?

Some warehouses support only a few runtimes and many just SQL. Because dbt is a Python package, we can also author transformations in dbt using Python, run them on our local computer, and push results to the remote warehouse.

dbt comes with a package manager so that we can benefit from solutions open-sourced by the dbt community.

dbt is a templating language that compiles short SQL or Python scripts into lengthy code executed on one of many supported warehouse engines.

With dbt, we avoid repeating ourselves—for example, we can write for loops and macros instead of copy-pasting the same code with a few parameters changed.

What is not a good reason to use dbt for authoring transformations in a warehouse?

What is not a good reason to use dbt for authoring transformations in a warehouse?

Splink does not require binary classification because it is an unsupervised technique.

Splink uses a Fellegi-Sunter model, whereas RecordLinkage is not limited to a single matching model.

Splink formulates every task as deduplication, whereas in RecordLinkage, we can choose between linkage and deduplication configurations.

Both frameworks author entity resolution tasks in Python. RecordLinkage also executes the computation in Python. Splink translates the Python script into SQL so that we can leverage our Snowflake SQL runtime for the heavy lifting.

What makes the Splink framework different from RecordLinkage?

What makes the Splink framework different from RecordLinkage?

These services provide zero-infrastructure and zero-configuration entity resolution after they have been integrated with our data warehouse.

We don't need to reinvent the wheel by developing an in-house entity resolution package.

The entity resolution engine runs on a dedicated server. Isolating it means fewer conflicts with the rest of our stack.

External experts in the entity resolution domain help us to apply best practices.

Buying an entity resolution service, like Tilores, comes with several key advantages. Which option is not an advantage of Tilores?

Buying an entity resolution service, like Tilores, comes with several key advantages. Which option is not an advantage of Tilores?

Evaluate your knowledge about the integration of entity resolution.

Introduction to Entity Resolution and Applications

A Quickstart Guide Using the RecordLinkage Package

Preprocessing

Indexing

Feature Engineering

Pairwise Matching

Clustering

Integration

Entity Resolution Fundamentals

Matching Products Across Two Online Shops

Conclusion

Appendix

Quiz on Integration