Search⌘ K
AI Features
Home
Courses
An Introduction to Entity Resolution in Python

Embeddings and Vector Search with LanceDB

Explore how pretrained language models create dense vector embeddings to identify similar records programmatically. Understand using LanceDB to efficiently index and search vectors, enabling scalable entity resolution without exhaustive pairwise comparisons.

We'll cover the following...

We can leverage pretrained LLM and the latest innovations in vector databases to efficiently index even unstructured records before pairwise matching. Let’s start with the following toy example of four product records:

Python 3.8
import pandas as pd
df = pd.DataFrame([
  [0, 'Universal IR/RF Aeros Remote Control- MX850/ Laser Etched Buttons/ Centrally Located Joystick/ Memory Back-Up/ One Hand Ergonomics/ Controls Up To 20 Components'],
  [1, 'Aeros IR/RF Remote Control - MX850 - TV, DVD Player, Audio Receivers, Cable Box, Satellite Receiver, PVR (Personal Video Recorder) - 100 ft - Universal Remote'],
  [3, 'Sony DVD-R Recordable Camcorder Media 3 Pack - 3DMR30L1H/ 30 Minute, 1.4 GB/ Accucore Technology/ Store Digital Video, Audio And Multimedia Files/ 3 Pack'],
  [4, 'Sony VAIO Neoprene Laptop Carrying Case - VGPAMC3/ Compatible With VAIO A Series 15 And FS Series 15.4 Widescreen Notebooks/ Helps Protect Your Notebook From Scratches']
], columns=['id', 'description'])

The first two descriptions likely ...