Word Embeddings: Word2Vec and GloVe

The reviews The battery is great and lasts all day and Excellent power life between charges express a similar product experience. A bag-of-words or TF-IDF comparison finds little overlap because great and excellent, battery and power, and lasts all day and between charges use different tokens.

An embedding is a dense numeric vector for a word. Instead of assigning one separate position to every vocabulary term, an embedding gives each word a short list of learned numbers. Words used in similar surrounding text can receive vectors that are close together in vector space. That gives a similarity method a chance to recognize related usage even when two reviews share no exact word.

Embeddings do not understand a review in the human sense. They capture patterns present in the text used for training. A model trained only on a few dozen customer reviews has far less evidence than a model trained on a large and varied collection, so its relationships can be unstable.

Learning vectors from nearby words

To move beyond token overlap, Word2Vec learns from a context window, the nearby words surrounding a target word. In a review such as battery lasts all day, the words lasts, all, and day provide context for battery. Across many examples, Word2Vec adjusts vectors so that words appearing in similar contexts become more alike.

Word2Vec uses a prediction task during training. One form predicts a target word from nearby context words. Another predicts nearby context words from a target word. The training task is not a sentiment label or a similarity score. It is a way to force the model to store context patterns in its vectors.

The following corpus contains short, tokenized customer reviews. Training creates a vector for each sufficiently frequent ...