How the CBOW model works

The continuous bag-of-words (CBOW) model works in a similar way to the skip-gram algorithm, with one significant change in the problem formulation. In the skip-gram model, we predict the context words from the target word. However, in the CBOW model, we predict the target word from contextual words. Let’s compare what data looks like for the skip-gram algorithm and the CBOW model by taking the example sentence:

The dog barked at the mailman.

For the skip-gram algorithm, the data tuples—(input word, output word)—might look like this:

(dog, the), (dog, barked), (barked, dog), and so on.

For CBOW, the data tuples would look like the following:

([the, barked], dog), ([dog, at], barked), and so on.

Consequently, the input of the CBOW has a dimensionality of 2×m×D,2 × m × D, where mm is the context window size, and DD is the dimensionality of the embeddings. The conceptual model of CBOW is shown in the figure below:

Get hands-on with 1200+ tech skills courses.