Search⌘ K
AI Features
Home
Courses
Performing Natural Language Processing with R

Analyzing Textual Comparisons with Document-Term Matrices

Explore how to create and analyze document-term matrices (DTMs) using R for textual comparisons. Understand tokenization, term frequency, and how DTMs preserve document context to reveal similarities and differences in text collections.

We'll cover the following...

Why use document-term matrices?

The following code lists the tokens and their frequencies:

R
# This displays leading n-grams ------------------------
shelleyText |>
  removePunctuation() |>
  removeWords(stopwords('english')) |>
  removeWords(c("I")) |>
  removeNumbers() |>
  stripWhitespace() |>
  Boost_tokenizer() |>
  vapply(paste, "", collapse=" ") |>
  table() |>
  sort(decreasing = TRUE) |>
  head(n = 10)

  • Line 3: We use the pipe (|>) operator to pass the shelleyText data through a series of text processing functions.

  • Line 9: This step involves tokenization, breaking the text into individual words or tokens. ...