R NLP Skill
Sub-skills
| Sub-skill |
Description |
| r-nlp-text |
tidytext, quanteda, text2vec |
| r-nlp-topic |
LDA, STM, topic modeling |
| r-nlp-sentiment |
sentimentr, syuzhet, lexicons |
Natural Language Processing and text mining in R.
Core NLP Packages
| Package |
Description |
| tidytext ★ |
Tidy text mining (tidyverse style) |
| quanteda ★ |
Quantitative text analysis |
| tm |
Comprehensive text mining framework |
| text2vec ★ |
Fast vectorization & word embeddings |
| NLP |
Basic NLP functions |
| openNLP |
Apache OpenNLP interface |
Text Processing
| Package |
Description |
| stringr |
Consistent string manipulation |
| stringi |
ICU-based string processing |
| SnowballC |
Snowball stemmers |
| koRpus |
Text analysis package |
| utf8 |
UTF-8 text handling |
Sentiment Analysis
| Package |
Description |
| syuzhet |
Sentiment extraction (3 dictionaries) |
| sentimentr |
Sentence-level sentiment |
| tidytext |
Sentiment lexicons (AFINN, Bing, NRC) |
Topic Modeling
| Package |
Description |
| topicmodels |
LDA and CTM topic models |
| LDAvis |
Interactive topic model visualization |
| stm |
Structural topic models |
Other
| Package |
Description |
| zipfR |
Word frequency distributions |
| MonkeyLearn |
MonkeyLearn API interface |
| corporaexplorer |
Dynamic text collection exploration |
Quick Examples
# tidytext workflow
library(tidytext)
library(dplyr)
# Tokenize
df %>%
unnest_tokens(word, text) %>%
anti_join(stop_words) %>%
count(word, sort = TRUE)
# Sentiment analysis
df %>%
unnest_tokens(word, text) %>%
inner_join(get_sentiments("bing")) %>%
count(sentiment)
# TF-IDF
df %>%
unnest_tokens(word, text) %>%
count(document, word) %>%
bind_tf_idf(word, document, n)
# quanteda
library(quanteda)
corpus <- corpus(texts)
tokens <- tokens(corpus, remove_punct = TRUE)
dfm <- dfm(tokens) %>%
dfm_remove(stopwords("en")) %>%
dfm_trim(min_termfreq = 5)
# Topic modeling
library(topicmodels)
dtm <- cast_dtm(df, document, word, n)
lda <- LDA(dtm, k = 5, control = list(seed = 1234))
topics <- tidy(lda, matrix = "beta")
# text2vec word embeddings
library(text2vec)
it <- itoken(texts, tokenizer = word_tokenizer)
vocab <- create_vocabulary(it)
vectorizer <- vocab_vectorizer(vocab)
tcm <- create_tcm(it, vectorizer, skip_grams_window = 5)
glove <- GloVe$new(rank = 50)
word_vectors <- glove$fit_transform(tcm, n_iter = 10)
Common Workflows
Text Preprocessing
library(tidytext)
library(dplyr)
library(stringr)
clean_text <- df %>%
mutate(
text = str_to_lower(text),
text = str_remove_all(text, "[[:punct:]]"),
text = str_remove_all(text, "[[:digit:]]"),
text = str_squish(text)
) %>%
unnest_tokens(word, text) %>%
anti_join(stop_words) %>%
mutate(word = SnowballC::wordStem(word))
Document-Term Matrix
# tidytext to DTM
dtm <- df %>%
unnest_tokens(word, text) %>%
count(doc_id, word) %>%
cast_dtm(doc_id, word, n)
# quanteda DFM
dfm <- corpus(texts) %>%
tokens() %>%
dfm()
Resources
1---2name: r-nlp3description: R natural language processing packages. Use for text mining, sentiment analysis, topic modeling, tokenization, and text vectorization.4metadata:5 source: https://skillmd.com/skills/leolin990405/r-nlp6---78# R NLP Skill910## Sub-skills1112| Sub-skill | Description |13|-----------|-------------|14| [r-nlp-text](r-nlp-text/SKILL.md) | tidytext, quanteda, text2vec |15| [r-nlp-topic](r-nlp-topic/SKILL.md) | LDA, STM, topic modeling |16| [r-nlp-sentiment](r-nlp-sentiment/SKILL.md) | sentimentr, syuzhet, lexicons |1718Natural Language Processing and text mining in R.1920## Core NLP Packages2122| Package | Description |23|---------|-------------|24| **tidytext** ★ | Tidy text mining (tidyverse style) |25| **quanteda** ★ | Quantitative text analysis |26| **tm** | Comprehensive text mining framework |27| **text2vec** ★ | Fast vectorization & word embeddings |28| **NLP** | Basic NLP functions |29| **openNLP** | Apache OpenNLP interface |3031## Text Processing3233| Package | Description |34|---------|-------------|35| **stringr** | Consistent string manipulation |36| **stringi** | ICU-based string processing |37| **SnowballC** | Snowball stemmers |38| **koRpus** | Text analysis package |39| **utf8** | UTF-8 text handling |4041## Sentiment Analysis4243| Package | Description |44|---------|-------------|45| **syuzhet** | Sentiment extraction (3 dictionaries) |46| **sentimentr** | Sentence-level sentiment |47| **tidytext** | Sentiment lexicons (AFINN, Bing, NRC) |4849## Topic Modeling5051| Package | Description |52|---------|-------------|53| **topicmodels** | LDA and CTM topic models |54| **LDAvis** | Interactive topic model visualization |55| **stm** | Structural topic models |5657## Other5859| Package | Description |60|---------|-------------|61| **zipfR** | Word frequency distributions |62| **MonkeyLearn** | MonkeyLearn API interface |63| **corporaexplorer** | Dynamic text collection exploration |6465## Quick Examples6667```r68# tidytext workflow69library(tidytext)70library(dplyr)7172# Tokenize73df %>%74 unnest_tokens(word, text) %>%75 anti_join(stop_words) %>%76 count(word, sort = TRUE)7778# Sentiment analysis79df %>%80 unnest_tokens(word, text) %>%81 inner_join(get_sentiments("bing")) %>%82 count(sentiment)8384# TF-IDF85df %>%86 unnest_tokens(word, text) %>%87 count(document, word) %>%88 bind_tf_idf(word, document, n)8990# quanteda91library(quanteda)92corpus <- corpus(texts)93tokens <- tokens(corpus, remove_punct = TRUE)94dfm <- dfm(tokens) %>%95 dfm_remove(stopwords("en")) %>%96 dfm_trim(min_termfreq = 5)9798# Topic modeling99library(topicmodels)100dtm <- cast_dtm(df, document, word, n)101lda <- LDA(dtm, k = 5, control = list(seed = 1234))102topics <- tidy(lda, matrix = "beta")103104# text2vec word embeddings105library(text2vec)106it <- itoken(texts, tokenizer = word_tokenizer)107vocab <- create_vocabulary(it)108vectorizer <- vocab_vectorizer(vocab)109tcm <- create_tcm(it, vectorizer, skip_grams_window = 5)110glove <- GloVe$new(rank = 50)111word_vectors <- glove$fit_transform(tcm, n_iter = 10)112```113114## Common Workflows115116### Text Preprocessing117```r118library(tidytext)119library(dplyr)120library(stringr)121122clean_text <- df %>%123 mutate(124 text = str_to_lower(text),125 text = str_remove_all(text, "[[:punct:]]"),126 text = str_remove_all(text, "[[:digit:]]"),127 text = str_squish(text)128 ) %>%129 unnest_tokens(word, text) %>%130 anti_join(stop_words) %>%131 mutate(word = SnowballC::wordStem(word))132```133134### Document-Term Matrix135```r136# tidytext to DTM137dtm <- df %>%138 unnest_tokens(word, text) %>%139 count(doc_id, word) %>%140 cast_dtm(doc_id, word, n)141142# quanteda DFM143dfm <- corpus(texts) %>%144 tokens() %>%145 dfm()146```147148## Resources149150- tidytext book: https://www.tidytextmining.com/151- quanteda: https://quanteda.io/152- text2vec: https://text2vec.org/153- tm: https://tm.r-forge.r-project.org/