Tidyverse Expert

Tidyverse Expert - Comprehensive Data Manipulation Skill

Master data manipulation in R using the tidyverse's coherent ecosystem of packages. This skill provides expert guidance on transforming, cleaning, and reshaping data with complete control over all transformation operations.

The Tidyverse Philosophy

The tidyverse is built on a shared design philosophy:

1. Tidy data - Each variable is a column, each observation is a row, each value is a cell
2. Composable functions - First argument is always data, enabling pipe workflows
3. Type stability - Functions return predictable output types
4. Human-centered API - Intuitive, readable code that mirrors analytical thinking

Philosophy: Build data pipelines incrementally by composing focused functions, not by using monolithic operations. This enables infinite flexibility while maintaining code clarity.

Core Packages

Data Transformation Packages

• dplyr - Data manipulation grammar (filter, select, mutate, summarize, join)
• tidyr - Data tidying and reshaping (pivot, nest, separate, complete)
• purrr - Functional programming tools (map, walk, reduce, safely)

Specialized Manipulation Packages

• stringr - Consistent string manipulation with regex support
• forcats - Factor (categorical variable) handling and reordering
• lubridate - Date-time parsing, manipulation, and arithmetic

Package-Specific Guidance

dplyr: Data Manipulation Grammar

See references/dplyr-reference.md for complete documentation.

Five Core Verbs:

1. filter() - Keep rows matching conditions
2. select() - Keep or drop columns
3. mutate() - Create or modify columns
4. summarize() - Aggregate data to summary statistics
5. arrange() - Order rows by column values

Key Advanced Features:

• across() - Apply functions to multiple columns
• rowwise() - Row-by-row operations
• Window functions - lag(), lead(), cumsum(), rank()
• Multiple table operations - joins, set operations, binding

Common Patterns:

# Column selection helpers
select(starts_with("x"), ends_with("_id"), contains("temp"))

# Conditional mutation
mutate(category = case_when(
  value < 10 ~ "low",
  value < 50 ~ "medium",
  TRUE ~ "high"
))

# Grouped summaries
summarize(
  across(where(is.numeric), list(mean = mean, sd = sd)),
  .by = group_var
)

tidyr: Data Tidying and Reshaping

See references/tidyr-reference.md for complete documentation.

Core Operations:

• Pivoting - pivot_longer(), pivot_wider() for reshaping
• Nesting - nest(), unnest() for list-columns
• Rectangling - unnest_wider(), unnest_longer() for JSON/hierarchical data
• Missing values - complete(), fill(), drop_na(), replace_na()
• Column splitting - separate(), separate_wider_*(), unite()

Key Concepts:

# Pivot longer - wide to long format
pivot_longer(cols = -id, names_to = "variable", values_to = "value")

# Pivot wider - long to wide format
pivot_wider(names_from = category, values_from = measurement)

# Nesting for grouped operations
nest(.by = group_var) |>
  mutate(model = map(data, ~lm(y ~ x, data = .x))) |>
  mutate(predictions = map2(model, data, predict))

purrr: Functional Programming

See references/purrr-reference.md for complete documentation.

Map Family - Apply functions to lists/vectors:

• map() - Returns list
• map_dbl(), map_int(), map_chr(), map_lgl() - Type-specific output
• map2(), pmap() - Iterate over multiple inputs
• imap() - Iterate with indices/names
• walk() - For side effects (no return value)

Error Handling:

• safely() - Capture errors without stopping
• possibly() - Return default value on error
• quietly() - Capture messages, warnings, output

Predicates & Logic:

• keep(), discard() - Filter by predicate
• some(), every(), none() - Test conditions
• detect(), detect_index() - Find first match

Common Patterns:

# Read multiple files
files |> map(read_csv) |> list_rbind()

# Safe operations
results <- data |> map(safely(risky_function))
errors <- results |> map("error") |> discard(is.null)
successes <- results |> map("result") |> discard(is.null)

# Nested iterations
params |> pmap(\(x, y, z) run_model(x, y, z))

stringr: String Manipulation

See references/stringr-reference.md for complete documentation.

Core Functions (all start with str_*):

• Detection - str_detect(), str_starts(), str_ends(), str_which()
• Extraction - str_extract(), str_extract_all(), str_match(), str_sub()
• Replacement - str_replace(), str_replace_all(), str_remove(), str_remove_all()
• Transformation - str_to_lower(), str_to_upper(), str_to_title(), str_trim()
• Splitting - str_split(), str_split_fixed(), str_split_i()

Pattern Matching:

# Regex patterns
str_detect(text, "\\d{3}-\\d{4}")  # Phone pattern
str_extract_all(text, "[A-Z]\\w+")  # Capital words

# String cleaning
str_trim() |> str_squish() |> str_to_lower()

forcats: Factor Handling

See references/forcats-reference.md for complete documentation.

Key Operations:

• Reordering - fct_reorder(), fct_infreq(), fct_inorder()
• Recoding - fct_recode(), fct_collapse(), fct_other()
• Level manipulation - fct_relevel(), fct_rev(), fct_shift()
• Missing values - fct_explicit_na(), fct_drop()

Common Use Cases:

# Reorder for plotting
mutate(country = fct_reorder(country, value))

# Collapse rare levels
mutate(category = fct_lump_min(category, min = 100))

# Order by frequency
mutate(item = fct_infreq(item))

lubridate: Date-Time Manipulation

See references/lubridate-reference.md for complete documentation.

Parsing Functions:

• ymd(), mdy(), dmy() - Parse dates
• ymd_hms(), mdy_hm() - Parse date-times
• parse_date_time() - Flexible parsing

Extraction:

• year(), month(), day(), wday()
• hour(), minute(), second()
• quarter(), week()

Arithmetic:

• Durations - ddays(), dhours(), dminutes() (exact)
• Periods - days(), months(), years() (human-friendly)
• Intervals - %--% operator, int_length(), int_overlaps()

Common Patterns:

# Parse dates
dates <- mdy(c("12/31/2023", "01/01/2024"))

# Date arithmetic
today() + days(7)
floor_date(now(), "month")

# Extract components
wday(date, label = TRUE)  # "Mon", "Tue", ...
month(date, label = TRUE)  # "Jan", "Feb", ...

Common Workflow Patterns

Pattern 1: Data Import and Initial Cleaning

raw_data <- read_csv("data.csv") |>
  janitor::clean_names() |>                    # Standardize column names
  mutate(
    date = mdy(date_col),                      # Parse dates
    category = str_trim(str_to_lower(category)), # Clean strings
    value = as.numeric(str_remove(value, "\\$")) # Clean currency
  ) |>
  filter(!is.na(key_column)) |>                # Remove invalid rows
  distinct()                                   # Remove duplicates

Pattern 2: Complex Grouping and Summarization

summary <- data |>
  group_by(category, year = year(date)) |>
  summarize(
    across(where(is.numeric), list(
      mean = \(x) mean(x, na.rm = TRUE),
      median = \(x) median(x, na.rm = TRUE),
      sd = \(x) sd(x, na.rm = TRUE)
    )),
    n = n(),
    .groups = "drop"
  )

Pattern 3: Pivoting and Reshaping

# Wide to long
long_data <- wide_data |>
  pivot_longer(
    cols = matches("\\d{4}"),  # Year columns
    names_to = "year",
    values_to = "value",
    names_transform = list(year = as.integer)
  )

# Long to wide with multiple values
wide_data <- long_data |>
  pivot_wider(
    names_from = metric,
    values_from = c(value, error),
    names_glue = "{metric}_{.value}"
  )

Pattern 4: Nested Data and Models

nested_models <- data |>
  nest(.by = group) |>
  mutate(
    model = map(data, \(df) lm(y ~ x, data = df)),
    tidy = map(model, broom::tidy),
    glance = map(model, broom::glance),
    augment = map2(model, data, broom::augment)
  ) |>
  unnest(glance) |>
  arrange(desc(r.squared))

Pattern 5: Multiple Joins

combined <- sales |>
  left_join(customers, by = "customer_id") |>
  left_join(products, by = "product_id") |>
  left_join(regions, by = c("state" = "region_code")) |>
  mutate(
    revenue = quantity * unit_price,
    margin = revenue - (quantity * cost)
  )

Best Practices

Pipe Workflows

✅ DO:

• Use native pipe |> (R ≥ 4.1) preferred over magrittr %>%
• Break long pipes into intermediate objects for debugging
• Put each step on its own line
• Use .by argument in dplyr 1.1+ instead of group_by() |> ... |> ungroup()

❌ DON'T:

• Create pipes longer than 10 steps without breaking
• Mix pipe and base R assignment in confusing ways
• Forget to ungroup after group operations (if not using .by)

Column Selection

✅ DO:

• Use tidy-select helpers: starts_with(), ends_with(), contains(), matches(), where()
• Use across() for multi-column operations
• Use : for column ranges: select(id:name)

❌ DON'T:

• Use df$column inside dplyr verbs (breaks data masking)
• Repeat similar operations instead of using across()

Type Conversion

✅ DO:

• Use readr::parse_*() functions for robust parsing
• Use lubridate for dates, not base R
• Use forcats for factors, not base R

❌ DON'T:

• Use as.Date() with ambiguous formats
• Create factors with factor() when you need ordered levels
• Ignore parsing warnings from read_csv()

Missing Values

✅ DO:

• Use tidyr::complete() to make implicit missing values explicit
• Use tidyr::fill() to carry forward/backward
• Use coalesce() for value replacement
• Specify na.rm = TRUE in summary functions

❌ DON'T:

• Forget that filter() drops NAs by default
• Use na.omit() carelessly (drops entire rows)

Performance Tips

1. Use data.table for large data: tidyverse is optimized for readability, data.table for speed
2. Filter early: Reduce data size before expensive operations
3. Avoid row-wise operations: Vectorize when possible; use rowwise() only when necessary
4. Use where() instead of across(everything()): More targeted selections
5. Pre-allocate with joins: Use left_join() instead of rbind() in loops

Debugging Strategies

1. Break pipes: Assign intermediate results to inspect
2. Use glimpse(): Quick data structure check
3. Use count(): Verify grouping and filtering
4. Use slice_sample(): Test on small subset first
5. Check joins: Use anti_join() to find non-matches

Common Pitfalls

Pitfall 1: Grouped Data Propagation

# ❌ Group persists unexpectedly
df |>
  group_by(category) |>
  summarize(mean_val = mean(value)) |>
  mutate(diff = mean_val - lag(mean_val))  # Still grouped!

# ✅ Use .by or ungroup()
df |>
  summarize(mean_val = mean(value), .by = category) |>
  mutate(diff = mean_val - lag(mean_val))

Pitfall 2: Implicit NA Behavior

# ❌ filter() drops NAs silently
filter(value > 10)  # Loses NA rows

# ✅ Be explicit
filter(value > 10 | is.na(value))

Pitfall 3: Factor Ordering

# ❌ Alphabetical ordering
ggplot(aes(x = category, y = value)) + geom_col()

# ✅ Meaningful ordering
mutate(category = fct_reorder(category, value)) |>
  ggplot(aes(x = category, y = value)) + geom_col()

Supporting Resources

Complete Reference Documentation

• dplyr - references/dplyr-reference.md
• tidyr - references/tidyr-reference.md
• purrr - references/purrr-reference.md
• stringr - references/stringr-reference.md
• forcats - references/forcats-reference.md
• lubridate - references/lubridate-reference.md

Practical Examples

• Real-world workflows: examples/workflow-examples.md
• Complete case studies: examples/case-studies.md

Reusable Templates

• Common patterns: templates/data-wrangling-templates.md

Integration with Other Skills

• Use ggplot2 skill for visualization after data preparation
• Use r-tidymodels skill for machine learning workflows
• Use r-datascience skill for complete analysis guidance
• Use tidyverse-patterns skill for modern syntax updates

Quick Reference: Function Lookup