Communication Content Analysis
Overview
This Skill runs content analysis as measurement, not as file processing. Corpus, units, codebook, reliability, and validation are decided before any model is run, and automated labels are treated as measures whose quality must be demonstrated.
Required start
- Confirm the construct or descriptive target with communication-construct/communication-scale when the analysis claims measurement.
- Write the corpus contract before collecting data.
- Decide the unit before choosing a model.
Corpus contract
Platform/source and collection method:
Time window and sampling strategy:
Deduplication rule:
Language and translation handling:
Unit definition (message/article/post/thread/turn):
Population and exclusions:
Licensing, ToS, and ethics boundaries:
Provenance record:
Sampling must be justified against the population the claim targets. A convenience corpus can describe itself but cannot stand in for a population.
Codebook and labels
- Define units, dimensions, categories, and unitizing rules.
- Give examples and non-examples for each category.
- Separate the codebook (construct definitions) from the coding instructions (how to apply them).
- For human annotation: coder training, independence, disagreement resolution logged.
- For LLM-assisted coding: record model, prompt version, date, temperature, few-shot examples, sample records, and a human-verified subset; never present raw LLM output as validated labels.
Reliability
- Choose the agreement statistic appropriate to the design (e.g., Krippendorff's alpha for multi-coder nominal/ordinal content; Cohen's kappa or percent agreement only with justification).
- Report reliability on a random sample separate from training; do not report training agreement as study reliability.
- Report the final coded dataset's reliability, not only the pilot.
- When LLM coding is used, report agreement between LLM and human reference set and the human-verified sample size, and state the claim boundary.
Method selection within the goal
| Task |
Prefer |
Avoid unless validated |
| Predefined narrow construct |
Dictionary or supervised classifier with validation |
Topic model |
| Discovering latent topical structure |
Topic model (LDA/STM/BERTopic) with diagnostics |
Pretending topics are constructs |
| Continuous similarity |
Embeddings with explicit reference texts |
Raw nearest-neighbor claims |
| Reproducible coding at scale |
LLM coding with human verification or supervised classifier |
Unvalidated LLM labels |
| Comparing meanings across time/platform |
Matched corpora and invariance checks, not raw frequency |
Frequency = meaning change |
Validation layers
- Software: sample checks, unit/schema checks, expected failures.
- Method: model diagnostics, reliability, leakage controls, time-split splits for prediction.
- Substantive: close reading of exemplars, known cases, negative controls, face validity against theory.
- Robustness: alternative preprocessing, seeds, thresholds, corpus boundaries.
Output shape
End with: corpus table and provenance, codebook, reliability results, validation plan, label-claim mapping, and the claims the analysis can and cannot support.
Common anti-patterns
- Running a topic model on all text because "it is text".
- Reporting model fit (perplexity/coherence) as if it validated a construct.
- No leakage control when supervised labels come from the same time window being predicted.
- Training and reliability on the same coded set.
- Interpreting frequency, co-occurrence, or classifier accuracy as meaning, causality, or public opinion.
- Downloading platform data without checking license/ToS and privacy boundaries.
See references/content-analysis-protocol.md for the corpus/codebook/protocol template.
1---2name: communication-content-analysis3description: Plan and audit computational content analysis of communication, media, and platform text for corpus construction, sampling, codebooks, dictionary/embedding/topic models, supervised classification, LLM-assisted coding, reliability, validation, and reporting. Use for content analysis, computational text, coding, reliability, 内容分析/计算内容分析/编码/信度/语料库/LLM编码. Do not use for construct definition or scale work, which route to communication-construct and communication-scale.4---56# Communication Content Analysis78## Overview910This Skill runs content analysis as measurement, not as file processing. Corpus, units, codebook, reliability, and validation are decided before any model is run, and automated labels are treated as measures whose quality must be demonstrated.1112## Required start13141. Confirm the construct or descriptive target with communication-construct/communication-scale when the analysis claims measurement.152. Write the corpus contract before collecting data.163. Decide the unit before choosing a model.1718## Corpus contract1920```text21Platform/source and collection method:22Time window and sampling strategy:23Deduplication rule:24Language and translation handling:25Unit definition (message/article/post/thread/turn):26Population and exclusions:27Licensing, ToS, and ethics boundaries:28Provenance record:29```3031Sampling must be justified against the population the claim targets. A convenience corpus can describe itself but cannot stand in for a population.3233## Codebook and labels3435- Define units, dimensions, categories, and unitizing rules.36- Give examples and non-examples for each category.37- Separate the codebook (construct definitions) from the coding instructions (how to apply them).38- For human annotation: coder training, independence, disagreement resolution logged.39- For LLM-assisted coding: record model, prompt version, date, temperature, few-shot examples, sample records, and a human-verified subset; never present raw LLM output as validated labels.4041## Reliability4243- Choose the agreement statistic appropriate to the design (e.g., Krippendorff's alpha for multi-coder nominal/ordinal content; Cohen's kappa or percent agreement only with justification).44- Report reliability on a random sample separate from training; do not report training agreement as study reliability.45- Report the final coded dataset's reliability, not only the pilot.46- When LLM coding is used, report agreement between LLM and human reference set and the human-verified sample size, and state the claim boundary.4748## Method selection within the goal4950| Task | Prefer | Avoid unless validated |51|---|---|---|52| Predefined narrow construct | Dictionary or supervised classifier with validation | Topic model |53| Discovering latent topical structure | Topic model (LDA/STM/BERTopic) with diagnostics | Pretending topics are constructs |54| Continuous similarity | Embeddings with explicit reference texts | Raw nearest-neighbor claims |55| Reproducible coding at scale | LLM coding with human verification or supervised classifier | Unvalidated LLM labels |56| Comparing meanings across time/platform | Matched corpora and invariance checks, not raw frequency | Frequency = meaning change |5758## Validation layers59601. Software: sample checks, unit/schema checks, expected failures.612. Method: model diagnostics, reliability, leakage controls, time-split splits for prediction.623. Substantive: close reading of exemplars, known cases, negative controls, face validity against theory.634. Robustness: alternative preprocessing, seeds, thresholds, corpus boundaries.6465## Output shape6667End with: corpus table and provenance, codebook, reliability results, validation plan, label-claim mapping, and the claims the analysis can and cannot support.6869## Common anti-patterns7071- Running a topic model on all text because "it is text".72- Reporting model fit (perplexity/coherence) as if it validated a construct.73- No leakage control when supervised labels come from the same time window being predicted.74- Training and reliability on the same coded set.75- Interpreting frequency, co-occurrence, or classifier accuracy as meaning, causality, or public opinion.76- Downloading platform data without checking license/ToS and privacy boundaries.7778See [references/content-analysis-protocol.md](references/content-analysis-protocol.md) for the corpus/codebook/protocol template.
haochengw372-hash