Weighted Engagement Scoring

Weighted Engagement Scoring

Overview

Calculate a composite engagement score E = sum(w_i * x_i) from multiple heterogeneous signals (votes, ratios, comment volume, sentiment quality, temporal recency) by normalizing each to a common scale and combining via documented weights. The core principle: raw engagement metrics on different scales cannot be meaningfully combined without normalization, and weights without documented rationale are arbitrary.

When to Use

• Multiple engagement signals available (score, upvote ratio, comment count, sentiment, timestamps) and need a single ranking metric
• Raw metrics are on incompatible scales (e.g., scores in thousands, ratios 0-1, comment counts in tens)
• Need to identify representative high-value content from a corpus
• Sorting by a single metric (e.g., top score) misses important content that excels on other dimensions
• Downstream analysis requires a ranked subset of "most engaged" or "most influential" content

When NOT to use:

• Only one engagement metric is available (just sort by that metric)
• Comparing scores across different platforms or communities without recalibrating weights
• Ranking users rather than content (engagement scoring measures content reception, not user quality)
• Content quality assessment without a sentiment or quality signal (engagement != quality)

digraph decision {
    rankdir=LR;
    "Multiple engagement\nmetrics available?" [shape=diamond];
    "Metrics on\ncomparable scales?" [shape=diamond];
    "Need single\nranking metric?" [shape=diamond];
    "Use weighted\nengagement scoring" [shape=box];
    "Sort by single metric" [shape=box];
    "Combine raw metrics\n(no normalization needed)" [shape=box];
    "Use multi-column\nfiltering instead" [shape=box];

    "Multiple engagement\nmetrics available?" -> "Sort by single metric" [label="no"];
    "Multiple engagement\nmetrics available?" -> "Metrics on\ncomparable scales?" [label="yes"];
    "Metrics on\ncomparable scales?" -> "Need single\nranking metric?" [label="no"];
    "Metrics on\ncomparable scales?" -> "Combine raw metrics\n(no normalization needed)" [label="yes"];
    "Need single\nranking metric?" -> "Use weighted\nengagement scoring" [label="yes"];
    "Need single\nranking metric?" -> "Use multi-column\nfiltering instead" [label="no"];
}

Core Formula

E = w_1 * norm(score) + w_2 * norm(upvote_ratio) + w_3 * norm(comments) + w_4 * norm(sentiment) + w_5 * norm(recency)

where:
  - norm(x) transforms each metric to [0, 1] range
  - sum(w_i) = 1.0 (weights are proportions)
  - Each w_i has a documented rationale

Workflow

Copy this checklist and track progress:

Weighted Engagement Scoring Progress:
- [ ] Step 1: Inventory available metrics and assess quality
- [ ] Step 2: Normalize each metric to [0, 1]
- [ ] Step 3: Select and document weights
- [ ] Step 4: Compute composite scores
- [ ] Step 5: Validate scores against human judgment
- [ ] Step 6: Run sensitivity analysis on weights
- [ ] Step 7: Identify high-value content using the composite score
- [ ] Step 8: Write findings to docs/analysis/08-weighted-engagement-scoring.md

Step 1: Inventory Available Metrics and Assess Quality

Before computing anything, catalog what signals are available and their characteristics.

Common engagement signals:

Quality checks before proceeding:

• What percentage of records have each metric? (Coverage)
• Are any metrics constant or near-constant? (Zero variance = no signal)
• Are two metrics highly correlated (r > 0.85)? (Redundant weighting risk)

import pandas as pd
import numpy as np

# Assess metric availability and basic statistics
metrics = ['score', 'upvote_ratio', 'num_comments', 'sentiment_compound']
for col in metrics:
    if col in df.columns:
        coverage = df[col].notna().mean() * 100
        print(f"{col}: coverage={coverage:.1f}%, mean={df[col].mean():.3f}, "
              f"std={df[col].std():.3f}, min={df[col].min():.3f}, max={df[col].max():.3f}")

# Check pairwise correlations
available = [c for c in metrics if c in df.columns]
print("\nCorrelation matrix:")
print(df[available].corr().round(3))

Step 2: Normalize Each Metric to [0, 1]

Every metric MUST be normalized before weighting. Combining raw metrics gives disproportionate influence to whichever metric has the largest numerical range.

Normalization methods (choose per metric):

def normalize_minmax(series):
    """Standard min-max to [0, 1]. Use for bounded, well-distributed metrics."""
    mn, mx = series.min(), series.max()
    if mx == mn:
        return pd.Series(0.5, index=series.index)  # No variance = neutral
    return (series - mn) / (mx - mn)

def normalize_robust(series, lower_pct=5, upper_pct=95):
    """Robust min-max using percentiles. Use for skewed metrics with outliers."""
    p_low = np.percentile(series.dropna(), lower_pct)
    p_high = np.percentile(series.dropna(), upper_pct)
    if p_high == p_low:
        return pd.Series(0.5, index=series.index)
    normalized = (series - p_low) / (p_high - p_low)
    return normalized.clip(0, 1)

def normalize_log_minmax(series):
    """Log-transform then min-max. Use for power-law distributed counts."""
    log_series = np.log1p(series)  # log(x + 1) handles zeros
    return normalize_minmax(log_series)

def compute_recency_score(timestamps, half_life_days=30):
    """Exponential decay: recent content scores higher.
    half_life_days controls how fast old content loses value."""
    now = timestamps.max()  # Use corpus max, not wall-clock time
    age_days = (now - timestamps).dt.total_seconds() / 86400
    decay = np.exp(-0.693 * age_days / half_life_days)  # 0.693 = ln(2)
    return decay  # Already in [0, 1]

# Apply normalization -- choose method per metric
df['norm_score'] = normalize_log_minmax(df['score'])
df['norm_ratio'] = normalize_minmax(df['upvote_ratio'])  # Already 0-1 but re-normalize within corpus
df['norm_comments'] = normalize_robust(df['num_comments'])
df['norm_sentiment'] = normalize_minmax((df['sentiment_compound'] + 1) / 2)  # Shift from [-1,1] to [0,1]
df['norm_recency'] = compute_recency_score(pd.to_datetime(df['created_utc'], unit='s'))

Normalization method selection guidance:

digraph norm_select {
    rankdir=TB;
    "Metric has known\nbounds (e.g. ratio)?" [shape=diamond];
    "Heavy skew or\npower-law?" [shape=diamond];
    "Extreme outliers?" [shape=diamond];
    "Standard min-max" [shape=box];
    "Log min-max" [shape=box];
    "Robust min-max\n(percentile)" [shape=box];
    "Rank-based" [shape=box];

    "Metric has known\nbounds (e.g. ratio)?" -> "Standard min-max" [label="yes"];
    "Metric has known\nbounds (e.g. ratio)?" -> "Heavy skew or\npower-law?" [label="no"];
    "Heavy skew or\npower-law?" -> "Extreme outliers?" [label="yes"];
    "Heavy skew or\npower-law?" -> "Robust min-max\n(percentile)" [label="no, moderate skew"];
    "Extreme outliers?" -> "Log min-max" [label="yes"];
    "Extreme outliers?" -> "Rank-based" [label="extreme"];
}

Step 3: Select and Document Weights

Weights MUST have documented rationale. Never use equal weights by default without stating why.

Weight selection approaches:

Recommended starting weights (for identifying influential discussions):

These are STARTING weights. Step 6 (sensitivity analysis) will test whether results are robust to reasonable weight variations.

# Define weights with rationale
weights = {
    'norm_score':     0.25,  # Community-validated value signal
    'norm_ratio':     0.15,  # Consensus quality indicator
    'norm_comments':  0.25,  # Discussion generation signal
    'norm_sentiment': 0.15,  # Quality filter (sentiment adjustment)
    'norm_recency':   0.20,  # Temporal relevance
}

assert abs(sum(weights.values()) - 1.0) < 1e-9, "Weights must sum to 1.0"

Step 4: Compute Composite Scores

# Compute weighted engagement score
df['engagement_score'] = sum(
    df[metric] * weight for metric, weight in weights.items()
    if metric in df.columns
)

# Handle partial metric availability (see Insufficient Data Handling)
available_weight = sum(
    weight for metric, weight in weights.items() if metric in df.columns
)
if available_weight < 1.0:
    # Re-normalize: scale up to account for missing metrics
    df['engagement_score'] = df['engagement_score'] / available_weight
    print(f"WARNING: Only {available_weight:.0%} of weight budget covered. "
          f"Missing metrics: {[m for m in weights if m not in df.columns]}")

# Basic score distribution
print(f"Engagement score: mean={df['engagement_score'].mean():.3f}, "
      f"median={df['engagement_score'].median():.3f}, "
      f"std={df['engagement_score'].std():.3f}")
print(f"Score range: [{df['engagement_score'].min():.3f}, {df['engagement_score'].max():.3f}]")

Step 5: Validate Against Human Judgment

Never trust a composite score without validation. Pull the top 10 and bottom 10 items by engagement score and manually inspect them.

Validation questions:

• Do the top-10 items genuinely represent high-value, high-engagement content?
• Do any top-10 items feel wrong? (If so, which metric is inflating them?)
• Do the bottom-10 items genuinely represent low-engagement content?
• Are there items you know are high-value that the score ranks poorly? (Missing signal)

# Pull top and bottom items for manual review
top_10 = df.nlargest(10, 'engagement_score')[
    ['title', 'engagement_score'] + list(weights.keys())
]
bottom_10 = df.nsmallest(10, 'engagement_score')[
    ['title', 'engagement_score'] + list(weights.keys())
]
print("=== TOP 10 by Engagement Score ===")
print(top_10.to_string())
print("\n=== BOTTOM 10 by Engagement Score ===")
print(bottom_10.to_string())

If validation reveals problems:

• Top items are all rage-bait/controversy: Increase sentiment weight
• Top items are all old viral content: Increase recency weight
• Top items are high-comment but shallow: Add or increase a depth/quality signal
• Score is dominated by one metric: That metric's normalization may be too spread; check the distribution

Step 6: Run Sensitivity Analysis

Test how much the ranking changes when weights vary by +/-0.10. If top-10 content is stable across reasonable weight perturbations, the model is robust. If rankings shift dramatically, the results depend heavily on subjective weight choices -- report this.

import itertools

def sensitivity_analysis(df, base_weights, perturbation=0.10, top_n=10):
    """Test ranking stability under weight perturbations."""
    base_top = set(df.nlargest(top_n, 'engagement_score').index)
    results = []

    for metric in base_weights:
        for direction in [-perturbation, +perturbation]:
            test_weights = base_weights.copy()
            test_weights[metric] += direction
            # Re-normalize remaining weights to maintain sum = 1.0
            others = {k: v for k, v in test_weights.items() if k != metric}
            other_sum = sum(others.values())
            if other_sum > 0:
                scale = (1.0 - test_weights[metric]) / other_sum
                for k in others:
                    test_weights[k] = others[k] * scale

            test_score = sum(
                df[m] * w for m, w in test_weights.items() if m in df.columns
            )
            test_top = set(test_score.nlargest(top_n).index)
            overlap = len(base_top & test_top) / top_n
            results.append({
                'perturbed_metric': metric,
                'direction': direction,
                'top_n_overlap': overlap,
            })

    results_df = pd.DataFrame(results)
    avg_stability = results_df['top_n_overlap'].mean()
    print(f"Average top-{top_n} stability: {avg_stability:.1%}")
    print(f"Minimum overlap: {results_df['top_n_overlap'].min():.1%}")
    print(results_df.to_string(index=False))
    return results_df

stability = sensitivity_analysis(df, weights)

Interpreting sensitivity results:

Step 7: Identify High-Value Content

Use the composite score to select representative high-value content for downstream analysis.

Selection strategies:

# Percentile-based selection (recommended default)
threshold_pct = 90  # Top 10%
threshold_value = df['engagement_score'].quantile(threshold_pct / 100)
high_value = df[df['engagement_score'] >= threshold_value].copy()
print(f"High-value content: {len(high_value)} items "
      f"(top {100 - threshold_pct}%, threshold={threshold_value:.3f})")

Step 8: Write Report

Write all findings to docs/analysis/08-weighted-engagement-scoring.md.

Report Output Template

The final report MUST be written to docs/analysis/08-weighted-engagement-scoring.md with this structure:

# Weighted Engagement Scoring Analysis

## Methodology

### Available Metrics
| Metric | Coverage | Distribution | Normalization Method |
|--------|----------|-------------|---------------------|
| ... | ...% | [skew/shape] | [method chosen] |

### Metric Correlations
[Pairwise correlation table for input metrics]

### Weight Selection
| Signal | Weight | Rationale |
|--------|--------|-----------|
| ... | ... | ... |

### Normalization Details
[Which method was applied to each metric and why]

## Score Distribution
- Mean: [value]
- Median: [value]
- Std dev: [value]
- Range: [min, max]
- [Note if distribution is clustered, bimodal, or well-spread]

## Validation
### Top-10 Manual Review
[Table of top-10 items with component scores]
[Assessment: do these genuinely represent high-value content?]

### Bottom-10 Manual Review
[Table of bottom-10 items with component scores]
[Assessment: do these genuinely represent low-value content?]

### Validation Outcome
[Pass/fail with specific observations]

## Sensitivity Analysis
| Perturbed Metric | Direction | Top-N Overlap |
|-----------------|-----------|---------------|
| ... | ... | ...% |

Average stability: [X%]
Interpretation: [robust/moderate/sensitive/unstable]

## High-Value Content Selection
- Selection method: [top-N / threshold / percentile / stratified]
- Items selected: [N] of [total] ([%])
- Score threshold: [value]

## Limitations and Caveats
- [Weight choices are analytical decisions, not ground truth]
- [Scores are relative within this corpus, not absolute]
- [Missing metrics and their impact]
- [Sensitivity findings if model is not robust]
- [Engagement is not quality -- high engagement includes controversy]

Good Patterns

• Normalize before weighting. This is non-negotiable. Raw metric combination produces meaningless scores dominated by whichever metric has the largest numerical range.
• Document weight rationale. Every weight choice should have a one-sentence justification. "Because it seemed right" is not a rationale.
• Run sensitivity analysis. If the top-10 shifts dramatically when weights change by 10%, the model is fragile. Report this.
• Validate against human judgment. Pull the top-10 and bottom-10 and manually check if the rankings make sense. Automated metrics cannot replace this step.
• Use the score to isolate representative content, not to make absolute quality claims. The score identifies content worth examining, not content that is objectively "best."
• Check metric correlations first. Two highly correlated metrics (e.g., score and comment count, r > 0.85) given equal weight effectively double-count one signal.
• Use log-transform on power-law metrics. Scores and comment counts follow power laws; a single viral post should not dominate the entire scoring range.

Anti-Patterns

Boundaries

SHOULD do:

• Calculate composite scores from multiple normalized engagement signals
• Identify high-value discussions using the composite score
• Document weight rationale for every weight in the model
• Perform sensitivity analysis to test ranking robustness
• Validate top and bottom items against human judgment
• Report scores as relative within the corpus

Should NOT do:

• Compare scores across different platforms or communities without recalibrating (different communities have different engagement baselines)
• Use scores to rank users rather than content
• Claim the weights are universal or optimal (they are analytical choices for a specific goal)
• Present the composite score as a quality metric without including a sentiment/quality signal
• Skip normalization for any metric regardless of how "similar" the scales appear

Insufficient Data Handling

digraph insufficient_data {
    rankdir=TB;
    node [shape=diamond, style=filled, fillcolor="#f0f0f0"];

    start [label="How many metrics\navailable?"];
    one [label="Only 1 metric", shape=box, fillcolor="#ffcccc"];
    two [label="2-3 metrics", shape=box, fillcolor="#fff3cc"];
    four_plus [label="4+ metrics", shape=box, fillcolor="#ccffcc"];

    start -> one [label="1"];
    start -> two [label="2-3"];
    start -> four_plus [label="4+"];

    one_action [label="STOP: No composite needed.\nSort by the single metric.\nReport why composite is not viable.", shape=box, fillcolor="#ffcccc"];
    two_action [label="PROCEED WITH CAVEATS.\nCompute composite but flag\nlimited signal diversity.\nSensitivity analysis is critical.", shape=box, fillcolor="#fff3cc"];
    four_action [label="PROCEED.\nFull weighted scoring pipeline.", shape=box, fillcolor="#ccffcc"];

    one -> one_action;
    two -> two_action;
    four_plus -> four_action;
}

Common Mistakes

Red Flags -- Stop and Reassess

• All normalized metrics are clustered near 0.5 -- normalization is compressing all variance
• One metric dominates the composite (>80% of variance in the final score) -- effective single-metric sort
• Top-10 validation reveals obvious ranking failures -- model does not match human judgment
• Sensitivity analysis shows <50% overlap stability -- results are weight-dependent artifacts
• Score distribution is bimodal with a gap -- check if two distinct content populations are being merged

References

• Weighted Composite Metric for User Experience (MDPI, 2025) -- PCA-derived weights for composite UX metrics
• Weighted Sum Method (ScienceDirect) -- Formal description of weighted additive models
• Normalization Techniques for MCDM (ScienceDirect, 2022) -- Comparison of normalization methods and their impact on rankings
• Building an Engagement Scoring Model (Accrease) -- Step-by-step engagement scoring for analytics data
• Weighted Scoring Prioritization (Six Sigma) -- Weight assignment and sensitivity testing methodology
• Content Engagement Score: Measuring Impact (Census) -- Multi-signal content scoring patterns
• Risks of Composite Scores in Scouting (Medium) -- Pitfalls of composite scoring models
• Weighting Criteria and Sensitivity Analysis (Pressbooks) -- Sensitivity testing for weighted models