# Theme Detection Methodology

> The Theme Detector uses a 3-Dimensional Scoring Model to identify, rank, and assess market themes.

- Skill: `tools-only/theme-detection-methodology-2` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add tools-only/theme-detection-methodology-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/theme-detection-methodology-2/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-29
- Page: https://skillmd.com/skills/tools-only/theme-detection-methodology-2

---

# Theme Detection Methodology

## Overview

The Theme Detector uses a **3-Dimensional Scoring Model** to identify, rank, and assess market themes. Unlike single-score ranking systems, this approach separates the intensity of a theme (heat), its maturity stage (lifecycle), and the reliability of the signal (confidence) into independent dimensions.

---

## Dimension 1: Theme Heat (0-100)

Theme Heat measures the **direction-neutral strength** of a theme. A high heat score means the theme is generating significant market activity, regardless of whether it is bullish or bearish.

### Components

#### 1.1 Momentum Score (weight: 35%)

Measures the direction-neutral momentum strength of constituent industries using a log-sigmoid function on multi-timeframe weighted returns (1W 10%, 1M 25%, 3M 35%, 6M 30%).

**Formula:**
```
weighted_return = perf_1w * 0.10 + perf_1m * 0.25 + perf_3m * 0.35 + perf_6m * 0.30
momentum_score = 100 / (1 + exp(-2.0 * (ln(1 + |weighted_return|) - ln(16))))
```

Midpoint at |15%| weighted return. Log transform compresses extreme values for better mid-range separation.

**Data Source:** FINVIZ industry performance (1W, 1M, 3M, 6M change %)

**Scoring Scale:**
| Absolute Change % | Score |
|-------------------|-------|
| >= 5%             | 100   |
| 3-5%              | 80    |
| 1-3%              | 60    |
| 0.5-1%            | 40    |
| < 0.5%            | 20    |

Note: Absolute value is used so both strong bullish and strong bearish moves generate high heat.

#### 1.2 Volume Score (weight: 20%)

Measures abnormal volume activity across the theme.

**Formula:**
```
volume_score = normalize(avg_relative_volume, scale=[0.5, 3.0] -> [0, 100])

avg_relative_volume = MEAN(stock_volume / stock_avg_volume) for all stocks in theme
```

**Data Source:** FINVIZ relative volume (volume / avg volume ratio)

**Scoring Scale:**
| Relative Volume | Score |
|-----------------|-------|
| >= 3.0          | 100   |
| 2.0-3.0         | 80    |
| 1.5-2.0         | 60    |
| 1.0-1.5         | 40    |
| < 1.0           | 20    |

#### 1.3 Uptrend Ratio Score (weight: 25%)

Measures the percentage of stocks in the theme that are in technical uptrends.

**Formula:**
```
uptrend_score = normalize(uptrend_ratio, scale=[0%, 100%] -> [0, 100])

uptrend_ratio = count(stocks_in_uptrend) / count(total_stocks)
```

**Data Source:** uptrend-dashboard output (3-point evaluation: price > 50-day SMA, 50-day SMA > 200-day SMA, 200-day SMA rising)

**If uptrend data is unavailable:** Use price-above-SMA200 as proxy from FINVIZ data. Score is reduced by 20% confidence penalty.

**Scoring Scale:**
| Uptrend Ratio | Score |
|---------------|-------|
| >= 80%        | 100   |
| 60-80%        | 80    |
| 40-60%        | 60    |
| 20-40%        | 40    |
| < 20%         | 20    |

Note: For bearish themes, invert the ratio (use downtrend ratio instead).

#### 1.4 Breadth Score (weight: 20%)

Measures how broadly the theme is participating (not just a few large-cap names).

**Formula:**
```
breadth_score = normalize(participation_rate, scale=[0%, 100%] -> [0, 100])

participation_rate = count(stocks_moving_in_direction > 1%) / count(total_stocks)
```

**Data Source:** FINVIZ stock-level performance data

**Scoring Scale:**
| Participation Rate | Score |
|-------------------|-------|
| >= 80%            | 100   |
| 60-80%            | 80    |
| 40-60%            | 60    |
| 20-40%            | 40    |
| < 20%             | 20    |

### Theme Heat Composite

```
theme_heat = (momentum_score * 0.35) + (volume_score * 0.20) + (uptrend_score * 0.25) + (breadth_score * 0.20)
```

---

## Dimension 2: Lifecycle Maturity

Lifecycle assessment classifies a theme into one of five stages: **Emerging**, **Accelerating**, **Trending**, **Mature**, or **Exhausting**. This is critical for distinguishing emerging opportunities from crowded trades.

### Components

#### 2.1 Duration Score (weight: 25%)

How long the theme has been active (consecutive weeks of elevated heat).

**Measurement:** Count weeks where theme_heat >= 40.

| Duration | Stage Signal |
|----------|-------------|
| 1-3 weeks | Early |
| 4-8 weeks | Mid |
| 9-16 weeks | Late |
| > 16 weeks | Exhaustion |

**Limitation:** Duration tracking requires historical data. On first run, duration defaults to "Unknown" and lifecycle uses other factors only.

#### 2.2 Extremity Clustering Score (weight: 25%)

Percentage of stocks in the theme near 52-week highs or lows.

**Formula:**
```
extremity_pct = count(within_5pct_of_52wk_high_or_low) / count(total_stocks)
```

| Extremity % | Stage Signal |
|-------------|-------------|
| < 20%       | Early |
| 20-40%      | Mid |
| 40-60%      | Late |
| > 60%       | Exhaustion |

**Data Source:** FINVIZ 52-week high/low data

#### 2.3 Price Extreme Saturation Score (weight: 25%)

Proportion of stocks near 52-week extremes (within 5%).

**Formula:**
```
bullish: pct = count(dist_from_52w_high <= 0.05) / total
bearish: pct = count(dist_from_52w_low <= 0.05) / total
score = min(100, pct * 200)
```

#### 2.4 Valuation Score (weight: 15%)

Average P/E ratio of theme constituents relative to S&P 500 P/E.

**Formula:**
```
relative_pe = avg_theme_pe / sp500_pe
```

| Relative P/E | Stage Signal |
|---------------|-------------|
| < 0.8         | Early (undervalued) |
| 0.8-1.2       | Mid (fair value) |
| 1.2-2.0       | Late (overvalued) |
| > 2.0         | Exhaustion (extreme) |

**Data Source:** FMP API for P/E ratios (optional; uses FINVIZ forward P/E as fallback)

#### 2.5 ETF Proliferation Score (weight: 10%)

Number of thematic ETFs tracking the theme. More ETFs indicate greater retail/institutional attention.

**Source:** `thematic_etf_catalog.md` (static reference)

| ETF Count | Score | Stage Signal |
|-----------|-------|-------------|
| 0         | 0     | Very Early |
| 1         | 20    | Early |
| 2-3       | 40    | Mid |
| 4-6       | 60    | Mid-Late |
| 7-10      | 80    | Late |
| > 10      | 100   | Exhaustion |

### Lifecycle Maturity Composite

```
maturity = (duration * 0.25) + (extremity * 0.25) + (price_extreme * 0.25) + (valuation * 0.15) + (etf_proliferation * 0.10)
```

### Lifecycle Stage Classification

The lifecycle stage is classified from the maturity score:

| Maturity Score | Stage |
|----------------|-------|
| 0-20 | Emerging |
| 20-40 | Accelerating |
| 40-60 | Trending |
| 60-80 | Mature |
| 80-100 | Exhausting |

**Note:** Media/Narrative Saturation is not included in the automated maturity calculation. Claude's WebSearch narrative confirmation can be used to qualitatively adjust the lifecycle assessment.

---

## Dimension 3: Confidence (Low / Medium / High)

Confidence measures **how reliable** the theme detection is, based on data quality and confirmation signals.

### Layers

#### 3.1 Quantitative Layer (base)

Based on data breadth and consistency:

| Condition | Level |
|-----------|-------|
| >= 4 industries matching, >= 20 stocks analyzed | High |
| 2-3 industries matching, >= 10 stocks analyzed | Medium |
| 1 industry matching or < 10 stocks | Low |

#### 3.2 Breadth Layer (modifier)

Cross-sector participation adds confidence:

| Condition | Modifier |
|-----------|----------|
| Theme spans 3+ sectors | +1 level (cap at High) |
| Theme spans 2 sectors | No change |
| Theme in 1 sector only | -1 level (floor at Low) |

#### 3.3 Narrative Layer (modifier, applied in Step 4)

WebSearch confirmation adjusts confidence:

| Narrative Finding | Modifier |
|-------------------|----------|
| Strong confirmation (multiple sources, clear catalysts) | +1 level |
| Mixed signals | No change |
| Contradictory narrative (bearish articles for bullish theme) | -1 level |

### Final Confidence

```
confidence = apply_modifiers(quantitative_base, breadth_modifier, narrative_modifier)
confidence = clamp(confidence, Low, High)
```

---

## Direction Detection

Theme direction (**leading** vs. **lagging**) is determined by relative rank, not absolute price change:

### Algorithm

1. Each industry gets a `rank_direction` based on its position in the momentum-ranked list: top half = "bullish" (leading), bottom half = "bearish" (lagging).
2. Theme direction is the majority vote of its constituent industries' `rank_direction`.

```python
# Industry-level (industry_ranker.py)
rank_direction = "bullish" if rank <= len(industries) // 2 else "bearish"

# Theme-level (theme_classifier.py)
direction = majority_vote([ind.rank_direction for ind in theme.industries])
```

### Important: Relative, Not Absolute

**LEAD/LAG direction is relative.** A "lagging" theme may still have positive absolute returns — it simply underperforms relative to other themes. This means:
- **LEAD** themes: Suitable for overweight / new positions
- **LAG** themes: Candidates for underweight reduction, **not** short signals

---

## Data Sources

### Primary: FINVIZ

**Elite Mode (recommended):**
- CSV export endpoint: `https://elite.finviz.com/export.ashx?v=151&f=ind_{code},cap_smallover,...&auth=KEY`
- Provides: ticker, company, sector, industry, market cap, P/E, change%, volume, avg volume, 52wk high/low, RSI, SMA20/50/200
- Rate limit: 0.5s between requests
- Coverage: Full stock universe per industry

**Public Mode (fallback):**
- HTML scraping: `https://finviz.com/screener.ashx?v=151&f=ind_{code},cap_smallover`
- Provides: Same fields but limited to page 1 (~20 stocks per industry)
- Rate limit: 2.0s between requests (aggressive scraping may trigger blocks)
- Coverage: Top 20 stocks per industry by market cap

### Secondary: FMP API (optional)

- P/E ratios for valuation analysis
- Not required; FINVIZ forward P/E used as fallback
- Useful for more granular valuation metrics

### Tertiary: uptrend-dashboard (optional)

- CSV output from the uptrend-dashboard skill
- Provides 3-point technical evaluation per stock
- Significantly improves uptrend ratio accuracy
- If unavailable, FINVIZ price-vs-SMA200 is used as proxy

### Quaternary: WebSearch (narrative layer)

- Used for narrative confirmation in Step 4
- Not automated; Claude performs searches during workflow
- Subjective assessment of media coverage and analyst sentiment

---

## Integration with uptrend-dashboard

When uptrend-dashboard data is available, the theme detector uses it for enhanced 3-point evaluation:

**3-Point Evaluation Criteria:**
1. Price > 50-day SMA (short-term trend)
2. 50-day SMA > 200-day SMA (medium-term trend, golden/death cross)
3. 200-day SMA is rising (long-term trend confirmation)

**Stocks meeting all 3 points** = in confirmed uptrend
**Stocks meeting 0 points** = in confirmed downtrend

This provides more accurate uptrend ratios than simple price-above-SMA200 proxy.

---

## Output Schema

### JSON Output Structure

```json
{
  "metadata": {
    "date": "2026-02-16",
    "mode": "elite",
    "themes_analyzed": 14,
    "industries_scanned": 145,
    "total_stocks": 5200,
    "uptrend_data_available": true,
    "execution_time_seconds": 150
  },
  "themes": [
    {
      "name": "AI & Semiconductors",
      "direction": "Bullish",
      "direction_strength": "Strong",
      "theme_heat": 82,
      "heat_components": {
        "momentum": 90,
        "volume": 75,
        "uptrend": 85,
        "breadth": 70
      },
      "lifecycle": {
        "stage": "Late",
        "duration_weeks": 12,
        "extremity_pct": 45,
        "relative_pe": 1.8,
        "etf_proliferation_score": 100,
        "etf_count": 11
      },
      "confidence": "High",
      "confidence_components": {
        "quantitative": "High",
        "breadth_modifier": "+1",
        "narrative_modifier": "pending"
      },
      "industries": [
        {
          "name": "Semiconductors",
          "change_pct": 4.2,
          "avg_relative_volume": 1.8,
          "uptrend_ratio": 0.75,
          "stock_count": 35
        }
      ],
      "top_stocks": [
        {"ticker": "NVDA", "change_pct": 6.5, "relative_volume": 2.1},
        {"ticker": "AVGO", "change_pct": 4.8, "relative_volume": 1.9}
      ],
      "proxy_etfs": ["SMH", "SOXX", "AIQ", "BOTZ"]
    }
  ],
  "industry_rankings": {
    "top_10": [...],
    "bottom_10": [...]
  },
  "sector_summary": {
    "Technology": {"uptrend_ratio": 0.65, "avg_change_pct": 2.1},
    "Energy": {"uptrend_ratio": 0.40, "avg_change_pct": -1.5}
  }
}
```

---

## Known Limitations

1. **Static theme definitions**: Cross-sector themes are predefined in `cross_sector_themes.md`. New themes (e.g., a sudden meme-stock theme) are not automatically detected.

2. **Industry granularity**: FINVIZ industry classification may not perfectly map to investment themes. Some industries span multiple themes.

3. **Single-stock dominance**: Large-cap stocks (e.g., NVDA in AI) can skew theme-level metrics. Market-cap weighting amplifies this effect.

4. **Temporal lag**: Weekly performance data does not capture intraday or same-day momentum shifts.

5. **ETF catalog staleness**: The thematic ETF catalog is manually maintained and may not reflect recent ETF launches or closures.

6. **Public mode data limits**: Only ~20 stocks per industry are captured, which may underrepresent small/mid-cap participation.

7. **Duration tracking**: First-run analysis cannot assess theme duration without historical baseline data.

8. **Narrative subjectivity**: Confidence adjustment from WebSearch is inherently subjective and depends on search result quality.

9. **Survivorship bias**: Analysis only covers currently listed stocks and active ETFs, missing delisted or closed instruments.

10. **FINVIZ data delays**: Public FINVIZ data is 15-minute delayed; Elite provides real-time during market hours.