tools-only ARD: Aggregating Crypto News
Document Control
| Field | Value |
|---|---|
| Skill Name | aggregating-crypto-news |
| Architecture Pattern | Data Aggregation Pipeline |
| Version | 2.0.0 |
| Author | Jeremy Longshore jeremy@intentsolutions.io |
Architectural Overview
Pattern: Multi-Source Data Aggregation
This skill implements a parallel fetching, aggregation, and filtering pipeline for RSS-based news sources.
┌─────────────────────────────────────────────────────────────────────────────┐
│ CRYPTO NEWS AGGREGATOR ARCHITECTURE │
└─────────────────────────────────────────────────────────────────────────────┘
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ RSS Source 1 │ │ RSS Source 2 │ │ RSS Source N │
│ (CoinDesk) │ │ (CoinTelegraph) │ │ (The Block) │
└────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘
│ │ │
└────────────────────────┼────────────────────────┘
│
▼
┌─────────────────────────┐
│ FEED FETCHER │
│ (Parallel Requests) │
│ - Timeout handling │
│ - Caching layer │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ FEED PARSER │
│ - XML/RSS parsing │
│ - Date normalization │
│ - Content extraction │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ DEDUPLICATOR │
│ - Title similarity │
│ - URL matching │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ SCORER │
│ - Keyword matching │
│ - Market-moving boost │
│ - Source quality │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ FILTER ENGINE │
│ - Time window │
│ - Coin matching │
│ - Category filtering │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ FORMATTER │
│ - Table output │
│ - JSON export │
│ - CSV export │
└─────────────────────────┘
Workflow
- Fetch: Parallel HTTP requests to RSS feeds with timeout handling
- Parse: Extract title, content, date, source from XML
- Dedupe: Remove duplicate articles based on title similarity
- Score: Calculate relevance score based on keywords and source
- Filter: Apply user filters (time, coin, category)
- Format: Output in requested format
Progressive Disclosure Strategy
Level 1: Simple Scan (Default)
python news_aggregator.py
Returns top 20 news from past 24h, relevance sorted.
Level 2: Filtered Scan
python news_aggregator.py --coin BTC --period 4h --top 10
Adds coin and time filtering.
Level 3: Category + Export
python news_aggregator.py --category defi --format json --output news.json
Full filtering with export.
Level 4: Advanced Configuration
python news_aggregator.py --sources coindesk,theblock --min-score 50 --verbose
Source selection and score thresholds.
Tool Permission Strategy
Allowed Tools (Scoped)
allowed-tools: Read, Bash(crypto:news-*)
| Tool | Scope | Purpose |
|---|---|---|
| Read | Unrestricted | Read config, sources list |
| Bash | crypto:news-* |
Execute news aggregation scripts |
Why These Tools
- Read: Load source configuration and settings
- Bash(crypto:news-*): Execute Python scripts for fetching and parsing
- No Write: Aggregation is read-only; exports use script file output
Directory Structure
plugins/crypto/crypto-news-aggregator/
└── skills/
└── aggregating-crypto-news/
├── PRD.md # Product requirements
├── ARD.md # This file
├── SKILL.md # Core instructions
├── scripts/
│ ├── news_aggregator.py # Main CLI entry point
│ ├── feed_fetcher.py # Parallel RSS fetching
│ ├── feed_parser.py # XML parsing and normalization
│ ├── scorer.py # Relevance scoring
│ └── formatters.py # Output formatting
├── references/
│ ├── errors.md # Error handling guide
│ └── examples.md # Usage examples
└── config/
├── settings.yaml # Configuration options
└── sources.yaml # RSS source registry
Data Flow Architecture
Input
- User request with optional filters (--coin, --period, --category)
- RSS feed URLs from source registry
Processing Pipeline
Input Request
│
├──► Load sources.yaml
│ │
│ ▼
├──► Parallel fetch (ThreadPoolExecutor)
│ │
│ ▼
├──► Parse each feed (feedparser library)
│ │
│ ▼
├──► Normalize entries to common schema:
│ {
│ "title": str,
│ "url": str,
│ "source": str,
│ "published": datetime,
│ "summary": str,
│ "category": str,
│ "relevance_score": float
│ }
│ │
│ ▼
├──► Deduplicate by title similarity (>80% match)
│ │
│ ▼
├──► Score relevance:
│ - Base score from source quality (1-10)
│ - Keyword boost (+10 per market-moving term)
│ - Recency boost (newer = higher)
│ │
│ ▼
├──► Apply filters:
│ - Time window (published > now - period)
│ - Coin match (title/summary contains symbol)
│ - Category match (source category or keywords)
│ │
│ ▼
└──► Format and output
Output Schema
{
"articles": [
{
"rank": 1,
"title": "Bitcoin Hits New All-Time High",
"url": "https://coindesk.com/...",
"source": "CoinDesk",
"published": "2026-01-14T10:30:00Z",
"age": "2h ago",
"category": "market",
"relevance_score": 85.5,
"coins_mentioned": ["BTC"]
}
],
"meta": {
"total_fetched": 500,
"after_dedup": 380,
"after_filter": 25,
"sources_used": 12,
"period": "24h",
"filters": {"category": "market"}
}
}
Error Handling Strategy
Error Categories
| Category | Examples | Strategy |
|---|---|---|
| Network | Timeout, DNS failure | Skip source, continue with others |
| Parse | Malformed XML, missing fields | Skip entry, log warning |
| Filter | Invalid coin symbol | Warn and proceed with partial filter |
| Export | File permission denied | Error with clear message |
Graceful Degradation
Full Success (all sources)
│
▼
Partial Success (some sources failed)
│ → Log warnings, return available data
│
▼
Cached Fallback (all sources failed)
│ → Return cached data with stale warning
│
▼
Complete Failure
│ → Clear error message with remediation
Composability & Stacking
As a Standalone Skill
- Fully functional for news aggregation
- No dependencies on other skills
As Data Provider
Skills that can consume news data:
- market-sentiment-analyzer: Feed articles for sentiment scoring
- crypto-signal-generator: Use news as signal input
- whale-alert-monitor: Correlate whale activity with news
Integration Pattern
# In another skill's script:
from pathlib import Path
import subprocess
import json
def get_news_for_coin(coin: str, period: str = "4h") -> list:
"""Fetch news using crypto-news-aggregator skill."""
news_script = Path(__file__).parent.parent.parent.parent / \
"crypto-news-aggregator/skills/aggregating-crypto-news/scripts/news_aggregator.py"
result = subprocess.run(
["python3", str(news_script), "--coin", coin, "--period", period, "--format", "json"],
capture_output=True, text=True
)
if result.returncode == 0:
return json.loads(result.stdout).get("articles", [])
return []
Performance & Scalability
Performance Targets
| Metric | Target | Approach |
|---|---|---|
| Fetch time (50 sources) | < 10s | Parallel fetching with ThreadPoolExecutor |
| Parse time (500 articles) | < 2s | Efficient XML parsing with feedparser |
| Memory usage | < 100MB | Stream processing, no full corpus in memory |
Optimization Strategies
- Parallel Fetching: Use ThreadPoolExecutor(max_workers=10)
- Caching: Cache responses with 5-minute TTL
- Lazy Parsing: Parse only after filter candidates identified
- Early Exit: Stop processing when --top limit reached
Scalability Considerations
- More Sources: Linear scaling with parallel fetch
- Higher Volume: Pagination and streaming for large exports
- Frequency: Caching prevents redundant fetches
Testing Strategy
Unit Tests
| Component | Test Cases |
|---|---|
| FeedFetcher | Timeout handling, cache hit/miss, parallel execution |
| FeedParser | Valid RSS, malformed XML, missing fields |
| Scorer | Keyword matching, source quality, recency boost |
| Formatters | Table, JSON, CSV output validation |
Integration Tests
- End-to-end with mock RSS server
- Real feeds (selected stable sources)
- Export file validation
Manual Testing
# Default scan
python news_aggregator.py
# Coin filter
python news_aggregator.py --coin BTC
# Category filter
python news_aggregator.py --category defi
# Export
python news_aggregator.py --format json --output test.json
Security & Compliance
Security Considerations
- No Authentication: RSS feeds are public, no credentials needed
- Input Validation: Sanitize coin symbols and category names
- Output Sanitization: Escape HTML in article titles/summaries
Data Privacy
- No user data collected or stored
- No tracking or analytics
- Cached data is ephemeral (5-minute TTL)
Rate Limiting
- Respect source rate limits (1 req/source/5min default)
- No aggressive crawling
- Caching reduces request volume