tools-only Web Archive Analysis Skill
Purpose
Query the Wayback Machine to discover historical technology usage and detect technology migrations over time.
Operations
1. query_cdx_api
Get historical snapshots from the Wayback Machine CDX API.
Endpoint:
GET http://web.archive.org/cdx/search/cdx
Parameters:
url: {domain}
output: json
filter: statuscode:200
collapse: timestamp:6 # Group by month (YYYYMM)
limit: 100
from: {start_year}
to: {end_year}
Example Request:
curl "http://web.archive.org/cdx/search/cdx?url=example.com&output=json&filter=statuscode:200&collapse=timestamp:6&limit=100"
Response Format:
[
["urlkey", "timestamp", "original", "mimetype", "statuscode", "digest", "length"],
["com,example)/", "20240115120000", "https://example.com/", "text/html", "200", "ABC123...", "45678"]
]
2. select_snapshots
Choose representative snapshots for analysis.
Selection Strategy:
def select_snapshots(all_snapshots):
# Get snapshots at regular intervals
intervals = [
"6 months ago",
"1 year ago",
"2 years ago",
"3 years ago",
"5 years ago"
]
selected = []
for interval in intervals:
target_date = calculate_date(interval)
closest = find_closest_snapshot(all_snapshots, target_date)
if closest:
selected.append(closest)
return selected
Snapshot Priority:
- Recent (baseline for comparison)
- 1 year ago (detect recent changes)
- 2-3 years ago (medium-term evolution)
- 5+ years ago (historical context)
3. fetch_archived_content
Retrieve archived pages for analysis.
Wayback URL Format:
https://web.archive.org/web/{timestamp}/{original_url}
Example:
https://web.archive.org/web/20230115120000/https://example.com/
Headers to Request:
Accept: text/html
User-Agent: TechStackAgent/1.0 (OSINT research)
4. compare_snapshots
Detect technology changes between snapshots.
Comparison Points:
{
"headers_to_compare": [
"Server",
"X-Powered-By",
"Set-Cookie"
],
"html_elements": [
"meta[name=generator]",
"script[src]",
"link[href]"
],
"patterns_to_track": [
"/wp-content/",
"/_next/",
"/_nuxt/",
"/static/js/"
]
}
Change Detection:
def detect_changes(old_snapshot, new_snapshot):
changes = []
# Compare technologies
old_tech = extract_technologies(old_snapshot)
new_tech = extract_technologies(new_snapshot)
added = new_tech - old_tech
removed = old_tech - new_tech
for tech in added:
changes.append({
"type": "technology_added",
"technology": tech,
"first_seen": new_snapshot.timestamp
})
for tech in removed:
changes.append({
"type": "technology_removed",
"technology": tech,
"last_seen": old_snapshot.timestamp
})
return changes
5. detect_migrations
Identify framework/platform migrations.
Common Migration Patterns:
{
"WordPress → Custom/React": {
"indicators": [
"/wp-content/ disappears",
"React globals appear",
"/_next/ or /static/js/ paths"
],
"typical_timeline": "6-18 months"
},
"AngularJS → Angular": {
"indicators": [
"ng-app disappears",
"ng-version appears",
"Angular 2+ patterns"
],
"typical_timeline": "12-24 months"
},
"jQuery → React/Vue": {
"indicators": [
"jQuery CDN removed",
"Modern framework globals",
"SPA patterns"
],
"typical_timeline": "6-12 months"
},
"On-prem → Cloud": {
"indicators": [
"CloudFront/Cloudflare headers appear",
"AWS/GCP/Azure signatures",
"CDN usage"
],
"typical_timeline": "3-12 months"
}
}
6. extract_historical_tech
Parse archived HTML for technology signals.
Process:
- Fetch archived page
- Apply same analysis as html_content_analysis skill
- Record technologies with timestamp
- Build timeline of technology usage
Output
{
"skill": "web_archive_analysis",
"domain": "string",
"results": {
"archive_coverage": {
"oldest_snapshot": "2015-03-15",
"newest_snapshot": "2024-01-10",
"total_snapshots": 450,
"snapshots_analyzed": 5
},
"snapshots_analyzed": [
{
"timestamp": "2024-01-10",
"url": "https://web.archive.org/web/20240110/...",
"technologies_detected": ["Next.js", "React", "Vercel"]
},
{
"timestamp": "2022-06-15",
"url": "https://web.archive.org/web/20220615/...",
"technologies_detected": ["React", "Create React App", "Heroku"]
},
{
"timestamp": "2020-01-20",
"url": "https://web.archive.org/web/20200120/...",
"technologies_detected": ["WordPress", "PHP"]
}
],
"technology_timeline": [
{
"technology": "WordPress",
"first_seen": "2015-03-15",
"last_seen": "2020-06-01",
"status": "removed"
},
{
"technology": "React",
"first_seen": "2020-03-01",
"last_seen": "present",
"status": "current"
},
{
"technology": "Next.js",
"first_seen": "2023-01-15",
"last_seen": "present",
"status": "current"
}
],
"migrations_detected": [
{
"type": "CMS → Modern Framework",
"from": "WordPress",
"to": "React/Next.js",
"approximate_date": "2020-Q1 to 2020-Q2",
"confidence": 85
},
{
"type": "Hosting Migration",
"from": "Heroku",
"to": "Vercel",
"approximate_date": "2023-Q1",
"confidence": 80
}
],
"current_vs_historical": {
"current_stack": ["Next.js", "React", "Vercel"],
"historical_stack": ["WordPress", "PHP", "Heroku"],
"major_changes": 2
}
},
"evidence": [
{
"type": "archived_snapshot",
"timestamp": "string",
"archive_url": "string",
"technologies": ["array"],
"analysis_timestamp": "ISO-8601"
}
]
}
Rate Limiting
- Wayback CDX API: 15 requests/minute
- Archived page fetches: 10/minute
- Cache CDX results to avoid repeated queries
Error Handling
- 404: Domain not archived
- 503: Wayback Machine overloaded - retry with backoff
- Timeout: Increase timeout for archived pages (can be slow)
- Continue with available snapshots on partial failures
Security Considerations
- Only access public archives
- Respect Wayback Machine rate limits
- Do not store archived content beyond analysis
- Note that archived content may contain outdated security vulnerabilities
- Log all queries for audit
Confidence Notes
Historical data provides contextual signals:
- Confirms technology transitions
- Validates current technology choices
- Lower weight than current direct evidence
- Base confidence: 60-75%