` with empty body)
- Articles are visible in `view-source:`
**Reference generator:** `feed_generators/ollama_blog.py` (simplest), `feed_generators/blogsurgeai_feed_generator.py` (more complete), `feed_generators/paulgraham_blog.py`
Use `type: requests` in `feeds.yaml`. Proceed to Step 1.
---
#### Case D: Dynamic site (JavaScript-rendered content)
Signals that Selenium is required:
- `curl`/`requests` returns a near-empty body or a loading spinner
- HTML contains `
`, `
`, or similar SPA shell
- Content only appears after JS execution
**Reference generators:** `feed_generators/xainews_blog.py` (Selenium + cache), `feed_generators/anthropic_news_blog.py` (Selenium + cache + incremental), `feed_generators/mistral_blog.py`
Use `type: selenium` in `feeds.yaml`. Proceed to Step 1.
---
### Step 1: Review Existing Feed Generators
**Always read the reference generator(s) for your case before writing any code:**
```bash
# For static sites
cat feed_generators/ollama_blog.py
cat feed_generators/blogsurgeai_feed_generator.py
# For dynamic/Selenium sites
cat feed_generators/xainews_blog.py
cat feed_generators/anthropic_news_blog.py
```
Study these to understand:
- Import structure and shared `utils` helpers
- `FEED_NAME` and `BLOG_URL` constants
- Date parsing patterns and fallback chains
- Article extraction logic and CSS selectors
- Cache + incremental update pattern (Selenium generators)
- Error handling approaches
### Step 2: Analyze the Blog Source
1. **Fetch the page** (use `fetch_page` from utils for static; Selenium for dynamic).
2. **Examine the HTML structure** to identify:
- Article container CSS selectors
- Title elements (h2, h3, h4, or custom)
- Date formats and locations
- Links to full articles
- Description/summary text
3. **Handle access issues**:
- If the site blocks automated requests (403/429), work with a local HTML file first
- The user can provide HTML via browser's "Save Page As"
- Support both local file and web fetching modes in the final script
### Step 3: Create the Feed Generator Script
Create `feed_generators/_blog.py` following the reference for your case.
**Naming conventions:**
- Script: `feed_generators/{site_name}_blog.py` (e.g. `acme_blog.py`)
- Feed output: `feeds/feed_{site_name}.xml` (e.g. `feed_acme.xml`)
- `FEED_NAME` constant: `"{site_name}"` (e.g. `"acme"`)
**Required for all generators:**
- `FEED_NAME` and `BLOG_URL` constants at module level
- `setup_logging()` from utils
- Robust date parsing with multiple format fallback (see `xainews_blog.py`)
- Article deduplication (track seen links with a set)
- Per-article error handling: log warning and continue, never crash the full run
- Articles sorted newest-first before feed generation
**Additional requirements for Selenium generators:**
- Use `setup_selenium_driver()` from utils
- Use `load_cache()` / `save_cache()` / `merge_entries()` from utils for incremental updates
- Support `--full` flag via `argparse` for full-reset runs (see `anthropic_news_blog.py`)
- Use `sort_posts_for_feed()` from utils
See [Reference Examples by Type](#reference-examples-by-type) for full structural details.
### Step 4: Update feeds.yaml
Add an entry to `feeds.yaml` in alphabetical order by key:
**For static (requests) sites:**
```yaml
site_name:
script: site_name_blog.py
type: requests
blog_url: https://example.com/blog
```
**For dynamic (Selenium) sites:**
```yaml
site_name:
script: site_name_blog.py
type: selenium
blog_url: https://example.com/blog
```
### Step 5: Add Makefile Target
Add targets to `makefiles/feeds.mk` in alphabetical order.
**For static (requests) sites:**
```makefile
.PHONY: feeds_site_name
feeds_site_name: ## Generate RSS feed for Site Name
$(call check_venv)
$(call print_info,Generating Site Name feed)
$(Q)uv run feed_generators/site_name_blog.py
$(call print_success,Site Name feed generated)
```
**For dynamic (Selenium) sites — always include both incremental and full-reset targets:**
```makefile
.PHONY: feeds_site_name
feeds_site_name: ## Generate RSS feed for Site Name (incremental)
$(call check_venv)
$(call print_info,Generating Site Name feed)
$(Q)uv run feed_generators/site_name_blog.py
$(call print_success,Site Name feed generated)
.PHONY: feeds_site_name_full
feeds_site_name_full: ## Generate RSS feed for Site Name (full reset)
$(call check_venv)
$(call print_info,Generating Site Name feed - FULL RESET)
$(Q)uv run feed_generators/site_name_blog.py --full
$(call print_success,Site Name feed generated - full reset)
```
### Step 6: Update README
Add a row to the table in `README.md` in **alphabetical order** by blog name.
**For scraped feeds** (Cases C and D):
```markdown
| [Site Name](https://example.com/blog) | [feed_site_name.xml](https://raw.githubusercontent.com/Olshansk/rss-feeds/main/feeds/feed_site_name.xml) |
```
**For native/official feeds** (Cases A and B):
```markdown
| [Site Name](https://example.com) | [Official RSS](https://example.com/feed.xml) |
```
The raw GitHub URL format must be exactly:
`https://raw.githubusercontent.com/Olshansk/rss-feeds/main/feeds/feed_{name}.xml`
### Step 7: Test and Verify
**Run the generator:**
```bash
# Static sites
uv run feed_generators/site_name_blog.py
# Dynamic sites (incremental)
uv run feed_generators/site_name_blog.py
# Dynamic sites (full reset)
uv run feed_generators/site_name_blog.py --full
```
**Verify output:**
```bash
ls -la feeds/feed_site_name.xml
head -50 feeds/feed_site_name.xml
```
**Validate the feed:**
```bash
uv run feed_generators/validate_feeds.py
```
**Run via Makefile:**
```bash
make feeds_site_name
```
**Integration checklist before declaring done:**
- [ ] Script follows naming pattern: `feed_generators/{name}_blog.py`
- [ ] Output file follows pattern: `feeds/feed_{name}.xml`
- [ ] Entry added to `feeds.yaml` with correct `type`
- [ ] Makefile target(s) added to `makefiles/feeds.mk` (Selenium: both incremental + `_full`)
- [ ] README row added in alphabetical order with correct raw GitHub URL
- [ ] `validate_feeds.py` passes with no errors
- [ ] Articles are sorted newest-first
- [ ] Duplicate articles are filtered out
- [ ] Individual article failures are caught and logged (don't crash the run)
## Reference Examples by Type
### Type 1: Static (requests + BeautifulSoup)
**Simplest:** `feed_generators/ollama_blog.py`
- Minimal imports, straightforward `fetch_page` + BeautifulSoup
- Good starting point when the HTML structure is clean
**More complete:** `feed_generators/blogsurgeai_feed_generator.py`
- `fetch_page` + BeautifulSoup + `dateutil.parser`
- Better date handling, good error patterns
**Complex static with local-file fallback:** `feed_generators/paulgraham_blog.py`
### Type 2: Dynamic (Selenium + cache)
**Selenium + cache, no local-file fallback:** `feed_generators/mistral_blog.py`
- Minimal Selenium setup
- Good for simple JS-rendered pages
**Selenium + cache + incremental + argparse:** `feed_generators/xainews_blog.py`
- Full incremental update pattern with `--full` reset flag
- Use this as the base template for most Selenium generators
**Selenium + cache + incremental + multiple entry points:** `feed_generators/anthropic_news_blog.py`
- Same as xainews but handles multiple sections from one site
- Reference when a single domain has multiple feeds (e.g. `/news`, `/research`, `/engineering`)
### Type 3: Multiple feeds from one site
**Reference:** `feed_generators/anthropic_eng_blog.py`, `feed_generators/anthropic_research_blog.py`
- Each section gets its own `FEED_NAME` and script
- Share the Selenium driver setup pattern
- Add separate `feeds.yaml` entries and Makefile targets per feed
## Common Patterns
### Official RSS Detection (Case B — run before writing any code)
```python
import requests
from bs4 import BeautifulSoup
def check_native_feed(url):
resp = requests.get(url, timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")
link = soup.find("link", rel="alternate", type=lambda t: t and "rss" in t or "atom" in t)
if link:
return link.get("href")
# Try common paths
for path in ["/feed", "/rss.xml", "/atom.xml", "/feed.xml", "/rss"]:
probe = requests.head(url.rstrip("/") + path, timeout=5)
if probe.status_code == 200:
return url.rstrip("/") + path
return None
```
### Incremental Updates (Selenium generators)
See `feed_generators/anthropic_news_blog.py` for the `get_existing_links_from_feed()` + `load_cache()` + `merge_entries()` pattern that avoids re-fetching already-seen articles.
### Robust Date Parsing
```python
DATE_FORMATS = [
"%B %d, %Y", # January 15, 2024
"%b %d, %Y", # Jan 15, 2024
"%Y-%m-%d", # 2024-01-15
"%d %B %Y", # 15 January 2024
"%B %Y", # January 2024
]
def parse_date(date_text):
for fmt in DATE_FORMATS:
with contextlib.suppress(ValueError):
return datetime.strptime(date_text.strip(), fmt).replace(tzinfo=pytz.UTC)
return stable_fallback_date() # from utils
```
### Local File Fallback (for blocked sites)
```python
import argparse, sys
def main():
parser = argparse.ArgumentParser()
parser.add_argument("html_file", nargs="?", help="Local HTML file (optional)")
args = parser.parse_args()
if args.html_file:
with open(args.html_file) as f:
html = f.read()
else:
html = fetch_page(BLOG_URL)
...
```
## Troubleshooting
### No articles found
- Verify CSS selectors match actual HTML structure
- Check if content is dynamically loaded → switch to Selenium (Case D)
- Add debug logging to show what selectors find
### Date parsing failures
- Add the specific format to `DATE_FORMATS` list
- Use `stable_fallback_date()` from utils as the final fallback
### Blocked requests (403/429 errors)
- Save page locally with browser "Save Page As"
- Use local file mode for development
- Try different `User-Agent` headers in `fetch_page`
- If consistently blocked, switch to Selenium (Case D)
---
> Source: [Olshansk/rss-feeds](https://github.com/Olshansk/rss-feeds) — distributed by [TomeVault](https://tomevault.io).