# Data Base

> Data acquisition for web scraping and data collection. Use when user needs "爬取数据/抓取网页/scrape data". Outputs structured JSON/CSV for analysis.

- Skill: `jinfanzheng/data-base` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add jinfanzheng/data-base`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jinfanzheng/data-base/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: jinfanzheng (https://skillmd.com/u/jinfanzheng)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jinfanzheng/data-base

---


## Mental Model

Data acquisition is **converting unstructured web content into structured data**. Choose tool based on page complexity: JS-heavy → chrome-devtools MCP, static → Python requests.

## Tool Selection

| Page Type | Tool | When to Use |
|-----------|------|-------------|
| Dynamic (JS-rendered, SPAs) | chrome-devtools MCP | React/Vue apps, infinite scroll, login gates |
| Static HTML | Python requests | Blogs, news sites, simple pages |
| Complex/reusable logic | Python script | Multi-step scraping, rate limiting, proxies |

## Anti-Patterns (NEVER)

- Don't scrape without checking robots.txt
- Don't overload servers (default: 1 req/sec)
- Don't scrape personal data without consent
- Don't use Chinese characters in output filenames (ASCII only)
- Don't forget to identify bot with User-Agent

## Output Format

- **JSON**: Nested/hierarchical data
- **CSV**: Tabular data
- Filename: `{source}_{timestamp}.{ext}` (ASCII only, e.g., `news_20250115.csv`)

## Workflow

1. **Ask**: What data? Which sites? How much?
2. **Select tool** based on page type
3. **Extract** and save structured data
4. **Deliver** file path to user or pass to data-analysis

## Python Environment

**Auto-initialize virtual environment if needed, then execute:**

```bash
cd skills/data-base

if [ ! -f ".venv/bin/python" ]; then
    echo "Creating Python environment..."
    ./setup.sh
fi

.venv/bin/python your_script.py
```

The setup script auto-installs: requests, beautifulsoup4, pandas, web scraping tools.

## References (load on demand)

For detailed APIs and templates, load: `references/REFERENCE.md`, `references/templates.md`