AI Training Platform
LLM Backend
- v5.0+: 公司 LLM Gateway (
http://127.0.0.1:18888/openai/v1) → DeepSeek V4 Pro - v4.x: 火山引擎直连 (
https://ark.cn-beijing.volces.com/api/v3) → SECRET - Gateway 不需要真实 API key,占位符
gateway-local-no-key-required即可 - config.py:
ark_base_url/ark_model_id/ark_api_key均指向 Gateway - timeout 提升到 120s 适配推理模型
Pitfall: DeepSeek V4 Pro 推理模型会截断 content(2026-08-16 验证)
DeepSeek V4 Pro(及其他 reasoning 模型)先输出 reasoning_content(思考过程)再输出 content(最终答案),两者共用 max_tokens。max_tokens=1200 时推理过程会吃掉大部分 token,导致最终正文被截成半句话(如结尾停在"……的关键")。
- 症状:LLM 返回的正文明显没写完,
finish_reason=stop但 content 不完整 - 诊断:看返回的
message有content+reasoning_content两个字段;usage.completion_tokens_details.reasoning_tokens就是被推理吃掉的 token 数(实测 67/90 都给了推理) - 修复:批量生成长文(教程正文、路径介绍等)时
max_tokens调到 6000,并加"content 长度 < 200 则判失败重试"的校验,防止把半截内容写进库
Quick Start
项目别名: 用户口中的「知渔」=「KnowHow知渔」= 本平台(品牌名 v5.x 起为「KnowHow知渔 · ai学习平台」)。 当前主目录:
/Users/hua/6-产品研发/渔芯社区/知渔/— 2026-10-30 实测:旧主目录ok-KnowHow知渔/与23-ai培训教程/均已不存在,项目已迁入渔芯社区工作区并升到 v5.5.0(Docker/Makefile 启动、pytest 560 passed,见该目录 CLAUDE.md;make run默认 127.0.0.1:8520)。本 skill 以下正文的旧路径均为历史口径,动手前一律以磁盘现状为准(沿下方 DB 多候选纪律 find 重定位)。
cd /Users/hua/6-产品研发/ok-KnowHow知渔
# 启动(推荐:项目自带脚本,后台运行,PID 写 logs/server.pid,日志 logs/server.stdout.log)
./start.sh --daemon # 后台 → http://127.0.0.1:8520
./start.sh # 前台 uvicorn
./start.sh --status # 查看运行状态
./start.sh --stop # 停止
# 健康检查
curl -s http://127.0.0.1:8520/api/ops/health # → {"ok":true,"version":"5.3.0",...}
# 直接跑后端(不推荐,绕过脚本)
python3 server/app.py # port 8520
# Re-seed DB
python3 server/seed.py
# Re-seed content
python3 server/seed_content.py
给用户打开前端测试: 后台启动后用 open http://127.0.0.1:8520 在默认浏览器打开 + browser_navigate 自验页面加载(首页导航 + 登录/注册界面)。
LLM Provider Configuration (v5.0+)
The platform routes all LLM calls through the company LLM Gateway at http://127.0.0.1:18888, not directly to volcengine.
Config locations that matter for provider switches:
| File | Field | Gateway Value |
|---|---|---|
server/config.py |
ark_base_url |
http://127.0.0.1:18888/openai/v1 |
server/config.py |
ark_model_id |
deepseek-v4-pro |
server/config.py |
ark_api_key |
"gateway-local-no-key-required" (placeholder — Gateway doesn't validate) |
server/config.py |
ark_timeout |
120 (reasoning models need more time) |
server/services/llm.py |
(reads from settings) | No hardcoded values — uses settings.* dynamically |
Pitfall: The LLMClient.available property checks bool(self.api_key). Always set a non-empty placeholder even when using Gateway (it doesn't validate keys but the client won't send requests without one).
Codex timeout warning: When using codex exec to work on this project, Codex may time out trying to fetch model metadata from the Gateway. Fall back to direct edits (terminal + patch tools) if Codex hangs.
See references/gateway-config.md for full Gateway behavior notes (reasoning model SSE format, model name aliases, switch script).
Architecture
FastAPI backend (server/app.py, port 8520)
├── LLM Gateway (127.0.0.1:18888 → DeepSeek V4 Pro)
├── SQLite (db/ai_learning.db)
│ ├── terms (206 cards, depth/parent_id for sub-cards)
│ ├── term_content (detail_md/examples/common_mistakes/related_terms)
│ ├── career_paths → path_phases → path_terms
│ ├── learning_log (SM-2 spaced repetition)
│ └── ai_figures, ai_industry_tree, ai_creators
└── Static files (web/index.html)
└── Single-page vanilla JS app, no framework
UI Layout — 左侧 Sidebar(渔芯前端参考标杆)
知渔前端是左侧 sidebar 布局,不是顶部横排导航栏(华哥 2026-08-16 明确纠正过):
<aside class="sidebar">左侧栏:sidebar-logo+sidebar-nav(闪卡复习/学习卡片/阶段测验等入口,用data-p属性标识)+sidebar-footer- 移动端才收成 ☰ 汉堡按钮(
menu-btn→toggleSidebar())+ 底部bottom-nav - 内容区在右侧,靠 JS 切换页面视图
华哥设计规范:任何"参考知渔框架"做的渔芯产品(如初创者 33-初创者),布局必须用左侧导航栏,不要做顶部横排导航。做前端改造前先确认标杆产品的真实布局(grep -nE "sidebar|<aside|<nav" web/index.html),别凭印象猜。
"路径 + 卡片"形式可复用:知渔的内容组织 = 学习路径(career_paths → path_phases → path_terms,即"阶段(phase) → 卡片链(cards)")+ 名词卡片(terms,统一卡片含 icon/名称/描述/难度/标签)。其他渔芯产品的结构化内容可套用同一形式,实测映射(初创者 2026-08-16):
- "案例" → 用路径式展示:案例详情 = 阶段时间线(原场景 → AI方案 → 可复制SOP → 效果数据),SOP 每步 = 一张带编号圆圈的卡片
- "行业方案/新360行" → 用卡片网格展示:每张卡 = icon + 名称 + 摘要 + 痛点 + 关联案例数,
grid-template-columns: repeat(auto-fill, minmax(240px,1fr))
Key APIs
GET /api/terms— list with depth + child_countGET /api/terms/{id}— full card with content + children + parentGET /api/terms/{id}/children— sub-card listGET /api/terms/{id}/depth-tree— recursive treePOST /api/terms/{id}/content— update detail contentGET /api/paths/{name}— learning path with phasesPOST /api/learn— SM-2 spaced repetitionGET /api/dashboard— learning dashboard (progress, mastery, streak, weak areas)GET /api/recommend?limit=10— smart recommendations (due review + weak terms + new cards)GET /api/wrong-book— weak area analysis based on low quiz scoresGET /api/quiz/{career}/{phase}?mode=normal— generate quizmode=normal— 5 questions (default)mode=review— questions from weak (score < 60) phasesmode=intense— 10 questions for intensive practice
v4.6 Data Flywheel Effect APIs
Crowd wisdom injection and trending topic leaderboard:
POST /api/generate/matrix(UPDATED in v4.6) — generates with crowd-sourced reference injection- Before generating, queries existing terms table for related cards (LIKE match on topic)
- Injects up to 15 existing cards into the LLM prompt as "crowd wisdom reference"
- LLM is instructed to: (1) fill blind spots (2) deepen existing content (3) fill in missing details
- Flywheel effect: more users → more reference cards → better generated content → more users
- Result: Quality increases non-linearly with user base
GET /api/generate/trending— get hot topic leaderboard- Returns:
{trending: [{name, card_count, total_learns, learner_count}], total_cards, total_groups} - Grouped by
group_name, ordered bytotal_learns DESC - Card count shown in UI as
(${card_count})next to each topic tag - Click handler calls
quickGen(topic)which auto-fills the input and triggers generation
- Returns:
v4.5.1 Project-Based Learning Framework
Six-stage progressive learning path (strict order):
| Stage | Focus | Content Type |
|---|---|---|
| 1. 理论基础 | What/why | Core concepts, definitions, problems solved |
| 2. 环境搭建 | Setup | Installation, config, hello-world, verification |
| 3. 核心能力 | 80% use case | Main APIs, basic workflows, essential operations |
| 4. 进阶提升 | Advanced | Edge cases, best practices, pitfall avoidance |
| 5. 性能优化 | Tuning | Performance analysis, bottlenecks, optimization |
| 6. 高效实战 | Real project | Automation, ecosystem, team collaboration |
Matrix generation prompt pattern:
system_prompt = f"""你是专业的技术学习路径设计师。
用户想学习「{topic}」,请生成一个完整的分阶段项目学习方案,包含 {card_count} 张高质量学习卡片。
## 学习阶段划分(必须严格按这个顺序)
1. 理论基础(2-3张)
2. 环境搭建(1-2张)
3. 核心能力(3-4张)
4. 进阶提升(2-3张)
5. 性能优化(1-2张)
6. 高效实战(1-2张)
每张卡必须包含:action_item(学完后要做的具体动作)
难度分布:60% 入门,30% 进阶,10% 深入
实操导向:至少 40% 的卡片是 is_practical: true
返回 JSON:{{topic, learning_phases, path_description, cards: [...]}}
"""
Card preview UI pattern:
- Group cards by learning phase with section headers
- Visual distinction: 📖 theoretical, ✋ practical
- Each card shows
action_itemin a highlighted callout box - Preview-first workflow: generate → review → batch import
Hot topics quick-start pattern:
- Show clickable tags below main input: "🔥 大家都在学: React (15), Agent (12)..."
- Tags styled with background color, rounded corners, hover effect
- Click fills input and triggers generation directly
- Builds social proof and reduces user effort (zero-type generation)
Data Flywheel Growth Stages
| Stage | User Scale | Effect |
|---|---|---|
| Cold start | 1-10 | First users build foundation cards across domains |
| Acceleration | 10-100 | Hot topics emerge, generation quality visibly improves |
| Self-growth | 100+ | Platform content rich enough that generation quality surpasses any single expert |
Quiz Question System
All 213 terms have pre-generated multiple-choice questions stored in the terms table:
quiz_q— the question textquiz_a— correct answer textquiz_options— JSON array of 4 options
Generation script: Run once to populate all terms with 4-choice questions:
import sqlite3, random
conn = sqlite3.connect('db/ai_learning.db')
for term_id, name, en, desc, depth in all_terms:
# Get 3 distractor terms from same depth
distractors = random.sample([t for t in terms if t[4]==depth and t[0]!=term_id], 3)
options = [name] + [d[1] for d in distractors]
random.shuffle(options)
question = f"关于 '{name}' 的正确描述是?" if depth > 0 else f"'{en}' 的中文术语名称是什么?"
conn.execute("""UPDATE terms SET quiz_q=?, quiz_a=?, quiz_options=? WHERE id=?""",
(question, name, str(options).replace("'", '"'), term_id))
Fallback: If no pre-generated question exists, dynamically generate true/false questions with keyword swapping (AI ↔ 数据库, 模型 ↔ 服务器, etc.).
Mastery Distribution Visualization
Dashboard API computes mastery levels from SM-2 ef (easiness factor):
ef >= 3.0→ 精通 (Master)ef >= 2.5→ 熟练 (Proficient)ef >= 2.0→ 掌握 (Competent)ef >= 1.5→ 了解 (Familiar)ef < 1.5→ 生疏 (Rusty)
Quiz Submit API (POST /api/quiz/submit)
Schema note: The quiz_log table only tracks per-quiz results, not per-answer correctness. Fields available:
id, career_name, phase_order, total_questions, correct_answers, score_pct, taken_at, user_id- ❌ No
term_idor per-answercorrectcolumns — cannot track which specific questions were wrong - ❌ No
created_at— usetaken_atinstead
Workaround: Track weak areas by phase_order instead — phases with score_pct < 70 indicate weak areas. Update all terms in that phase as needing review.
Progressive Depth System
Three-tier learning: state.learnDepth (0/1/2)
| Level | Content | Trigger |
|---|---|---|
| L1 | Card name + description | Default |
| L2 | + Sub-cards | depth >= 1 |
| L3 | + Full detail (collapsible sections) | depth >= 2 |
setLearnDepth(d) switches depth, clears richCards cache, re-renders.
TTS Voice System
Browser SpeechSynthesis API (free, no API key). Voice: zh-CN "婷婷".
Two modes:
tts.mode='lecture'— 讲稿式分段朗读 with highlight synctts.mode='quick'— 单段速读
tts.buildLecture(card, rich) generates natural lecture sections:
- "你好,我们来学习「X」这个概念。"
- English name
- Simple definition
- Detail paragraphs (split at 80 chars)
- Examples
- Common mistakes
- "总结一下...你学会了吗?"
Visual sync: tts.onHighlight(idx) callbacks highlight .detail-section elements with .tts-highlight class.
Content Generation Template
Use POST /api/terms/{id}/content with:
{
"detail_md": "**加粗标题**\n\n### 小标题\n- 要点\n\n| 表格 | 数据 |",
"examples": "案例1 | 案例2 | 案例3",
"common_mistakes": "❌ 错误认知 → ✅ 正确理解",
"related_terms": "关联名词1 T001 | 关联名词2 T002"
}
Markdown rendering: simpleMD() function converts **bold**→<b>, ### heading→<h4>, `code`→<code>, \n→<br>.
Patch Tool Escape-Drift Workaround
The Hermes patch tool double-escapes backslashes when replacing JavaScript regex patterns in HTML files. For any replacement involving \\ in JS (regexes, template literals, replace()), use a Python script via terminal instead:
with open('index.html', 'r') as f: content = f.read()
content = content.replace(old_bytes, new_bytes)
with open('index.html', 'w') as f: f.write(content)
Also: when using patch in JS template literals with ${...}, the tool may escape-quote \\\\\" sequences incorrectly. Fall back to Python.
CSS comment syntax bug: The patch tool sometimes silently drops the closing */ of CSS comments, causing broken styles. Always verify CSS comment syntax after patching, and add missing */ manually if needed.
Feature Iteration Workflow
For each version increment:
- Implement feature: Backend routes first, then frontend UI
- Update CHANGELOG.md: Summarize new features, optimizations, fixes
- Update README.md: Bump version + new tagline
- Run full test suite:
python -m pytest tests/— target 100% green - Cleanup: Move deprecated scripts to
_archive/(don't delete — preserves git history) - Git commit:
git add -A && git commit -m "feat: vX.Y ..." - Git tag:
git tag vX.Y - Report: 72 tests all green is the standard baseline (v4.4+)
Release checklist template:
✅ vX.Y 版本发布完成!
🎯 新增功能:
• [功能1] - 描述
• [功能2] - 描述
📊 版本信息:
• Git 标签: vX.Y
• 测试结果: 72 / 72 ✅ 全绿
• 改动文件: N 个
• CHANGELOG: 已同步
DB Seeds
server/seed.py— terms + paths from JSON filesserver/seed_content.py— detailed content for terms- Terms JSON:
01-名词卡片/terms-full.json - Paths JSON:
career-paths.json
Adding a Learning Path (新增学习路径)
新增一条职业/学习路径 = 写 3 张表(career_paths / path_phases / path_terms)+ 按需新建卡片。核心数据源是 SQLite db/ai_learning.db;career-paths.json 只是 seed 源,运行时 API 只读数据库。
不要用 seed.py 加单条路径——它的 seed_terms/seed_paths 会 DELETE FROM 后全量重导(清空用户学习进度)。单条新增必须增量 INSERT。
表结构速查
| 表 | 关键字段 |
|---|---|
career_paths |
name(UNIQUE), description, total_cards, level, level_label |
path_phases |
career_name, phase_order(从1), phase_name, phase_desc, phase_outcome |
path_terms |
career_name, phase_order, term_id, term_order(从1) |
卡片表 terms:id(T001), ⭐⭐⭐⭐), name, en, description, difficulty(⭐layout(hub/pipe/stack/comp/cycle), labels(用|分隔), group_name, sort_order, content_md(教程正文), quiz_q/quiz_a/quiz_options(JSON 数组字符串)。
新卡片编号与分组
- 最大编号:
SELECT id FROM terms ORDER BY CAST(substr(id,2) AS INTEGER) DESC LIMIT 1(2026-08 为 220,从 T221 继续) group_name用现有分类:AI开发工具/AI Agent/基础概念/前沿方向/RAG与知识/部署与MLOps等
幂等写入模式(可直接套用)
- 卡片:
INSERT OR IGNORE INTO terms (...),或先SELECT id判断再插 - 路径:
INSERT INTO career_paths ... ON CONFLICT(name) DO UPDATE SET ... - phases / terms:先
DELETE FROM path_phases WHERE career_name=?(和path_terms)再插入,保证干净 - terms 插入自动触发 FTS 触发器同步全文搜索,无需手动 rebuild
步骤顺序
- 先备份(这个库有损坏前科,见下文 DB Corruption Recovery):
cp db/ai_learning.db /tmp/ai_learning.db.bak_$(date +%s) - 幂等写入卡片 + 路径 + phases + terms(建议
write_file写一个临时.py脚本再python3执行,避免 heredoc 中文+emoji 触发安全扫描) - API 验证:
GET /api/paths?search=<关键词>能搜到;GET /api/paths/{name}返回 phases/cards 完整(注意list_paths有@cached(ttl=300),新路径可能延迟可见,用 get 详情验证最准) - 预热介绍页:
GET /api/paths/{name}/intro调 LLM 生成easy_explain/goal/prerequisites/deliverables/work_ready_combo落库path_intro;不预热则用户第一次打开路径时前端触发(LLM 不可用会 503)。批量预热多条路径时 LLM 网关易限流,单条返回 503 时sleep 5后重试即可成功
Pitfall: 卡片脚本 content_md 转义坑(2026-08-16 验证)
批量写卡片脚本时,content_md 里的代码块/多行内容必须用 \n 转义成单行字符串,不能写真实换行——Python 字符串字面量不支持跨行,会报 unterminated string literal。另外 quiz_options 用单引号包裹时,值里含单引号(如 grep ':80 ')会提前终止字符串,报 { was never closed。
防坑:写完脚本先 python3 -c "import py_compile; py_compile.compile('脚本', doraise=True)" 做语法检查再执行。发现转义问题用 str.replace 精确替换修复(execute_code 在 cron 模式被阻止时,用 write_file 写修复脚本 + terminal 执行)。
Pitfall: 批量操作 terms 别加 depth=0 过滤(2026-08-16 验证)
terms.depth 字段:0=主卡,1/2=子卡片。批量补 content_md/description 等时,若在 WHERE 里加 AND depth=0,会把占多数的子卡片(depth≥1)排除——实测 83 张空卡里 depth=0 只有 15 张,depth=1/2 有 68 张,脚本"正常跑完"其实只处理了主卡。
防坑:
- 批量生成/更新内容时不要按 depth 过滤,除非明确只处理主卡
- 生成完用
SELECT COUNT(*) FROM terms WHERE content_md IS NULL OR content_md=''验证是否真补全,别只看脚本退出码(exit_code=0≠ 跑完了目标数量)
Pitfall: path_intro 缺 easy_explain 列(2026-08-16 修复)
schema.py 迁移 22 建 path_intro 表时漏了 easy_explain 列,但 routes/paths.py 的 _persist_intro / _load_intro 一直在读写它 → 所有路径介绍页 500(报 table path_intro has no column named easy_explain)。
修复:MIGRATIONS 追加
(39, "ALTER TABLE path_intro ADD COLUMN easy_explain TEXT DEFAULT ''", "path_intro.easy_explain"),
运行中的库立即生效(WAL 支持并发写,不必重启服务):
sqlite3 db/ai_learning.db "ALTER TABLE path_intro ADD COLUMN easy_explain TEXT DEFAULT '';"
sqlite3 db/ai_learning.db "INSERT OR IGNORE INTO schema_version (version,note) VALUES (39,'path_intro.easy_explain');"
批量内容补齐工作流(content_md + phase_outcome)
"内容丰富"类任务 = 批量调用 LLM Gateway 给空字段补内容。先审计缺口,再批量生成。
字段完整度审计(先查再补)
cd /Users/hua/6-产品研发/ok-KnowHow知渔
sqlite3 db/ai_learning.db "SELECT COUNT(*) FROM terms WHERE content_md IS NULL OR content_md='';" # 教程正文缺口
sqlite3 db/ai_learning.db "SELECT COUNT(*) FROM path_phases WHERE phase_outcome IS NULL OR phase_outcome='';" # 阶段产出缺口
sqlite3 db/ai_learning.db "SELECT COUNT(*) FROM path_phases;" # 阶段总数(218 个阶段里 165 空 = 76% 属常态)
Pitfall — 表名是 path_phases 不是 phases:查阶段数据时 SELECT ... FROM phases 报 no such table: phases。阶段数据在 path_phases(字段 career_name, phase_order, phase_name, phase_desc, phase_outcome),卡片链在 path_terms(term_id 关联 terms,用 term_order 排序)。
批量生成脚本(可复用)
脚本:scripts/enrich_content_md.py(补 terms.content_md)+ scripts/enrich_phase_outcomes.py(补 path_phases.phase_outcome)。核心要点:
- 走 LLM Gateway
http://127.0.0.1:18888/openai/v1/chat/completions,模型deepseek-v4-pro,key 占位gateway-local-no-key-required max_tokens必须 6000(推理模型 reasoning_content 会吃掉 token,见上文 DeepSeek 截断 pitfall),并加"content 长度 < 阈值则判失败"校验- phase_outcome 脚本用
path_phases LEFT JOIN path_terms LEFT JOIN terms ... GROUP_CONCAT(t.name)把阶段内卡片名喂给 LLM,产出"学完能独立做什么"一句 20~40 字 - 逐条 UPDATE + commit,单条失败只影响那一行,脚本继续
- 165 个阶段约 40~60 分钟(约 4 张/分钟),用
terminal background=true + notify_on_complete=true跑,别前台阻塞;用process(action=poll)+ 查库计数确认进度(脚本 stdout 可能因缓冲延迟显示)
跑完验证
sqlite3 db/ai_learning.db "SELECT COUNT(*) FROM terms WHERE content_md IS NULL OR content_md='';" # 应 = 0
sqlite3 db/ai_learning.db "SELECT COUNT(*) FROM path_phases WHERE phase_outcome IS NULL OR phase_outcome='';" # 应 = 0
别只看脚本退出码/打印——exit_code=0 ≠ 目标数量补全(见上文 depth=0 过滤坑)。
Job Database (ai_jobs)
The ai_jobs table tracks real AI job postings for the career guidance section. Schema and update workflow in references/job-database.md.
Weekly Cron Update
Run as a cron job every Monday. The workflow:
- Query existing jobs:
SELECT id, title, company, term_ids FROM ai_jobs ORDER BY id - Search for new AI jobs (see anti-bot note below)
- Insert 5-10 new jobs with all required fields
- Expire stale jobs (update status + updated_at) — either ~5 hand-picked, or a deterministic month-sweep of any whole batch whose month-level
posted_dateis >1 month old (seereferences/job-database.md→ "Expiring Jobs") - Report: new count, expired count, active count, category breakdown
Reusable script: scripts/update_ai_jobs.py (parameterized INSERT + expire + report; copy, fill new_jobs/expire_ids, run). The term_ids map in references/job-database.md is the authoritative one — verify IDs against terms before writing.
Pitfall: sqlite3 db < file.sql does NOT stop on parse errors — retrying the fixed file creates DUPLICATE rows (2026-09-14 incident)
The sqlite3 CLI prints parse errors but keeps executing subsequent statements. The weekly file (1 UPDATE-expire + N INSERTs) that fails on a mid-file quote bug has ALREADY committed everything before the error; re-running the corrected file then (a) re-expires the prior run's still-active inserts, (b) re-inserts every job as a duplicate with fresh autoincrement ids. Verified arithmetic from 2026-09-14: 3 attempts (3 quote bugs fixed one at a time) → 11 ghost rows (ids 48–58, all status='expired') duplicating the 7 real jobs (ids 59–65). The active set stays correct — each retry's UPDATE expires the prior copies — so the UI is unaffected and the corruption hides in expired-row counts only. Tell-tale symptom: an "expired count larger than expected" in the weekly report.
Prevention — run the weekly file atomically (file starts with BEGIN; and ends with COMMIT;):
sqlite3 -bail "/path/to/ai_learning.db" < /tmp/ai_jobs_weekly_update.sql
-bail stops at the first error; BEGIN/COMMIT makes the run all-or-nothing so a failed attempt leaves zero partial state and can simply be re-run after the fix.
Recovery after an already-duplicated run:
SELECT title, company, posted_date, COUNT(*) c FROM ai_jobs
GROUP BY title, company, posted_date HAVING c > 1; -- find ghosts
DELETE FROM ai_jobs WHERE id BETWEEN 48 AND 58; -- 2026-09-14 leftover ghosts (expired dupes of the 7 live jobs)
Root cause of the failed parses — ASCII apostrophes inside Chinese JD prose: quoting a term inside requirements text as '数据元年' / '超级个体' / '大脑' with ASCII single quotes terminates the SQL string literal early (Parse error near "数据元年"). Use Chinese quotes “ ” for any quoted term in Chinese prose, and lint the file before executing (BSD grep lacks -P, use python):
python3 -c "import re,sys; s=open(sys.argv[1],encoding='utf-8').read(); bad=[l for l in s.splitlines() if re.search('[\u4e00-\u9fff]'[\u4e00-\u9fff]',l)]; print('BAD:',bad[:3]) if bad else print('OK')" /tmp/ai_jobs_weekly_update.sql
Anti-Bot Challenges (verified 2026-08-03)
Status of major Chinese job/news sites when scraped by Hermes browser tools in default local mode (no residential proxy):
| Site | Status | Notes |
|---|---|---|
| 猎聘 liepin.com | ✅ WORKS | Job cards render to real DOM (a[data-nick="job-detail-job-info"]), full JD text visible on detail pages (/job/<id>.shtml), update date stamped in body. First-choice source. |
| 36kr.com | ✅ WORKS | Article search /search/articles/<kw>?sort=date returns dated cards; article detail pages expose full text via document.body.innerText. Best for market-trend intelligence (e.g., "具身智能招聘市场热"). |
| BOSS直聘 zhipin.com | ❌ BLOCKED | Canvas rendering, security-check overlay, encrypts job IDs. No DOM access without residential proxy + cookie replay. Skip entirely in cron mode. |
| 百度搜索 baidu.com | ❌ BLOCKED | Returns CAPTCHA wall (wappass.baidu.com/static/captcha/...). Even simple site-restricted searches fail. |
| Google 搜索 | ❌ BLOCKED | DNS / connection timeout from current network. Not reachable from cron environment. |
Strategy: 猎聘 (real jobs w/ salaries) + 36kr (market trends & company expansion news) is sufficient for one weekly update. Do NOT spend tool budget on blocked sites — confirm block once, then move on.
Web-search fallback (verified 2026-08-17): When browser scraping is blocked/unavailable, web_search alone is enough to compile the weekly update. Search snippets from BOSS直聘/猎聘 carry title | company | location | salary | experience inline, and news articles (腾讯/小米 2027 校招, DeepSeek Harness 扩编, 21财经 AI训练师薪资) supply concrete roles + salary bands + requirements you can write 4-6 bullets from. Note: the default web_extract backend is DuckDuckGo which is search-only — it cannot extract URL content and returns "DuckDuckGo (ddgs) is a search-only backend". Don't waste calls on web_extract; rely on web_search snippets + report body instead. web_search may intermittently time out (ConnectTimeout) — retry the same query, or rephrase, rather than abandoning.
Pitfall — tool budget: A full weekly update must complete within ~25-30 browser tool calls. Budget breakdown that fits:
- 1 call: search/listings page → extract job URLs from DOM in one
browser_consolecall - 3-4 calls: open detail pages for the top candidates (one
browser_navigate+ onebrowser_consoleextract per job) - 1 call: final DB write via
terminal python3 -c "..."heredoc - Reserve 2-3 calls for verification + retries
If you find yourself doing more than 5 sequential
browser_navigatecalls into detail pages, you are over-budget — write the rows from listing-page data only and note which need JD enrichment next week.
See references/job-scraping-recipes.md for concrete selectors and extraction patterns.
Cron-Mode Constraints
execute_codeis BLOCKED in cron mode — useterminalwithpython3 -cor write-file-then-execute instead- Heredoc pitfall:
python3 << 'PYEOF' ... PYEOFin terminal may be flagged by the security scanner (confusable Unicode detection on CJK text mixed with ASCII). Workaround: usewrite_fileto create a.pyfile, then execute it withpython3 /path/to/script.py. This avoids the scanner entirely and produces a re-runnable artifact. skill_viewname-resolution failure (2026-09-14 verified):skills_listcan show a skill whileskill_view(name)returnsnot found— registry names may differ from the on-disk directory (example from 2026-09: registeredcron-mode-tool-constraintslived atskills/devops/***SECRET***/; 2026-10-30 curator move: that directory now lives insidecron-self-evolution-toolkit/references/***SECRET***/; retrying the same name loops forever). Workaround that always works: locate the file withsearch_files(pattern: **/<name>/SKILL.md,path: /Users/hua/.hermes) andread_fileit directly. Same fallback for reading a sibling-profile skill you must consult but not modify.- Pipe-to-interpreter (
curl | python3) triggers security approval — split into two steps: download to temp file, then process terminalnetwork requests may time out after 30s — use--max-timeflag
Required Job Fields
Every job MUST have: title, company, location, salary, experience, education, requirements, skills (pipe-separated), term_ids (pipe-separated T-IDs from terms table), url, source, category, posted_date, updated_at, status (active/expired).
Pitfall — terms column is name, not term: SELECT id, term FROM terms fails with no such column: term. The card name column is name (SELECT id, name, group_name FROM terms ORDER BY id). Verify term IDs exist before writing them: SELECT id FROM terms WHERE id IN ('T061',...). The authoritative term→ID map (accurate as of 2026-08, T001–T287) lives in references/job-database.md — DO NOT trust memory or the legacy table in old INSERT examples (they carry wrong mappings like T147=Agent开发, which is actually "AI for Science").
See references/job-database.md for full schema and example INSERT.
Multiple DB Locations & Schema Drift (re-verified 2026-09-07)
There are now MULTIPLE "知渔" SQLite DBs — always confirm which path the task/cron actually points to before writing (do NOT assume the skill's documented 主目录), and re-locate when the named path is missing. The cron prompt itself can carry a stale path: on 2026-09-07 it still hard-coded 渔芯独角兽/知渔/db, which no longer exists. Discovery that worked: find /Users/hua -name "ai_learning.db" -not -path "*/Library/*" -not -path "*/.Trash/*", then confirm the candidate holds data (SELECT COUNT(*) FROM ai_jobs; SELECT COUNT(*) FROM terms;) before writing.
| Path | State |
|---|---|
/Users/hua/Documents/New project/ai-learning/db/ai_learning.db |
LIVE — docker compose + sqlite platform in the "New project" workspace (own AGENTS.md); 220 terms (T001–T220); ai_jobs at 65 rows (7 active / 58 expired) after the 2026-09-14 weekly run (ids 48–58 are a known leftover duplicate batch — see the -bail pitfall above) |
/Users/hua/6-产品研发/渔芯独角兽/知渔/db/ai_learning.db |
GONE — 知渔/ dir no longer exists under 渔芯独角兽/ (cron prompts keep carrying this stale path — re-verify every run) |
/Users/hua/6-产品研发/渔芯独角兽/00-基本完成/知渔/db/ai_learning.db |
parallel snapshot (287 terms incl. T227–T287; not re-verified since 2026-09-07) |
/Users/hua/6-产品研发/ok-KnowHow知渔/db/ai_learning.db |
older snapshot (not re-verified since 2026-09-07) |
Terms count drifts per copy (206 / 220 / 287) — a T-ID valid in one copy may not exist in another. Always validate term_ids against the LIVE terms table of the DB being written.
Pitfall — the empty target DB may carry a SIMPLIFIED ai_jobs schema. The fresh 知渔/db table had only id,title,company,category,salary,location,skills,requirements,term_ids,updated_at — missing source, status, experience, education, url, posted_date. The task requires source (来源) and status tracking (标记新岗位/下线过期岗位), so you MUST migrate the schema before INSERT, never assume it matches production:
cols = {r[1] for r in cur.execute("PRAGMA table_info(ai_jobs)")}
missing = [("experience","TEXT DEFAULT ''"), ("education","TEXT DEFAULT '本科及以上'"),
("url","TEXT DEFAULT ''"), ("source","TEXT DEFAULT '公开招聘'"),
("posted_date","TEXT"), ("status","TEXT DEFAULT 'active'")]
for name, ddl in missing:
if name not in cols:
cur.execute(f"ALTER TABLE ai_jobs ADD COLUMN {name} {ddl}")
When the table is empty (0 rows), there are no stale jobs to expire — report expired=0 instead of hunting for old rows. Canonical reference script: 00-基本完成/知渔/db/update_ai_jobs_20260817.py shows the full workflow (backup → INSERT new jobs → expire stale → report). The terms table in the production DB has grown to T287 with new group_names 网络安全(T227–T241) / 爬虫技术(T242–T257) / 网络技术(T258–T287) — always re-verify term_ids against the LIVE terms table (SELECT id,name,group_name FROM terms), never trust memory.
Database Corruption Recovery
The ai_learning.db SQLite file can become corrupted while the server is running (e.g. sqlite_autoindex_users_1 index damage). Symptoms: SELECT works but INSERT/UPDATE fails with sqlite3.DatabaseError: database disk image is malformed. Integrity check (PRAGMA integrity_check) reports specific index corruption.
Recovery procedure (works even when .dump/.recover fail):
# 1. Clone to a new file — this filters out corruption
sqlite3 "/path/to/ai_learning.db" ".clone /tmp/ai_learning_clone.db"
# 2. Verify the clone is clean
sqlite3 /tmp/ai_learning_clone.db "PRAGMA integrity_check;" # should return "ok"
# 3. Backup the corrupted original (keep for forensics)
cp /path/to/ai_learning.db "/path/to/ai_learning.db.corrupted_bak_$(date +%Y%m%d_%H%M%S)"
# 4. Remove WAL/SHM files (they're tied to the old DB)
rm -f /path/to/ai_learning.db-shm /path/to/ai_learning.db-wal
# 5. Deploy the clean clone
cp /tmp/ai_learning_clone.db /path/to/ai_learning.db
The .clone approach preserves all tables and data while dropping the corrupted index structures. FTS5 virtual tables may emit benign warnings during clone (they're rebuilt on access) — this is harmless. Run all DB writes against the clone, then deploy it to production.
When to suspect corruption: if SELECT COUNT(*) works but any write fails with "malformed", the DB is corrupted. Don't try REINDEX or PRAGMA writable_schema — clone is the safest recovery path.
See references/db-corruption-recovery.md for detailed diagnosis steps.
Doubao Image Generation Workflow (豆包生图)
For generating visual flashcard illustrations using Doubao (豆包) AI image generator.
Directory Structure
Two project locations are supported (note lowercase 'ai'):
- Primary:
/Users/hua/6-产品研发/23-ai培训教程/ - Secondary:
/Users/hua/Desktop/渔芯科技/workspace/AI培训平台工程/
02-豆包生图Prompt/ # Prompt files (ready for generation = ok- prefix)
├── 00_生图总览.md # Progress tracking
├── ok-领域_*.md # Completed prompt files (marked as ready)
└── 领域_*.md # Draft prompt files (not yet ready)
03-豆包生成图/ # Renamed images (standard naming convention)
└── T001-AI(人工智能)_v1.png, T002-*_v2.png, etc.
04-最终精选图/ # Best images selected for platform
05-学习平台/ # Final platform integration
Image Naming Pattern
Standard format: T{XXX}-{中文术语}_v{N}.png
Example: T001-AI(人工智能)_v1.png, T002-机器学习_v4.png
Batch Renaming Workflow
First step: ALWAYS search for existing Doubao-generated images before creating new ones. The user typically generates images manually first.
# Search for recently downloaded PNGs/JPEGs
find ~ -name "*.png" -mtime -1 2>/dev/null | head -30
find ~ -name "*.jpg" -mtime -1 2>/dev/null | head -30
ls -lt ~/Downloads/ 2>/dev/null | head -30
ls -lt ~/Desktop/ 2>/dev/null | head -30
When execute_code is blocked (cron mode): Use sequential terminal commands. This is more reliable and avoids security restrictions:
cd "/Users/hua/6-产品研发/23-ai培训教程/02-豆包生图Prompt"
# T001 - AI(人工智能)- first 4 images (no t-prefix)
cp "AI名词知识卡片风格分析.png" "../03-豆包生成图/T001-AI(人工智能)_v1.png"
cp "AI名词知识卡片风格分析 (1).png" "../03-豆包生成图/T001-AI(人工智能)_v2.png"
cp "AI名词知识卡片风格分析 (2).png" "../03-豆包生成图/T001-AI(人工智能)_v3.png"
cp "AI名词知识卡片风格分析 (3).png" "../03-豆包生成图/T001-AI(人工智能)_v4.png"
# T002 - 机器学习 (t002 prefix)
cp "t002AI名词知识卡片风格分析.png" "../03-豆包生成图/T002-机器学习_v1.png"
cp "t002AI名词知识卡片风格分析 (1).png" "../03-豆包生成图/T002-机器学习_v2.png"
cp "t002AI名词知识卡片风格分析 (2).png" "../03-豆包生成图/T002-机器学习_v3.png"
cp "t002AI名词知识卡片风格分析 (3).png" "../03-豆包生成图/T002-机器学习_v4.png"
# T003-T012 follow similar pattern with continuous numbering
# T005 starts at (4), T006 at (8), T007 at (12), T008 at (16), etc.
When execute_code is available: Use the Python pattern for larger batches:
import os
import re
import shutil
source_dir = "02-豆包生图Prompt"
target_dir = "03-豆包生成图"
os.makedirs(target_dir, exist_ok=True)
# T-ID to Chinese name mapping (from prompt files)
t_names = {
"T001": "AI(人工智能)",
"T002": "机器学习",
# ... add all terms
}
# Group and sort images
for filename in sorted(os.listdir(source_dir)):
if not filename.endswith('.png'):
continue
# Detect T-ID: match t002 prefix or bare numbers for T001
match = re.match(r'^t(\d{3})', filename, re.IGNORECASE)
if match:
t_id = f"T{match.group(1).upper()}"
else:
t_id = "T001" # Files without t-prefix are T001 (first term)
# Sort within group: no parentheses = v1, then by number in parentheses
# Rename to standard format and copy to target_dir
User Mandate for Image Renaming
NON-NEGOTIABLE: User-established rule (2026-07-13)
"必须每一张都识别一次再编号。" "Must identify EVERY SINGLE image before renaming."
You MAY NOT:
- Assume pattern holds based on first few samples
- Trust manifest file numbers without verification (manifest said T001 had 36 images; actual pattern is 4 per term)
- Proceed with bulk renaming if you cannot reliably identify images
When vision tools fail (they DO fail, often):
- Be honest immediately — "My vision tools cannot reliably identify these images"
- Propose clear options to the user:
- Option A: Proceed with timestamp-based ordering + manifest (user spot-checks)
- Option B: User verifies 2-3 key points to confirm pattern, then proceed
- Option C: Wait for user to provide verified mapping
- Never pretend you identified images when you couldn't
Key Pitfalls
- T001 has no prefix: The first batch (T001) generates with no t-prefix in the filename
- T001 may have MANY images: The first term (T001) may have 36+ images from multiple generation runs, not just 4 — verify actual count, don't trust manifest
- Vision tools hallucinate:
browser_visionandvision_analyzeare unreliable. They may return unrelated content (e.g., medical terminology instead of AI labels) or claim "cannot see image". Verify every result, don't trust blindly. - Mark
ok-prefix: When a prompt file is ready for generation, prefix withok-(e.g.,ok-领域_基础概念.md) - Don't generate fallback images first: The user often generates images manually in Doubao first and needs them renamed/organized. ALWAYS search for existing images in Downloads/Desktop first before generating fallback PIL images. Search paths:
~/Downloads/,~/Desktop/, and Doubao app containers. Usefind ~ -name "*.png" -mtime -1to locate recently generated images. - Clarify before generating: If the user says "rename images" or "change filenames", they mean their existing Doubao-generated images, not new generated ones. Only generate PIL images as a last resort when no existing images can be found.
- execute_code blocked in cron mode: When running as a cron job,
execute_codeis blocked for security. Use sequentialterminalcpcommands instead for batch file operations:cp "source-file.png" "../03-豆包生成图/T001-AI(人工智能)_v1.png" cp "source-file (1).png" "../03-豆包生成图/T001-AI(人工智能)_v2.png" # Continue with explicit commands - Directory path case sensitivity: The project directory may be
23-ai培训教程(lowercase 'ai') not23-AI培训教程. Always verify the actual path first.
Progress Tracking
Update 00_生图总览.md with:
- Table of domains with status (⏳待处理 / ✅待生成 / 已完成)
- Per-term image count
- Total image count in
03-豆包生成图/ - Links to the directory
Script: See scripts/rename_images.py for the ready-to-use batch renaming utility.
Fallback Image Generation (PIL/Python)
When Doubao web login or API authentication is unavailable, generate concept images programmatically with Python PIL. This produces neural-network style visuals suitable for educational flashcards.
Script location: scripts/generate_concept_images.py
Usage:
python3 scripts/generate_concept_images.py \
--term-id T001 \
--term-name "AI(人工智能)" \
--output-dir "03-豆包生成图/" \
--styles 4
What it generates (4 style variations):
- 科技蓝 (Tech Blue): Dark blue background with glowing blue nodes
- 未来紫 (Future Purple): Purple theme for futuristic look
- 活力橙 (Vibrant Orange): Warm colors for energy concepts
- 清新绿 (Fresh Green): Green theme for growth/nature-related AI
Visual elements included:
- 5-layer concentric neural network node layout
- 150+ random connecting lines (data flow)
- Center brain-shape halo (5 circles expanding outward)
- Chip icon (with pins) for hardware reference
- 20+ diagonal data stream particles
Image specs: 1024x1024 PNG, ~70KB each.
When to use:
- Doubao login required but no session available
- API key authentication failures
- Rate limiting or service downtime
- Need consistent visual style across all cards
- Generating placeholder images before AI generation
图片-卡片映射机制与图文核对(2026-08-16 实战)
图片靠文件名匹配卡片,映射存数据库 card_images 表
图片本身不在 terms 表里(terms 无 image 字段)。映射链路:
04-最终精选图/t{编号}-{中文名}-v01.png ← 文件名以 term_id 开头(大小写不敏感)
↓ scan_card_images.py (POST /api/cards/sync)
card_images 表 (term_id, style_key, url, variant, is_primary)
↓ GET /api/styles/{style}/primary
前端 IMG_MAP[card.id] → 图片路径
card_images表字段:term_id, style_key(ai/comic/humor), url(/styles/ai/xxx.png), variant, is_primary- 扫描器
scripts/scan_card_images.py:只新增/更新,不删除已不存在的文件记录;同term_id+style_key+variant冲突时ON CONFLICT DO UPDATE - 前端 primary = 每个 term_id 里
variant最小 +is_primary=1的那条
图文不符检查流程(华哥要求"图片内容与文字内容不符"时)
- 文件名层面比对(最快,先做):提取图片文件名里的中文名,和
terms.name比对,抓出"文件名概念 ≠ 卡片名概念"的 - vision 抽查坐实:对嫌疑图
vision_analyze,确认图片左上角印的编号/标题和卡片名对不上(注意:豆包生图会把编号 TXXX 印进图里,可作铁证) - 修复要改两处:
- 文件系统:错位图重命名归位 / 孤儿图移到
_orphan_bak_日期/(可逆,不直接删) - 数据库:
card_images表UPDATE归位图的 url、DELETE孤儿图记录——只改文件系统不够,前端读库
- 文件系统:错位图重命名归位 / 孤儿图移到
常见错位模式(渔芯实测)
- 旧卡片图残留:一批卡片(如 Agent 框架 OpenHands/SWE-agent/AutoGPT)被删/重构后,旧图占着编号,挂到新卡片上(T205 挂 OpenHands 图但卡片名是"MCP协议")
- 编号笔误孤儿图:
t312-llama.cpp(应为 T132)、t447(应为 T147)、t503(应为 T053)——多打一位数字,图成了孤儿 - 乱码编号:
t0277-System Prompt(应为 T054)、t0056-流式输出(应为 T056)——图内容对但编号错,应归位而非删除
培训内容方法论(原 ai-training-content)
零基础员工 AI 培训的内容设计方法论。核心是需求驱动(demand-driven,反向设计):从学员目标出发("我要做 X → 需要学 Y"),不是从知识树出发。
7 原则:① 从需求出发(支持 39 职业路径 + 任务关键词匹配)② 先学最需要的(UbD 反向设计,--minimal 把 52 卡路径砍到 ~20 卡
…(truncated)