听课模式流式音频
核心规则
is_speakable且暂无可播音频时视为 buffering,不自动跳过。- 音频来源单选:优先
audioTracks,不可播时才回退audio_url/audio_segments。 - 分段去重 key 使用
element_id + position + segment_index,并允许后到分段提升isFinal。
工作流
- 合并增量
audio_segments后再写回状态,不直接全量替换。 - 仅新增当前步骤可播片段时不重置整段播放;结构变化才重启。
- 用户手动切 marker 时立即清除当前 buffering 状态。
run请求体listen必须跟随当前真实听课状态:只有learningMode === 'listen'且课程tts_enabled可用时才发送listen=true;阅读模式即使课程支持听课也必须发送listen=false。LIKE_STATUS仅表示流结束信号,TTS 还需校验is_speakable或已有可播音频。requestAudioForBlocksupports on-demandgenerated-blocks/:id/ttscalls, including programmaticlisten=falserequests. This hook capability does not imply a reading-mode playback button:LessonConversationhides audio actions in both normal and preview reading mode. Its mode-level gate is!isClassroomMode && (previewMode || isListenModeActive) && !isPreviewReadMode; eligible content must also satisfy the speakability rules. Listen mode uses the existing automatic backfill queue described below; do not add an independent request fromonStepChange.listen-mode映射elementList时显式透传isAudioStreaming。- 统一分发层处理
type/error或event_type/error,立即弹出 destructive toast。 - 音频播放 icon 的可见性需受
is_speakable控制:当is_speakable === false时不展示播放按钮。 - 处理
audio_segment/audio_complete时禁止直接把Record<string, unknown>断言为目标类型,必须先做字段级归一化再写入upsertAudioSegment/upsertAudioComplete。 - 用 Storybook 或
markdown-flow-ui回放data.json排查时,elementList的组装路径必须和ai-shifu听课模式保持同构,禁止额外做 story 专用的音频归属重排补偿。 - 听课模式组装
elementList时,若内容本身是<iframe data-tag="video" ...></iframe>形式的视频 iframe,即使后端element_type仍是通用类型,也要优先把 slide element 的type推断为video。 - 听课模式消费
record历史接口时,若payload.audio.subtitle_cues存在,必须在elementList中显式透传为subtitle_cues,不要只保留content导致字幕时间轴元数据在渲染层丢失。 - 听课模式
run的audio_segment若已生成当前累计字幕 cue,必须同步透传到对应 element patch 的payload.audio.subtitle_cues,不要等到audio_complete才返回字幕。 - 课程设置里的
tts_enabled虽然沿用后端字段名,但前端产品语义视为“听课模式开关”;改动作者端文案或 learner 端默认模式时,必须同时检查设置页文案、课程信息映射和/c/...布局层默认learningMode是否一致。 - 阅读/听课模式切换只切换课时展示形态,不应硬断正在进行的
/runSSE;只有重修、离开课时、lesson reset 等真实生命周期结束场景才触发全局 stream stop。 - 进入听课模式后持续扫描当前 lesson 已有正文中
is_speakable !== false且有有效element_bid的内容块;历史回放内容天然可补,流式/run新内容必须等后端在原有事务 commit 之后发出audio_backfill_ready才能调用generated-blocks/:id/tts?listen=true,禁止用尚未提交的 streaming element 直接补音频。补生成对同一 generated block 做 in-flight 去重;第一段音频可播后允许进入播放,整条 TTS SSE 继续补齐后续 position;没有可补正文或补音频失败时,不再提示重修或强制切回阅读,只保持当前模式并用 toast/缓冲态反馈。 - 听课模式把后端
subtitle_cues[].text透传给markdown-flow-ui/slide前,要先过滤尾部终止标点:默认删除句号、逗号、冒号、分号等收尾符号,但保留问号、叹号、省略号,以及成对标点的后半个(如”、)、》)。 - 听课模式组装 item 时,如果 final
element事件只把完整audio_url放在 element 顶层或payload.audio,而当前 item 已经有audioTracks/audio_segments,必须把完整 URL 回填到对应 position 的audioTracks;不要让 Slide 映射层因为优先使用audioTracks而丢掉完整 mp3 URL。 - 当需要在线上显式区分 buffering 卡住原因时,优先把 loading 文案拆成“未收到当前页音频 / 正在加载当前页音频 / 正在等待当前页后续音频片段”三类可配置文案,不要继续只透传单一字符串。
AudioPlayer已经播放过流式audio_segment后,不要在同一播放会话中切到audio_complete.audio_url续播;完整 URL 只用于未来重播或零 segment 缓存回放,否则 mp3 拼接/seek 偏差会导致句首几个字重复。
备注
- 回放疑难问题时优先复用 Storybook 的原始
runfixture,按 SSE 顺序重放data:片段。
Verification
The listen-mode queue in LessonConversation waits for persisted
audio_backfill_ready for newly streamed content, deduplicates in-flight blocks,
and stops applying results after the lesson scope ends. History and committed
new content have different eligibility. listen-mode-utils.ts determines candidates,
while useLessonConversation records persisted readiness and supplies
requestAudioForBlock. Neither mode switching nor a slide change should create
a competing queue implementation. The lesson-change effect resets queue
bookkeeping and the lesson ref; it does not immediately close a silent old TTS
stream. requestAudioForBlock settles a stale stream when a guarded audio event
arrives, or through normal stream termination/error or the 120-second listen
backfill idle timeout. Do not describe this as eager lesson-switch cancellation.
Run listen-mode-utils.test.ts for candidate selection and block deduplication,
and src/lib/run-with-concurrency.test.ts for bounded worker concurrency.
LessonConversation.test.tsx tests imported projection/loading helpers; it does not
render the component or verify its backfill effect orchestration. For changes
to that effect, add focused orchestration coverage for in-flight deduplication
and lesson-scope result guards instead of treating these helper suites as proof.
Run the adjacent
use-lesson-conversation.test.tsx cases for programmatic on-demand audio requests,
forced listen=true backfill, persisted-ready gating, streaming failures, and
backfill idle timeout. The on-demand request test calls the hook directly and
does not establish a reading-mode UI action; changes to visible audio controls
need rendered component coverage. Also run ListenModeSlideRenderer.test.tsx when the mapping or
playback restore contract changes. Except for the shared concurrency suite, these files live in
src/app/c/[[...id]]/components/chat/.
Learning-mode initialization and restored preferences
For /c/... initialization, inspect src/app/c/[[...id]]/layout.tsx and the
adjacent components/learning-mode-preference.ts, learning-mode-url.ts,
learning-mode-storage.ts, and learning-mode-analytics.ts before changing effects.
- Resolve a valid modern
modeURL value first, then the legacylistenoverride, then the course-scoped stored preference, then the course default. A default oflistenrequires both default-listen and TTS capability to be enabled; otherwise useread. Preserve the helper's TTS/classroom capability checks and its pending-capability behavior. Legacylistenis normalized intomodewithout overriding an existing validmodevalue. - Storage is scoped by
course_learning_mode:<course id>and acceptsread,listen, orclassroom; missing or invalid values read as null. The current layout deliberately does not write an empty preference merely because a default resolved without a URL override. The former instruction to persist that first-load default immediately is obsolete. Explicit URL selections and existing preferences use the layout's synchronization effect; pending classroom resolution can defer that write. Manual switches update the URL and current mode, letting the same effect persist the choice. - Read and retain the original stored value before synchronization writes.
learner_last_learning_modeis a restored-preference event, not an event for every initialization: exclude preview, URL overrides and missing preferences. Wait for required capability data, require the stored mode to equal the resolved mode, and reject restoration if an explicit selection changed storage while capability data was pending. Deduplicate per course within the mounted layout using its existing refs. Do not classify a newly written value as a previously saved choice or change the existing payload/event names. - Base follow-up visibility on the resolved current mode, including a capability fallback from listen to read; use the Ask placement skill for shared state and action-row placement.
Run learning-mode-preference.test.ts, learning-mode-url.test.ts,
learning-mode-storage.test.ts, and learning-mode-analytics.test.ts from that
components directory, plus src/__tests__/c-preview-layout.test.tsx, with
npm test -- --runInBand --runTestsByPath from src/web. The layout suite covers
restoration/selection separation, URL exclusions, unavailable or delayed TTS,
and selection while capability is pending. It does not specifically assert the
empty-storage default-write guard; inspect that effect rather than inferring
this coverage from the helper tests.
Subtitle punctuation cleanup order
Use stripDisallowedSubtitleTrailingPunctuation in src/lib/subtitle-utils.ts,
as called by resolveListenSlideSubtitleCues in the learner ChatUi utilities.
Walk the suffix from right to left: preserve paired closers in their original
order while removing disallowed punctuation on either side of those closers.
Stop at allowed question/exclamation endings or ellipses; do not strip the
closers or replace this with a single trailing-punctuation regular expression.
For example, suffixes 。”, ),, and ?”。 retain ”, ), and ?”.
Run the strips disallowed trailing punctuation from subtitle cues case in
src/app/c/[[...id]]/components/chat/listen-mode-utils.test.ts. It covers closing
quotes, a closing parenthesis followed by a period, punctuation after a quote,
and question/ellipsis preservation. Retain these cases when changing cleanup,
and add the parenthesis-plus-comma variant if changing that boundary.