Azure AI Content Understanding Domain Knowledge
This skill provides domain knowledge for Azure AI Content Understanding, a multimodal AI service that extracts semantic content from documents, video, audio, and image files.
[COPILOT GUIDANCE]: Always consult the official documentation first before answering user questions. Use fetch_webpage to read the relevant doc page when the reference material below is insufficient or may be outdated.
When a user's question is broad or ambiguous, ask them to clarify:
- "Which modality are you working with — documents, images, audio, or video?"
- "Are you using a prebuilt analyzer, or building a custom one?"
- "Are you asking about the Python SDK specifically, or the service in general?"
Official Documentation
The authoritative source for Content Understanding is: https://learn.microsoft.com/azure/ai-services/content-understanding/
Always read the relevant page (via fetch_webpage) before answering if the reference material below does not cover the topic.
Key Documentation Pages
Search tip: If the above pages don't cover the user's question, search the doc tree at https://learn.microsoft.com/azure/ai-services/content-understanding/.
Python SDK Resources
Field-description rule: the two-stage pipeline
Custom analyzer extraction is a two-stage pipeline:
- Stage 1 — content extraction (OCR + layout). The service reads the
file and produces structured text plus layout metadata (sections, tables,
headings). The original pixels are not what the LLM in stage 2 sees.
- Stage 2 — field extraction (LLM). The LLM reads the stage-1 markdown
and uses your field descriptions to identify values.
Implications for fieldSchema.fields[*].description:
✅ Reference text content and structure: labels ("Invoice #"),
section headings ("Bill To"), adjacent labels, alternative phrasings,
format examples.
❌ Do not reference visual appearance: colour, font, font size, bold or
italic, or "the box at the top-right" without text anchors.
Good description:
"Invoice issue date, found near the 'Invoice #' label at the top right.
May also be labelled 'Invoice Date', 'Date', or 'Issued'. Format is
usually MM/DD/YYYY. Examples: '01/15/2024', 'January 15, 2024'."
Used by cu-sdk-author-analyzer
and cu-sdk-author-analyzer-classify-route.
Choosing baseAnalyzerId
Every custom analyzer extends a built-in prebuilt analyzer via
baseAnalyzerId. Pick the row that matches the modality of the content
you're analyzing (documents, audio, video, image). Typos here are a common
first-time error; the local schema validator (in
_shared/schema_validator.py) rejects any value not in this
table.
| Content type |
baseAnalyzerId |
| Documents (PDF, image of a page) |
prebuilt-document |
| Audio (mp3, wav, m4a) |
prebuilt-audio |
| Video (mp4, mov) |
prebuilt-video |
| Image-only analyzer |
prebuilt-image |
⚠️ Only modality-level prebuilts are valid as baseAnalyzerId for
custom analyzers. *Search variants (prebuilt-documentSearch,
prebuilt-audioSearch, prebuilt-videoSearch), task-specific prebuilts
(prebuilt-invoice, prebuilt-receipt, prebuilt-idDocument), and
prebuilt-layout are not accepted here — the service returns
InvalidBaseAnalyzerId. Those prebuilts can still be called directly as
standalone analyzers via
client.begin_analyze(analyzer_id="prebuilt-invoice", ...). See the
analyzer-reference docs.
Used by cu-sdk-author-analyzer
(custom analyzer) and
cu-sdk-author-analyzer-classify-route
(both inner extractors and the outer classifier).
Classify-and-route rule
When using config.contentCategories to classify and route mixed-document
packets:
- Category descriptions follow the same text-anchored rule as field
descriptions. Describe each category by the text that appears on its
pages (headings, labels), not by visual style.
config.enableSegment must be true so the classifier carves the
packet into segments before routing each one.
- Inner analyzers must already exist before the outer classifier is
created.
- Category fill rate is per-category, not packet-wide. A field that
only appears in invoice segments should be evaluated against the number
of invoice segments, not the total number of segments.
- No top-level
fieldSchema on the outer classifier. The outer
analyzer's job is classification + routing only; field extraction
belongs in the inner analyzers.
Used by cu-sdk-author-analyzer-classify-route.
Related Skills
cu-sdk-setup — Set up Python environment and run samples
cu-sdk-sample-run — Run specific samples interactively
cu-sdk-author-analyzer — Author + test a custom analyzer for one document type
cu-sdk-author-analyzer-classify-route — Author + test a classify-and-route pipeline for mixed-document packets
1---2name: cu-sdk-common-knowledge3description: Domain knowledge for Azure AI Content Understanding. Use this skill to answer questions about Content Understanding concepts, analyzers, field schemas, API operations, and SDK usage. Always consult official documentation before answering.4metadata:5 source: https://skillmd.com/skills/azure-azure-sdk-for-python/cu-sdk-common-knowledge6---78# Azure AI Content Understanding Domain Knowledge910This skill provides domain knowledge for Azure AI Content Understanding, a multimodal AI service that extracts semantic content from documents, video, audio, and image files.1112> **[COPILOT GUIDANCE]:** Always consult the official documentation first before answering user questions. Use `fetch_webpage` to read the relevant doc page when the reference material below is insufficient or may be outdated.13>14> When a user's question is broad or ambiguous, ask them to clarify:15> - "Which modality are you working with — documents, images, audio, or video?"16> - "Are you using a prebuilt analyzer, or building a custom one?"17> - "Are you asking about the Python SDK specifically, or the service in general?"1819## Official Documentation2021The authoritative source for Content Understanding is: **https://learn.microsoft.com/azure/ai-services/content-understanding/**2223Always read the relevant page (via `fetch_webpage`) before answering if the reference material below does not cover the topic.2425### Key Documentation Pages2627| Topic | URL |28|-------|-----|29| **Overview** | https://learn.microsoft.com/azure/ai-services/content-understanding/overview |30| **What's new** | https://learn.microsoft.com/azure/ai-services/content-understanding/whats-new |31| **Content Understanding Studio** | https://learn.microsoft.com/azure/ai-services/content-understanding/quickstart/content-understanding-studio?tabs=portal%2Ccu-studio |32| **Service limits** | https://learn.microsoft.com/azure/ai-services/content-understanding/service-limits |33| **Region & language support** | https://learn.microsoft.com/azure/ai-services/content-understanding/language-region-support |34| **Prebuilt analyzers** | https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/prebuilt-analyzers |35| **Create custom analyzer** | https://learn.microsoft.com/azure/ai-services/content-understanding/tutorial/create-custom-analyzer?tabs=portal%2Cdocument&pivots=programming-language-python |36| **Document markdown** | https://learn.microsoft.com/azure/ai-services/content-understanding/document/markdown |37| **Document elements** | https://learn.microsoft.com/azure/ai-services/content-understanding/document/elements |38| **Video overview** | https://learn.microsoft.com/azure/ai-services/content-understanding/video/overview |39| **Video elements** | https://learn.microsoft.com/azure/ai-services/content-understanding/video/elements |40| **Audio overview** | https://learn.microsoft.com/azure/ai-services/content-understanding/audio/overview |41| **Image overview** | https://learn.microsoft.com/azure/ai-services/content-understanding/image/overview |42| **REST API reference** | https://learn.microsoft.com/rest/api/contentunderstanding/operation-groups |4344> **Search tip:** If the above pages don't cover the user's question, search the doc tree at `https://learn.microsoft.com/azure/ai-services/content-understanding/`.4546### Python SDK Resources4748| Resource | Link |49|----------|------|50| **Python package on PyPI** | https://pypi.org/project/azure-ai-contentunderstanding/ |51| **Python SDK README** | https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/contentunderstanding/azure-ai-contentunderstanding/README.md |52| **Python SDK Samples** | https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/contentunderstanding/azure-ai-contentunderstanding/samples |5354## Field-description rule: the two-stage pipeline5556Custom analyzer extraction is a **two-stage pipeline**:57581. **Stage 1 — content extraction (OCR + layout).** The service reads the59 file and produces structured text plus layout metadata (sections, tables,60 headings). The original pixels are *not* what the LLM in stage 2 sees.612. **Stage 2 — field extraction (LLM).** The LLM reads the stage-1 markdown62 and uses your field descriptions to identify values.6364Implications for `fieldSchema.fields[*].description`:6566✅ Reference **text content and structure**: labels (`"Invoice #"`),67section headings (`"Bill To"`), adjacent labels, alternative phrasings,68format examples.6970❌ Do **not** reference visual appearance: colour, font, font size, bold or71italic, or "the box at the top-right" without text anchors.7273Good description:7475> "Invoice issue date, found near the 'Invoice #' label at the top right.76> May also be labelled 'Invoice Date', 'Date', or 'Issued'. Format is77> usually MM/DD/YYYY. Examples: '01/15/2024', 'January 15, 2024'."7879Used by [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md)80and [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md).8182## Choosing `baseAnalyzerId`8384Every custom analyzer extends a built-in prebuilt analyzer via85`baseAnalyzerId`. Pick the row that matches the **modality** of the content86you're analyzing (documents, audio, video, image). Typos here are a common87first-time error; the local schema validator (in88[`_shared/schema_validator.py`](../_shared/)) rejects any value not in this89table.9091| Content type | `baseAnalyzerId` |92|---|---|93| Documents (PDF, image of a page) | `prebuilt-document` |94| Audio (mp3, wav, m4a) | `prebuilt-audio` |95| Video (mp4, mov) | `prebuilt-video` |96| Image-only analyzer | `prebuilt-image` |9798> ⚠️ **Only modality-level prebuilts are valid as `baseAnalyzerId` for99> custom analyzers.** `*Search` variants (`prebuilt-documentSearch`,100> `prebuilt-audioSearch`, `prebuilt-videoSearch`), task-specific prebuilts101> (`prebuilt-invoice`, `prebuilt-receipt`, `prebuilt-idDocument`), and102> `prebuilt-layout` are **not** accepted here — the service returns103> `InvalidBaseAnalyzerId`. Those prebuilts can still be called directly as104> standalone analyzers via105> `client.begin_analyze(analyzer_id="prebuilt-invoice", ...)`. See the106> [analyzer-reference docs](https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/analyzer-reference#baseanalyzerid).107108Used by [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md)109(custom analyzer) and110[`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md)111(both inner extractors and the outer classifier).112113## Classify-and-route rule114115When using `config.contentCategories` to classify and route mixed-document116packets:1171181. **Category descriptions follow the same text-anchored rule** as field119 descriptions. Describe each category by the text that appears on its120 pages (headings, labels), not by visual style.1212. **`config.enableSegment` must be `true`** so the classifier carves the122 packet into segments before routing each one.1233. **Inner analyzers must already exist** before the outer classifier is124 created.1254. **Category fill rate is per-category**, not packet-wide. A field that126 only appears in invoice segments should be evaluated against the number127 of invoice segments, not the total number of segments.1285. **No top-level `fieldSchema`** on the outer classifier. The outer129 analyzer's job is classification + routing only; field extraction130 belongs in the inner analyzers.131132Used by [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md).133134## Related Skills135136- `cu-sdk-setup` — Set up Python environment and run samples137- `cu-sdk-sample-run` — Run specific samples interactively138- `cu-sdk-author-analyzer` — Author + test a custom analyzer for one document type139- `cu-sdk-author-analyzer-classify-route` — Author + test a classify-and-route pipeline for mixed-document packets