cu-sdk-author-analyzer

name

cu-sdk-author-analyzer

description

Iteratively author and test a custom Azure AI Content Understanding analyzer for a folder of **document** files (PDFs, page images) of a single type. Walks layout extraction → schema authoring → validation → batch test → agent review → refine cycle using the typed ContentUnderstandingClient. Document modality only — audio, video, and image analyzers are planned for a later update. Use when the user wants to author a custom analyzer for invoices, contracts, forms, or any other single-type document set.

Author a Custom Analyzer (single document type)

Author a custom Content Understanding analyzer for one document type end-to-end: extract layout, draft a field schema, validate locally, create the analyzer, batch-test it on sample files, and read a quality summary.

This is an iterative, human-in-the-loop workflow. You will typically run the schema → test → review cycle multiple times to refine field descriptions before you're happy with the extraction quality.

The workflow uses the typed ContentUnderstandingClient shipped in this package — the same client samples/sample_create_analyzer.py and samples/sample_analyze_binary.py use.

[COPILOT INTERACTION MODEL]: This skill is designed to be interactive. At each step marked with [ASK USER], pause execution and prompt the user for input or confirmation before proceeding. Do NOT silently skip these prompts. Use the ask_questions tool when available.

[USE INSTEAD]: If the user's packet contains multiple different document types (for example, an invoice, a bank statement, and a loan application in one PDF), route them to the cu-sdk-author-analyzer-classify-route skill instead. This skill assumes one document type per analyzer.

[ASK USER] Modality check (first thing to confirm):

"Are you working with document files — PDFs or images of pages? This skill currently supports document modalities only. Audio, video, and image analyzers are planned for a future update."

• If the user says document → continue with this skill.
• If the user says audio, video, or image → stop this skill and point them at the matching SDK sample (samples/sample_create_analyzer.py with prebuilt-audio / prebuilt-video / prebuilt-imageAnalyzer) or the REST tutorial.

Prerequisites

Required: venv active, SDK installed, .env with CONTENTUNDERSTANDING_ENDPOINT (plus CONTENTUNDERSTANDING_KEY or az login), and samples/sample_update_defaults.py run once for this resource. Full setup lives in cu-sdk-setup.

[COPILOT] Probe first, then route on failure — do not duplicate setup logic here.

python -c "import sys; print('venv:', 'ok' if sys.prefix != sys.base_prefix else 'INACTIVE')"
( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_ENDPOINT=https?://' .env >/dev/null && echo 'endpoint: ok' ) || echo 'endpoint: MISSING'
( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_KEY=.+' .env >/dev/null && echo 'key: set' ) || echo 'key: empty'
az account show >/dev/null 2>&1 && echo 'az: ok' || echo 'az: not logged in'

Failure	Route to
venv INACTIVE or azure.ai.contentunderstanding import fails	cu-sdk-setup Steps 2–3, then resume
endpoint MISSING	cu-sdk-setup Step 4 (env vars), then resume
endpoint ok, key empty, az: not logged in	cu-sdk-setup Step 4 auth section (run az login or add CONTENTUNDERSTANDING_KEY to .env), then resume
All env checks pass but service calls fail with model errors	cu-sdk-setup Step 6 (sample_update_defaults.py), then resume
All ok	✅ Proceed to Step 1.

Never ask the user to paste an endpoint or API key into chat — they edit .env directly or run az login.

[ASK USER] "How many representative documents do you have, and where are they?" — fewer than 3 is fine, but more is better for testing coverage.

Package directory

sdk/contentunderstanding/azure-ai-contentunderstanding

Scripts and templates

.github/skills/cu-sdk-author-analyzer/
├── scripts/
│   ├── extract_layout.py     # Stage 1
│   ├── extract_layout.sh
│   ├── create_and_test.py    # Stage 2
│   └── create_and_test.sh
└── templates/
    └── schema_template.json  # Starter schema for Step 2

See scripts/README.md for a one-page reference.

Workflow

Step 1 — Extract layout for representative documents

The model behind Content Understanding sees the text and structure the service extracts from your file, not the original pixels. Reviewing the layout output is the fastest way to know what labels and headings you can anchor your field descriptions to.

[ASK USER] "Point me at one of your sample documents (or a folder of them). I'll run layout extraction so we can see what the model will be looking at."

Run:

python .github/skills/cu-sdk-author-analyzer/scripts/extract_layout.py \
    --input <path-to-folder-or-file> \
    --output .local_only/layout/

This produces one <doc>.layout.md and one <doc>.layout.json per input. Open the .layout.md file in VS Code and look for the text anchors you want to extract from — labels ("Invoice #:"), section headings ("Bill To"), table headers, etc.

Reference: this is the same call pattern as samples/sample_analyze_binary.py using prebuilt-documentSearch.

Step 2 — Draft a JSON field schema

Start from the template instead of writing from scratch:

mkdir -p .local_only/schemas
cp .github/skills/cu-sdk-author-analyzer/templates/schema_template.json \
   .local_only/schemas/<name>_v1.json

Then edit .local_only/schemas/<name>_v1.json: set baseAnalyzerId, replace every REPLACE: placeholder, and add/remove fields. The schema is a JSON object with two required top-level keys:

• baseAnalyzerId — which prebuilt analyzer your custom analyzer extends. Use the table below.
• fieldSchema.fields — the named fields you want to extract.

[COPILOT] Read best practices before drafting fields. Before writing any field description, fetch the official CU best-practices page and apply its guidance:

🔗 https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/best-practices

Key principles that affect schema quality:

• Be specific and concrete in field descriptions — vague descriptions produce vague extractions.
• Use text anchors (labels, headings, neighbouring fields) — never visual cues like colour, font, or position-without-text-context. This matches the two-stage pipeline rule in cu-sdk-common-knowledge.
• Include format examples and alternative label names when a value can appear in multiple wordings or formats.
• Prefer "method": "extract" when the value appears verbatim; use "generate" only when the model needs to synthesise (summary, classification label) and "classify" for fixed enumerations.
• Keep the field count focused on what the user actually needs. Extra fields cost tokens and can dilute extraction quality on the fields that matter.

The template demonstrates all three extraction methods (extract, generate, classify) plus the nested object and array of objects shapes. Delete the example fields you don't need.

Template comment keys: any key whose name starts with _ (e.g. _comment, _optional_enableOcr) is stripped from the request body by create_and_test.py before it hits the service. Use them freely as inline documentation.

models.completion is effectively required: whenever fieldSchema is present, the service needs a completion model. Leave the models block in the template populated unless you've run samples/sample_update_defaults.py to set resource defaults. Omitting it fails with InvalidRequest: 'models.completion' is not set after a misleadingly successful [CREATE] log line. create_and_test.py prints a [WARN] if the schema is missing it.

Choosing baseAnalyzerId

See the prebuilt-analyzer table in cu-sdk-common-knowledge § Choosing baseAnalyzerId. The local validator (Step 3) rejects any value not on that list.

Example single-type schema

{
  "baseAnalyzerId": "prebuilt-document",
  "description": "Extract invoice header and totals.",
  "config": {
    "estimateFieldSourceAndConfidence": true,
    "returnDetails": true
  },
  "models": {
    "completion": "gpt-4.1",
    "embedding": "text-embedding-3-large"
  },
  "fieldSchema": {
    "name": "invoice_v1",
    "description": "Invoice header fields.",
    "fields": {
      "invoiceNumber": {
        "type": "string",
        "method": "extract",
        "description": "Invoice number printed near the 'Invoice #' label at the top of the page.",
        "estimateSourceAndConfidence": true
      },
      "totalAmount": {
        "type": "number",
        "method": "extract",
        "description": "Grand total at the bottom of the document; typically labelled 'Total' or 'Amount Due'. Excludes any 'Subtotal' value.",
        "estimateSourceAndConfidence": true
      }
    }
  }
}

Reference: see samples/sample_create_analyzer.py for the typed-model equivalent. The script accepts a JSON dict directly (the SDK's begin_create_analyzer is overloaded for both).

Field-description rule (two-stage pipeline): descriptions must reference text content and structure (labels, headings, neighbouring fields), not visual appearance (colour, font, size). See cu-sdk-common-knowledge § "two-stage pipeline".

Step 3 — Validate the schema locally

python .github/skills/cu-sdk-author-analyzer/scripts/create_and_test.py \
    --schema .local_only/schemas/invoice_v1.json \
    --input samples/sample_files/sample_invoice.pdf \
    --output .local_only/test_results/v1

The script runs the local validator first. If anything is wrong (unknown baseAnalyzerId, missing fieldSchema, malformed entries) it exits with code 2 before any service call.

Step 4 — Read the stdout summary

After the script finishes you get something like:

========================================================================
[SUMMARY]

category: (single)  (3 document segments)
------------------------------------------
  field                          fill rate   avg conf
  invoiceNumber                  100.0%      0.962
  totalAmount                     66.7%      0.481

lowest-confidence fields:
  0.461  totalAmount  (mixed_financial_docs)
  0.732  invoiceNumber  (mixed_financial_docs)
========================================================================

For each input document the script writes two files into --output:

• <doc>.json — full per-document AnalysisResult (fields, grounding, confidences).
• <doc>.llm.md — same result rendered via the SDK's to_llm_input helper: YAML front matter (category, page range, fields) plus the document text. Drop this straight into an LLM prompt, or skim it in VS Code for a fast human review.

Step 5 — Agent review and iterate

[COPILOT] After the summary prints, do the following automatically before asking the user anything:

1. Open each <doc>.llm.md (and the underlying <doc>.json for grounding) in .local_only/test_results/ and compare extracted field values against the source content. Use .local_only/layout/<doc>.layout.md as ground truth — it shows the text the model actually saw.
2. Flag any field where:
   • The extracted value looks wrong or is null when a value should be present
   • Confidence is below 0.7
   • The grounding location is unexpected
3. Present findings to the user with concrete diffs:
   • "Field invoiceNumber extracted 'INV-001' — does this look correct?"
   • "Field totalAmount was empty, but I see '$1,234.56' near the 'Total' label in the layout — should I tighten the description?"
4. For each correction the user confirms, append to .local_only/ground_truth_<schema-name>.json:

   [
     { "doc": "invoice_001.pdf", "field": "totalAmount", "correct_value": "1234.56" },
     { "doc": "invoice_002.pdf", "field": "invoiceDate", "correct_value": "2026-01-15" }
   ]
   

   This file is the agent's memory of correct answers across iterations.
5. Use the corrections to refine the field descriptions in a new schema version (.local_only/schemas/<name>_v2.json).
6. Re-run Step 3–4 with the new schema. The --reuse flag in create_and_test.py names analyzers by schema hash, so unchanged schemas are a no-op on the create side and a changed schema gets a fresh analyzer automatically.

Repeat until all key fields reach fill rate ≥ 80% and avg confidence ≥ 0.85, or the user is satisfied.

Stop and report to the user when any of:

• Targets are met (success) — then proceed to Step 6 to hand off.
• Three consecutive iterations show no improvement (need a different approach — different baseAnalyzerId, different method, or schema redesign — escalate to the user).
• The user signals they're done.

Step 6 — Hand off the finished analyzer

This step is required when Step 5 succeeds. Do not skip it. Without a clean handoff the user has a working analyzer in their resource but no idea what its ID is, where the final schema lives, or how to call it from their own code.

[COPILOT] When Step 5 reaches success, report the following to the user in one message before stopping:

1. Final analyzer ID — the actual ID printed by create_and_test.py on its last successful run (e.g. invoice_v3_a1b2c3d4 when --reuse is used). The user will need this to call the analyzer from their own code.
2. Final schema file — the path to the last iteration of the schema JSON (e.g. .local_only/schemas/invoice_v3.json). Recommend the user save it somewhere outside .local_only/ for future reference; they can also inspect any existing analyzer's schema directly via the SDK (see samples/sample_get_analyzer.py).
3. Next-step samples — point the user to:
   • samples/sample_create_analyzer.py — how to (re)create a custom analyzer from a schema JSON in their own code.
   • samples/sample_analyze_binary.py and samples/sample_analyze_url.py — how to call the analyzer on real input from their own code.
   Remind the user that the analyzer you just built is already deployed to their resource and ready to use directly via its ID — they only need to re-create it from the schema if they want to bootstrap it in another resource.

Step 7 — Clean up (optional)

By default the analyzer is kept in your resource so you can re-use it. Pass --ephemeral to delete it at the end of a run:

python .github/skills/cu-sdk-author-analyzer/scripts/create_and_test.py \
    --schema .local_only/schemas/invoice_v1.json \
    --input samples/sample_files/sample_invoice.pdf \
    --output .local_only/test_results/v1 \
    --ephemeral

Iteration helper — --reuse: add --reuse to name the analyzer by a sha1 of its schema (<schema-stem>_<hash[:8]>) and skip creation when an analyzer with that ID already exists. Re-running with the same schema is a no-op on the create side, so you don't pile up stale analyzers while iterating. Edit the schema → hash changes → new analyzer is created.

For explicit lifecycle management see samples/sample_get_analyzer.py, samples/sample_list_analyzers.py, samples/sample_update_analyzer.py, and samples/sample_delete_analyzer.py.

Exit codes

Code	Meaning
0	All documents analyzed successfully.
1	At least one service-side failure (network, throttling, invalid response).
2	Local user error — schema validator failure, missing flag, bad input path. No service call was made.

Related skills

• cu-sdk-setup — install the SDK, configure env.
• cu-sdk-sample-run — run one reference sample.
• cu-sdk-common-knowledge — service concepts and field-description rules.
• cu-sdk-author-analyzer-classify-route — multi-doc-type packets.