cu-sdk-author-analyzer-classify-route

name	cu-sdk-author-analyzer-classify-route
description	EXPERIMENTAL. Create and test a classify-and-route Azure AI Content Understanding pipeline for packets that contain multiple document types (e.g. invoice + bank statement + loan application in one PDF). Walks per-type schema authoring → outer classifier wiring → batch test → category-aware stdout summary using the typed ContentUnderstandingClient. Use when the user has mixed-document packets.

Author a Classify-and-Route Analyzer (mixed document packets)

[EXPERIMENTAL - REQUIRED FIRST MESSAGE]: Before asking questions, probing the environment, or running commands, tell the user: "This analyzer-authoring skill is experimental and may change. Review generated schemas and test results before relying on them."

Build a classify-and-route pipeline: one outer classifier analyzer that segments and labels a multi-document packet, plus one inner extractor analyzer per document type. The packet flows through the outer analyzer once; each segment is automatically routed to the matching inner analyzer for field extraction.

This is an iterative, human-in-the-loop workflow. You will typically run the schema → test → review cycle multiple times to refine both the outer classifier descriptions and each inner schema's field descriptions before you're happy with both classification accuracy and extraction quality.

[USE INSTEAD]: If every page in the user's documents is the same type (only invoices, only contracts, etc.), use cu-sdk-author-analyzer instead. Classify-and-route is for mixed packets.

[ASK USER] Modality check (first thing to confirm):

"Are you working with document files — PDFs or images of pages? This skill supports document modalities only."

• If the user says document → continue with this skill.
• If the user says audio, video, or image → stop this skill. Point them at the REST tutorial.

[COPILOT INTERACTION MODEL]: At each step marked with [ASK USER], pause execution and prompt the user before proceeding.

Prerequisites

Required: venv active, SDK installed, .env with CONTENTUNDERSTANDING_ENDPOINT (plus CONTENTUNDERSTANDING_KEY or az login), and samples/sample_update_defaults.py run once for this resource. Full setup lives in cu-sdk-setup.

[COPILOT] Probe first, then route on failure — do not duplicate setup logic here.

python -c "import sys; print('venv:', 'ok' if sys.prefix != sys.base_prefix else 'INACTIVE')"
( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_ENDPOINT=https?://' .env >/dev/null && echo 'endpoint: ok' ) || echo 'endpoint: MISSING'
( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_KEY=.+' .env >/dev/null && echo 'key: set' ) || echo 'key: empty'
az account show >/dev/null 2>&1 && echo 'az: ok' || echo 'az: not logged in'

Failure	Route to
venv INACTIVE or azure.ai.contentunderstanding import fails	cu-sdk-setup Steps 2–3, then resume
endpoint MISSING	cu-sdk-setup Step 4 (env vars), then resume
endpoint ok, key empty, az: not logged in	cu-sdk-setup Step 4 auth section (run az login or add CONTENTUNDERSTANDING_KEY to .env), then resume
All env checks pass but service calls fail with model errors	cu-sdk-setup Step 6 (sample_update_defaults.py), then resume
All ok	✅ Proceed to the Packet check below.

Never ask the user to paste an endpoint or API key into chat — they edit directly or run .

[ASK USER] Packet check:

1. "Does each document in your packet contain more than one type of form (e.g. an invoice page followed by a bank statement page)?" — if no, route to cu-sdk-author-analyzer.
2. "What types of documents appear in your packets?" — capture as the list of inner analyzers.

Architecture

                       ┌──────────────────────────────┐
   mixed packet  ───►  │  outer (classifier) analyzer │
                       │  baseAnalyzerId: prebuilt-…  │
                       │  config.enableSegment: true  │
                       │  config.contentCategories:   │
                       │    invoice          ────────►│──┐
                       │    bank_statement   ────────►│──┼──► per-segment fields
                       │    loan_application ────────►│──┘
                       └──────────────────────────────┘
                                                       │
                       inner analyzers (1 per type)    │
                       ───────────────────────────     ▼
                       invoice extractor   ◄──── routes here for invoice pages
                       bank statement ext. ◄──── routes here for bank pages
                       loan app extractor  ◄──── routes here for loan pages

Key rules (also captured in cu-sdk-common-knowledge § "classify-and-route"):

1. Category descriptions reference text anchors, not visual cues (matches the two-stage pipeline rule for fields).
2. config.enableSegment must be true so the classifier can carve up the packet before routing.
3. Inner analyzers must exist before the outer classifier is created. The provided script handles ordering automatically.
4. Category fill rate is per-category, not packet-wide. The script's stdout summary uses the right denominator.

Package directory

sdk/contentunderstanding/azure-ai-contentunderstanding

Scripts and templates

.github/skills/cu-sdk-author-analyzer-classify-route/
├── scripts/
│   ├── create_and_test_router.py
│   └── create_and_test_router.sh
└── templates/
    └── classifier_template.json   # Starter outer-classifier schema for Step 3

Workflow

Step 1 — Identify the document types

Run layout extraction (same as the single-type skill) on a representative packet to see the section headings:

python .github/skills/cu-sdk-author-analyzer/scripts/extract_layout.py \
    --input <packet.pdf> \
    --output .local_only/layout/

[COPILOT] Read .local_only/layout/<packet>.layout.md yourself and identify section headings, page breaks, and content shifts that suggest different document types. Then present your analysis to the user for confirmation — do not ask the user to read the layout cold.

Example presentation:

"Based on the layout in <packet>.layout.md I see these document types:

• Pages 1–2 — appears to be an invoice (top heading 'Invoice', contains 'Invoice #' label and a line-item table)
• Pages 3–4 — appears to be a bank statement (top heading 'Account Statement', contains account number and a transaction table)
• Page 5 — appears to be a loan application (top heading 'Loan Application', applicant fields)

Does this look right? Anything to add, remove, or rename before I draft schemas?"

Only fall back to a blank [ASK USER] ("What types do you see?") if the layout is too ambiguous to suggest types confidently.

Step 2 — Draft one inner schema per type

Treat each type as a single-doc-type analyzer (pick baseAnalyzerId from the table in cu-sdk-common-knowledge § Choosing baseAnalyzerId, then add fieldSchema.fields). Field descriptions follow the two-stage pipeline rule — reference text and structure, never visual appearance. See cu-sdk-author-analyzer Step 2 for the full field-schema walkthrough.

[COPILOT] Read best practices before drafting fields. Before writing any field description (in any inner schema or the outer classifier's category descriptions), fetch the official CU best-practices page and apply its guidance:

🔗 https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/best-practices

The same key principles apply here as in cu-sdk-author-analyzer: be specific and concrete, use text anchors (not visual cues), include alternative labels and format examples, prefer extract for verbatim values, and keep the field count focused.

Reference: samples/sample_create_classifier.py ships a complete worked example using samples/sample_files/mixed_financial_docs.pdf with three categories — Invoice, Bank_Statement, Loan_Application.

Step 3 — Draft the outer classifier schema

The outer schema has no fieldSchema. Its job is classification + routing. Start from the template:

mkdir -p .local_only/schemas
cp .github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json \
   .local_only/schemas/<name>_classifier_v1.json

Example after editing (descriptions intentionally generic so they survive minor wording variants — see the Category description rule below the example for guidance on writing your own):

{
  "baseAnalyzerId": "prebuilt-document",
  "description": "Classify mixed financial packets and route to per-type extractors.",
  "config": {
    "enableSegment": true,
    "omitContent": true,
    "contentCategories": {
      "invoice": {
        "description": "A commercial invoice. Look for an invoice / bill heading or label, an invoice number, line-item table with quantities and prices, and a total amount.",
        "analyzerId": "invoice"
      },
      "bank_statement": {
        "description": "A bank or account statement. Look for an account-statement heading, an account number, a statement period or date range, and a transaction table with running balance.",
        "analyzerId": "bank_statement"
      },
      "loan_application" 
         
         
      
    
  
   
     
     
  

The analyzerId value in each category is an alias that the script resolves at runtime, matching the --inner-schema alias=path flags you pass. Two exceptions skip alias resolution:

• Values starting with prebuilt- (e.g. prebuilt-invoice) are used as-is — no inner schema needed. Useful for routing a category at a service prebuilt extractor.
• Categories without an analyzerId at all are classification-only — the segment is labelled but no fields are extracted.

Why omitContent: true? When omitted, the service also returns the raw, un-segmented document content as an extra entry in contents. That entry has no category, no fields, and shows up in the summary as a confusing (uncategorized) row. Setting omitContent: true removes it.

Category description rule — the example values above are demo-specific.

The category descriptions in the JSON above are tied to the demo packet samples/sample_files/mixed_financial_docs.pdf (which uses the literal headings Invoice, Bank Statement, Loan Application). Do not copy them verbatim. When authoring for the user's own packet, write descriptions based on what you observed in the user's layout output from Step 1 — use the actual headings, labels, and structural markers from their documents.

Keep descriptions:

• Generic over surface form — describe the kind of content ("contains a transaction table with running balance") rather than hardcoding one specific header string, so minor wording variants still classify correctly.
• Concrete enough to be discriminative — include at least one anchor that distinguishes this category from the others in the packet.
• Text-anchored, not visual — reference headings, labels, and neighbouring text, never colour / font / position-without-text. Same reason as the field-description rule in cu-sdk-common-knowledge.

Step 4 — Validate, create, and batch-test

python .github/skills/cu-sdk-author-analyzer-classify-route/scripts/create_and_test_router.py \
    --outer-schema .local_only/schemas/classifier.json \
    --inner-schema invoice=.local_only/schemas/invoice.json \
    --inner-schema bank_statement=.local_only/schemas/bank_statement.json \
    --inner-schema loan_application=.local_only/schemas/loan_application.json \
    --input samples/sample_files/mixed_financial_docs.pdf \
    --output .local_only/test_results/v1

Shortcut — --schema-dir: if your inner schema filenames match the outer-schema category aliases (e.g. .local_only/schemas/invoice_v1.json for category invoice), replace every --inner-schema alias=path with a single --schema-dir .local_only/schemas/. The script picks the newest matching file per alias (alphabetical sort, so invoice_v2.json wins over invoice_v1.json).

Iteration helper — --reuse: add --reuse to name analyzers by a sha1 of their schema (<stem>_<hash[:8]>) and skip creation when an analyzer with that ID already exists. Re-running with the same schemas is a no-op on the create side, so you don't pile up stale analyzers while iterating. Edit a schema → hash changes → new analyzer is created.

The script:

1. Validates every schema (exits with code 2 if any fails — no service call made).
2. Errors out if the outer schema references an alias that has no matching --inner-schema, or if you supply an --inner-schema that no category uses.
3. Creates inner analyzers first, then patches and creates the outer classifier.
4. Analyzes every input file, writing one JSON per file under --output.
5. Prints a category-aware stdout summary (per-category fill rate uses each category's segment count, not the packet-wide total).

Step 5 — Read the category-aware summary

Example output:

========================================================================
[SUMMARY] (category-aware)

category: bank_statement  (1 segments)
--------------------------------------
  field                          fill rate   avg conf
  AccountNumber                  100.0%      0.918
  StatementPeriod                100.0%      0.882

category: invoice  (1 segments)
-------------------------------
  field                          fill rate   avg conf
  InvoiceNumber                  100.0%      0.962
  TotalAmount                    100.0%      0.531

category: loan_application  (1 segments)
----------------------------------------
  field                          fill rate   avg conf
  ApplicantName                  100.0%      0.875
  LoanAmount                     100.0%      0.799

lowest-confidence fields across all categories:
  0.531  [invoice] TotalAmount  (mixed_financial_docs)
  0.799  [loan_application] LoanAmount  (mixed_financial_docs)
  0.875  [loan_application] ApplicantName  (mixed_financial_docs)
========================================================================

For each input document the script writes two files into --output:

• <doc>.json — full AnalysisResult with all per-segment fields and grounding.
• <doc>.llm.md — the same result rendered via the SDK's to_llm_input helper. For classify-and-route, the helper expands each classified segment into its own block with the category in the YAML front matter, separated by ***** dividers — drop it into an LLM prompt or skim it in VS Code.

Template comment keys: any key whose name starts with _ (e.g. _comment, _optional_returnDetails) is stripped from both the outer and inner schemas before the request body is sent. Use them freely as inline documentation.

models.completion for inner schemas: each inner schema (which has a fieldSchema) needs models.completion set unless the resource has defaults configured via samples/sample_update_defaults.py. create_and_test_router.py prints a [WARN] per inner schema that is missing it, before any service call.

Step 6 — Agent review and iterate

[COPILOT] After the category-aware summary prints, do the following automatically before asking the user anything:

1. For each entry in result.contents[] in <doc>.json, verify the assigned category matches what the page actually is. Use .local_only/layout/<doc>.layout.md as ground truth.
   • Misclassified segment → the outer classifier's contentCategories.<key>.description needs strengthening, not the inner schema. Add a discriminating anchor from the layout to the category description and recreate the outer classifier.
2. For each correctly-classified segment, compare extracted field values against the layout the same way as the single-type skill (see cu-sdk-author-analyzer). Flag fields where:
   • The extracted value looks wrong or is null when a value should be present
   • Confidence is below 0.7
   • The grounding location is unexpected
3. Present a per-category diff to the user:
   • "In the invoice segment, field TotalAmount extracted '1234' — I see '$1,234.56' near the 'Total' label. Should I tighten the description?"
   • "Page 5 was classified as loan_application but the heading is actually 'Borrower Agreement' — should I add that wording to the loan_application category description?"
4. Record user-confirmed corrections in per-category ground-truth files (.local_only/ground_truth_<category>.json) so the agent remembers correct answers across iterations:

   [
     { "doc": "mixed_packet_1.pdf", "segment": 0, "field": "TotalAmount", "correct_value"  
   
   

Step 7 — Hand off the finished pipeline

This step is required when Step 6 succeeds. Do not skip it. Without a clean handoff the user has a working classify-and-route pipeline in their resource but no idea what the outer classifier ID is, which inner analyzers it routes to, where the final schemas live, or how to call it from their own code.

[COPILOT] When Step 6 reaches success, report the following to the user in one message before stopping:

1. All analyzer IDs built — list both the outer classifier and every inner extractor, with the actual IDs printed by create_and_test_router.py (e.g. classifier_v2_a1b2c3d4 → routes to invoice_v3_b2c3d4e5, bank_statement_v2_c3d4e5f6, loan_application_v3_d4e5f6a7). The user only needs to call the outer analyzer in their own code; it routes to the inner ones automatically.
2. All schema files — list the path to every schema JSON in the final iteration: the outer classifier and each inner schema (e.g. .local_only/schemas/classifier_v2.json, .local_only/schemas/invoice_v3.json, etc.). Recommend the user save them somewhere outside .local_only/ for future reference; they can also inspect any existing analyzer's schema directly via the SDK (see samples/sample_get_analyzer.py).
3. Next-step samples — point the user to:
   • samples/sample_create_classifier.py — how to (re)build the full classify-and-route pipeline from schema JSON in their own code (handles inner-first creation ordering and contentCategories.analyzerId wiring the same way our script did).
   • samples/sample_analyze_binary.py and samples/sample_analyze_url.py — how to call the analyzer on real input from their own code. Use the outer classifier's analyzer ID as analyzer_id.
   Remind the user that the analyzers you just built are already deployed to their resource and ready to use directly via their IDs — they only need to re-create them from the schemas if they want to bootstrap into another resource.

Step 8 — Clean up (optional)

By default the script leaves both the outer classifier and all inner analyzers in your resource so you can re-use them. Pass --ephemeral to delete all of them at the end of the run.

Exit codes

Code	Meaning
0	Every document analyzed successfully.
1	At least one service-side failure.
2	Local user error — schema validation, missing inner alias, bad path. No service call made.

Related skills

• cu-sdk-author-analyzer — single doc type.
• cu-sdk-common-knowledge — service concepts, two-stage pipeline, baseAnalyzerId table, classify-and-route rules.
• cu-sdk-sample-run — run individual SDK samples (e.g. sample_create_classifier.py) for deeper reference.
• cu-sdk-setup — install the SDK, configure env.