Validates and optionally fixes Agent 365 observability code so an agent actually emits MAC Activity telemetry. Use when an Agent 365 agent's telemetry isn't reaching MAC Activity or Defender, when spans export (HTTP 200 / "sent") but Activity stays empty, when the caller or agent id looks wrong in reporting, or to pre-flight whether an agent will emit MAC Activity before shipping. Diagnoses and fixes existing instrumentation; do NOT use it to add observability from scratch — use instrument-observability for that. Checks exporter activation, runtime agent-identity binding (vs blueprint id), the S2S FMI / OBO token shape, S2S vs OBO endpoint selection, the required semantic spans (invoke_agent/chat/execute_tool/output_messages), Activity attributes, and live blueprint grants and inheritance through read-only a365 Microsoft Graph diagnostics. Read-only by default; asks before applying safe fixes. Python, Node.js, and .NET.
Instrucciones de origen · Vista previa de solo lectura
name
a365-code-validator
description
Validates and optionally fixes Agent 365 observability code so an agent actually emits MAC Activity telemetry. Use when an Agent 365 agent's telemetry isn't reaching MAC Activity or Defender, when spans export (HTTP 200 / "sent") but Activity stays empty, when the caller or agent id looks wrong in reporting, or to pre-flight whether an agent will emit MAC Activity before shipping. Diagnoses and fixes existing instrumentation; do NOT use it to add observability from scratch — use instrument-observability for that. Checks exporter activation, runtime agent-identity binding (vs blueprint id), the S2S FMI / OBO token shape, S2S vs OBO endpoint selection, the required semantic spans (invoke_agent/chat/execute_tool/output_messages), Activity attributes, and live blueprint grants and inheritance through read-only a365 Microsoft Graph diagnostics. Read-only by default; asks before applying safe fixes. Python, Node.js, and .NET.
{"preToolUse":[{"type":"command","command":"node ${CLAUDE_PLUGIN_ROOT}/hooks/preToolUse/path-guard.js","timeout":5000}],"stop":[{"type":"command","command":"node ${CLAUDE_PLUGIN_ROOT}/hooks/stop/validate-a365-code-validator.js","timeout":15000},{"type":"prompt","prompt":"Verify the validation result was presented to the user with:\n1. exporter activation status;\n2. identity binding status (agent id vs blueprint id);\n3. semantic span coverage (invoke_agent/chat/execute_tool/output_messages);\n4. endpoint/token mode (S2S vs OBO);\n5. live blueprint grant and effective-inheritance status, or why the check was skipped;\n6. concrete next debug commands;\n7. if blockers were found, the user was asked whether to fix now, create a fix plan, or stop.\nReturn {\"ok\": false, \"reason\": \"<missing item>\"} if any item is absent.\nOtherwise return {\"ok\": true}.\n","timeout":30000}]}
A365 Code Validator
Trigger phrases — any of these will activate this skill automatically:
"validate a365 code"
"run a365 code validator"
"check a365 observability code"
"debug a365 activity missing"
"debug MAC activity for this agent"
"validate Agent 365 telemetry"
"check why Agent Activity is empty"
"check A365 exporter flags"
"validate gen_ai agent id"
"check A365 span types"
Overview
This skill performs a report-first validation of an existing agent codebase to answer:
"If this agent runs now, will its spans reach the A365 backend and become eligible for MAC Activity?"
It is designed for incident response. It checks the exact classes of bugs that cause
silent or confusing A365 Activity failures:
Exporter configured but not actually enabled.
gen_ai.agent.id set to a Blueprint ID instead of the runtime Agent Identity / Source Agent ID.
Generic HTTP/OpenAI spans emitted without A365 semantic operations.
S2S/OBO endpoint mismatch.
Missing token resolver or wrong S2S token shape.
Stale local Blueprint ID or missing Blueprint permissions in the target tenant.
Inheritable permission policy configured without grants on the Blueprint service principal.
MAC reporting expectations confused with raw ingest success.
No source code is modified during validation. After the report, the skill asks whether to
apply safe fixes, create a fix plan only, or stop.
Output is support-safe (applies to every phase)
This report may be pasted into a customer or partner thread, so it must be safe to share:
never print internal cluster/database names, Kusto queries, private endpoint or route
templates, correlation IDs, or real tenant/agent IDs. Report file/function names and public,
supportable signals only. This is non-negotiable — it holds even when a user asks for "just
the raw details."
Phase 0 — Create and Display Task List
Show the user this checklist BEFORE Phase 1. Exactly one task is in progress at a time.
Claude Code uses TaskCreate / TaskUpdate; VS Code Copilot Chat / GitHub Copilot CLI
should show and update a markdown checklist.
TaskCreate: "Detect stack and A365 artifacts"
TaskCreate: "Run static A365 code validator"
TaskCreate: "Inspect identity binding and semantic spans"
TaskCreate: "Check live blueprint permissions and runtime state"
TaskCreate: "Summarize blockers and next fixes"
TaskCreate: "Offer guided remediation"
Phase 1 — Detect Stack and A365 Artifacts
Mark task in progress: "Detect stack and A365 artifacts"
Read, in parallel when possible:
.a365-workspace-detection.local.json if present
a365.config.json
a365.generated.config.json
.env, .env.example, .env.production, .env.local
appsettings.json, appsettings.Development.json
package.json
requirements.txt, pyproject.toml
*.csproj
Detect:
Signal
Meaning
microsoft-opentelemetry
Python A365 distro present
@microsoft/opentelemetry
Node.js A365 distro present
Microsoft.OpenTelemetry
.NET A365 distro present
agentBlueprintId / agenticAppId
Distinguish Blueprint ID from runtime Agent Identity
Generic HTTP spans alone are not enough. Flag code that only sets baggage but never
creates or auto-generates supported semantic spans.
3.4 Activity/run context coverage
If the developer expects MAC Activity or Defender agent-activity views (not just exporter
HTTP 200), inspect whether real runs carry the public Agent 365 activity attributes:
Attribute
What to validate
gen_ai.agent.name
Human-readable agent name exists; otherwise UI can show a GUID/blank
gen_ai.conversation.id
Present on every span; for non-chat agents use a logical run/job/incident ID
microsoft.channel.name
Present on every span; use msteams for Teams, or a product/source value such as icm, web, scheduler
microsoft.session.id
Prefer a stable run/session ID; optional, but useful for grouping
user.id
Human caller OID for HumanToAgent; do not confuse it with the agent's own user
microsoft.agent.user.id
Agent's own Agent User OID when testing AI teammate / Agent-User / OBO
gen_ai.input.messages / gen_ai.output.messages
Present on invoke/chat/output spans where applicable
gen_ai.request.model, gen_ai.provider.name on chat spans
When reviewing a mixed app, separate the two objectives:
S2S/export success proves telemetry plumbing.
AI teammate/HumanToAgent or EventToAgent run context proves Activity/reporting eligibility.
Do not require Teams-specific values for autonomous agents; require equivalent logical run
context instead.
Caller identity caveat: for HumanToAgent, user.id is the caller ID used for "who ran
this agent" reporting. If it is missing or set to the agent identity / agent user instead
of the human caller object ID, export can succeed while caller/user Activity remains blank
or incomplete.
3.5 S2S vs OBO transport mode
Do not expose or require internal service URLs in the report. Validate only the structural
intent:
Auth mode
Structural expectation
S2S / application
Uses service-to-service export mode and a token for the runtime Agent Identity
OBO / agentic-user
Uses delegated export mode and a token whose delegated scope includes observability write
Report whether the code appears to select the correct transport mode (useS2SEndpoint /
a365_use_s2s_endpoint / UseS2SEndpoint) for S2S, without printing backend route templates.
For Python S2S, prefer a365_use_s2s_endpoint=True in code rather than relying only on
A365_USE_S2S_ENDPOINT=true in runtime environment. Also check the token shape: the S2S
observability token must come from the 3-hop FMI exchange (principal == runtime Agent Identity),
not a bare client-credentials call — see checklist §5 for the failure signatures.
3.6 Blueprint permission inheritance wording
Do not claim that application app roles can never inherit from blueprints. Current public
Entra Agent ID docs describe both inheritableScopes and inheritableRoles. If a repo or
tenant shows delegated scope inheritance working but S2S app roles requiring direct assignment,
report it as a provisioning/configuration behavior to verify, not as the intended scalable model.
Mark task complete.
Phase 4 — Check Live Blueprint Permissions and Prepare Runtime Verification
Mark task in progress: "Check live blueprint permissions and runtime state"
Live Blueprint permission check (read-only Microsoft Graph)
The static validator can only inspect files. When the project has a non-empty
agentBlueprintId and tenantId, complement it with the a365 CLI's read-only Microsoft
Graph diagnostics. These commands verify the tenant state without changing permissions:
blueprint-scopes lists the delegated scopes and application roles actually granted on
the Blueprint service principal. This is the authoritative permissions view; do not infer
grants from requiredResourceAccess.
inheritance verifies that every configured resource uses kind=allAllowed for scopes and
roles and that the Blueprint service principal has something to inherit. It exits non-zero
when any resource has Effective inheritance: NONE or BROKEN.
Run these checks by default only when:
a365 query-entra inheritance --help confirms the installed CLI supports the command,
an existing Azure/a365 sign-in is available for the target tenant, and
the check will not trigger a new interactive login.
The validator must not install or update the CLI, start an interactive sign-in, run
a365 setup permissions, or issue Graph POST/PATCH/DELETE requests. If a prerequisite is
missing, report Blueprint permissions as not checked, explain why, and provide the two
commands for the developer to run after a365-setup refreshes the CLI/login.
If the local config is missing, copied from another project, or suspected stale, use the
read-only name-based resolver instead:
Compare the live Blueprint ID printed by the command with the local agentBlueprintId, but
do not print either real ID in the support-safe report:
Result
Classification
Live and local Blueprint IDs resolve to the same Blueprint
current
Local ID is not found, but name-based lookup resolves a different Blueprint
critical stale/copied config
Neither lookup resolves a Blueprint
high tenant/setup gap
Graph returns 401/403 or the caller lacks the required Entra role
not checked authentication/authorization gap, not proof that the Blueprint is broken
Interpret permissions in the context of authMode:
Auth mode
Required observability grant
obo / agentic-user
Delegated Agent365.Observability.OtelWrite
s2s
Application role Agent365.Observability.OtelWrite
Unknown / mixed
Report which delegated scopes and app roles exist; do not assume the intended side
Treat a missing required OtelWrite grant as critical. Treat a non-zero inheritance result
for another required resource as high. Permission names and resource display names are safe
to summarize, but redact tenant, Blueprint, application, service-principal, and agent IDs.
For remediation, report the CLI guidance without applying it:
Run a365 setup requirements when the output points to a missing wids claim or CLI consent.
Run the relevant a365 setup permissions ... flow as a Global Administrator to reconcile
grants and inheritance.
For a stale Blueprint ID, hand off to a365-setup so the developer can explicitly choose the
correct reuse/re-run/fresh path. Do not rewrite the ID automatically.
If backend verification is required, keep customer-facing validation on public/supportable
surfaces:
SDK/exporter logs show export success and no chunk failures,
direct OTel responses have partialSuccess empty/null where applicable,
Microsoft Defender Advanced Hunting CloudAppEvents shows the expected AgentId /
TargetAgentId / AlternateId after indexing.
If the user is on an internal service team, they may also use their team's approved
dashboard/playbook to confirm request acceptance and downstream delivery, but do not print
those internal queries or endpoints in the validator report.
(Report only support-safe signals — see Output is support-safe above.)
If code checks pass but Activity is still empty — verify tenant-side
Most "exporter returns 200/sent but nothing shows in MAC" incidents are not code bugs.
When the code checks above pass, have the user confirm, in the target tenant:
Provenance (verified 2026-07-06): the observability resource-SP GUID, the
Agent365.Observability.OtelWrite scope, and the ingestion-lag figure below are preview-era
Agent 365 facts and will change. Re-verify against current Agent 365 onboarding docs before
quoting them to a customer.
Licensing state — confirm the target tenant/user meets the current Agent 365 licensing
prerequisites from public onboarding docs.
Tenant enrollment state — confirm Agent 365 is enabled for the target tenant through the
supported onboarding or admin process.
Observability resource SP present — az ad sp show --id 9b975845-388f-4429-889e-eab1ef63949c
returns a service principal in the tenant. A 404 / AADSTS500011 ("resource principal … not
found") means the observability app isn't provisioned there — an onboarding step, not a code fix.
Ingestion lag — Defender CloudAppEvents populates before the admin center; give it ~5 min.
Mark task complete: "Check live blueprint permissions and runtime state".
Phase 5 — Summarize Blockers and Next Fixes
Mark task in progress: "Summarize blockers and next fixes"
Return a concise report. Default to the short format below; include the optional details block
only when the user asks for deeper debugging or when a blocker needs a code pointer to be
actionable.
## A365 Code Validator Result**Status:** PASS / PASS WITH RISKS / BLOCKED
**Blockers**1.**<shorttitle>** — <one-sentenceimpact>.
Fix: <one-sentencefix>.
2.**<shorttitle>** — <one-sentenceimpact>.
Fix: <one-sentencefix>.
**High-risk checks**-<shortitem> — <whytoverify>.
**Identity binding**- Runtime agent id: <source / missing / fromconfig>- Blueprint id: <source / missing / fromconfig>- Verdict: <correct | wrong | unclear>**Blueprint permissions**- Live Blueprint: <current | stale/missing | notchecked>- Grants: <verifiedsummary | missingrequiredpermission | notchecked>- Effective inheritance: <OK | NONE | BROKEN | notchecked>**Semantic spans**- Found: <invoke_agent/chat/execute_tool/output_messages or "none">- Missing: <opsor "none">**Recommended fix order**1.<fixrootcausefirst>2.<thenfixconfig/env>3.<thenverifyspans/identity>**Optional details**- Files: `<file>:<line or function>`, ...
- Blueprint check: <read-onlycommandresultorreasonitwasskipped>- Runtime check: <local/App Service env check or SDK log grep>- Backend verification: use the team's approved telemetry playbook; do not paste internal
cluster names, private endpoints, or correlation IDs into this report.
Style rules:
Keep the default report to 8 bullets or fewer.
Put details under Optional details, not in the main blocker list.
Use exact file/function names for evidence, but do not paste long code blocks.
Redact per Output is support-safe above (no internal infra, tenant/agent IDs, or correlation IDs).
A skipped live Graph check is a verification gap, not automatically a code blocker.
Do not suggest a365 publish unless the agent is an AI Teammate / M365 package flow;
blueprint-based observability does not use a365 publish.
Mark task complete.
Phase 6 — Offer Guided Remediation
Mark task in progress: "Offer guided remediation"
If there are no critical or high findings, say:
No blocking fixes needed. Keep the runtime verification commands handy for live validation.
Then mark the task complete and stop.
If there are critical or high findings, ask the user with AskUserQuestion:
I found blockers. What would you like me to do next?
Options:
Option
Meaning
apply_safe_fixes
Apply only low-risk, deterministic code/config-template fixes
make_fix_plan
Produce an implementation plan, no edits
report_only
Stop after the report
Default: make_fix_plan.
Safe fixes allowed after confirmation
Only apply these automatically when the user chooses apply_safe_fixes:
Blueprint permissions are never a safe code fix. Do not run a365 setup permissions,
mutate Graph, grant admin consent, or rewrite a Blueprint ID from this skill. Report the
mismatch and hand off to a365-setup or the appropriate administrator.
Python exporter kwarg
If code calls use_microsoft_opentelemetry(...) or builds kwargs for that call and already
sets enable_a365=True, add the explicit exporter flag:
Add missing, commented or empty template entries only:
ENABLE_A365_OBSERVABILITY_EXPORTER=true
For S2S mode, add the version-appropriate S2S transport setting only if the code/reference for
that project already names it. Do not invent a setting name.
Config field scaffolding for runtime agent identity
If the project already has a settings/config class, add a clearly named optional field such as
agent_identity_app_id / AGENT_IDENTITY_APP_ID.
Use an empty default or placeholder; never hardcode real IDs.
**Applied fixes**- ...
**Still needs user input**- ...
**Validation to rerun**-`validate a365 code`- runtime exporter log check
Run the smallest available project tests if the edited repo has an obvious test command and the
user did not decline validation. Otherwise, state that no obvious test command was run.
