ado-assessment

name	ado-assessment
description	Review an Azure DevOps organization or project for adherence to Azure DevOps best practices. Use when: running an ADO Quick Review, evaluating org or project configuration, checking adoption of recommended settings, identifying improvement opportunities, ADO best practices, org review, project review.

Azure DevOps Quick Review

Review an Azure DevOps organization or project for adherence to Azure DevOps best practices by collecting settings via Azure CLI / REST API and evaluating them against documented Microsoft recommendations.

This skill does NOT call AzSK.ADO. It brings best-practice knowledge derived from those controls and uses native Azure CLI and REST API calls to collect and evaluate settings.

Workflow

Phase 1 — Collect Settings

Gather the current configuration of an ADO organization or project using az devops CLI commands and REST API calls. The reference files below contain the exact commands and API endpoints for each resource type.

Phase 2 — Assess Against Best Practices

Compare collected settings against the best-practice controls documented in the reference files. Each control has a severity (High/Medium/Low), a rationale, and remediation guidance. Generate a report showing which best practices have been adopted and which remain as improvement opportunities.

Prerequisites

# Azure CLI 2.81.0+
az version

# Azure DevOps extension
az extension add --name azure-devops

# Login
az login

# Set defaults
az devops configure --defaults organization=https://dev.azure.com/<YourOrg> project=<YourProject>

Authentication

# Interactive login
az login

# PAT-based login (for automation)
$env:AZURE_DEVOPS_EXT_PAT = '<your-pat>'
az devops login

# Get REST API bearer token
$token = az account get-access-token --resource "499b84ac-1321-427f-aa17-267ca6975798" --query accessToken -o tsv

Resource Types Covered

Resource	Controls	Scope
Organization	35+ controls	Auth, users, extensions, pipelines, feeds, audit
Project	25+ controls	Visibility, admins, pipelines, permissions, credential scanning
Build Pipelines	20+ controls	Secrets, permissions, forks, branches, task groups
Release Pipelines	12+ controls	Secrets, approvals, permissions, task groups
Service Connections	15+ controls	Auth, scope, access, permissions, branches
Agent Pools	10+ controls	Permissions, auto-provision, patching, secrets
Repos / Feeds / Secure Files / Environments	20+ controls	Access, permissions, branches, approvals
Variable Groups	10+ controls	Secrets, permissions, access, branches
Users / PATs	6+ controls	PAT scope, expiry, alternate credentials

Each resource is evaluated against documented Microsoft best practices.

Reference Files

Read the relevant reference file based on the user's assessment scope.

File	Keywords	Covers
references/org-settings.md	organization, org, AAD, external users, public projects, extensions, audit, OAuth, SSH, feeds, conditional access, admin	Organization-level settings — collection commands and best-practice controls
references/project-settings.md	project, visibility, admin, credential scanner, permissions, pipeline scoping, inactive	Project-level settings — collection commands and best-practice controls
references/build-controls.md	build, pipeline, CI, secrets, fork, task group, variable, branch, YAML	Build pipeline best-practice controls — collection commands and assessments
references/release-controls.md	release, deployment, approval, CD, production, stage	Release pipeline best-practice controls — collection commands and assessments
references/service-connection-controls.md	service connection, endpoint, SPN, certificate, subscription, ARM	Service connection best-practice controls — collection commands and assessments
references/agent-pool-controls.md	agent, pool, self-hosted, auto-provision, auto-update	Agent pool best-practice controls — collection commands and assessments
references/repo-feed-environment-controls.md	repository, repo, feed, artifact, secure file, environment, branch protection	Repos, feeds, secure files, and environment best-practice controls
references/variable-group-controls.md	variable group, secret, key vault, linked	Variable group best-practice controls
references/user-pat-controls.md	user, PAT, personal access token, alternate credentials, inactive	User and PAT hygiene best-practice controls
references/errors.md	error, 401, 403, permission denied, troubleshoot	Common errors and troubleshooting

Assessment Output Format

When reporting results, use this format for each control:

| Status | Severity | Control | Finding |
|--------|----------|---------|---------|
| PASS   | High     | AAD authentication enabled | Organization uses AAD-backed auth |
| FAIL   | High     | External user access | 3 external users found with access |
| FAIL   | Medium   | Audit streaming | Audit streaming is not configured |

Summarize at the end with counts: X PASS | Y FAIL | Z NOT CHECKED

Saving Results

After the assessment completes, save the results as Markdown files in the workspace root. Create separate files for the organization and each project:

File	Contents
{org-name}-org-assessment.md	Organization-level controls (auth, users, extensions, pipelines, feeds, audit, agent pools, PATs)
{org-name}-{project-name}-assessment.md	Project-level controls (visibility, admins, pipeline settings, permissions) plus all project-scoped resources: build pipelines, release pipelines, service connections, repos, feeds, environments, variable groups, secure files

File naming

• Use the organization short name and project name, lowercased and with spaces replaced by hyphens.
• Examples for org pbi-demo with projects eShop Web Demo and CL-DEMO:
  • pbi-demo-org-assessment.md
  • pbi-demo-eshop-web-demo-assessment.md
  • pbi-demo-cl-demo-assessment.md

File structure

Each file must include:

1. Header — assessment date, organization/project name, assessor
2. Control results table — the standard Status | Severity | Control | Finding table
3. Summary counts — X PASS | Y FAIL | Z NOT CHECKED
4. Critical findings — list of FAIL items sorted by severity (High first) with remediation steps

When to write

Write the files at the end of Phase 2 (after all controls are assessed and results are shown to the user). Always create the files — do not ask for confirmation.