alibabacloud-cadt-probe

name

alibabacloud-cadt-probe

description

Use CADT probe service to discover cloud resources under an Alibaba Cloud account. Use this skill when users need to: (1) Discover existing cloud resources via CADT probe (2) Inventory existing resource list (3) Export resource list for CADT architecture import (4) Understand resource distribution within an account (5) Query related/associated resources for a specific resource (e.g., find security groups and vswitches linked to an ECS instance) Trigger words: resource probe, CADT probe, probe, resource inventory, list resources, view cloud resources, related resources, associated resources, resource relationships

CADT Alibaba Cloud Resource Probe

Use the CADT probe service (BPStudio API) to discover cloud resources under an Alibaba Cloud account.

Quick Start

Prerequisites

pip install alibabacloud_bpstudio20210931==8.1.1 alibabacloud_credentials==1.0.5

Configure credentials using the Alibaba Cloud default credential chain. Choose one of the following methods:

• Run aliyun configure to set up a credential profile
• Set environment variables ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET (auto-detected by the default chain)
• Use an ECS instance RAM role (when running on ECS)
• Configure ~/.alibabacloud/credentials file

Pre-conditions

The Cloud Config service must be enabled in the Alibaba Cloud console, as CADT probe relies on resource data provided by Cloud Config:

• Enable at: https://config.console.aliyun.com/
• Monitoring scope must be set to "All resources in account"
• After initial activation, wait 30-60 minutes for the initial scan to complete

RAM Policy

This skill requires the following RAM permissions to execute BPStudio probe operations and discover cloud resources across 20+ Alibaba Cloud services.

For the complete permission list including a least-privilege RAM policy JSON, see references/ram-policies.md.

Key permissions:

Service	Action	Purpose
BPStudio	bpstudio:ExecuteOperation	Execute all probe sync operations (CreateOneClickJob, GetOneClickStatus, GetProbeResourceSummary, GetProbeRelatedItems, GetLastProbeTime, GetProbeRegions, GetProbeResult)
Config	config:ListDiscoveredResources + 5 others	Resource discovery via Cloud Config
VPC / SLB / ALB / NLB / ACK / ...	Various Describe/List actions	Product-specific resource discovery and relation queries

[MUST] Permission Failure Handling: When any command or API call fails due to permission errors at any point during execution, follow this process:

1. Read references/ram-policies.md to get the full list of permissions required by this SKILL
2. Use ram-permission-diagnose skill to guide the user through requesting the necessary permissions
3. Pause and wait until the user confirms that the required permissions have been granted

Run Probe

python scripts/probe.py

Parameters

Parameter	Required	Description
--regions	No	Comma-separated list of regions to probe and filter results (e.g., cn-hangzhou,cn-beijing). Probes all regions when not specified
--output	No	Output JSON file path. Prints to console only when not specified
--list-types	No	Comma-separated resource type filter list (serviceTypes) for GetProbeResult / GetProbeResourceSummary; empty means no filter
--skip-probe	No	Skip task creation and polling, directly pull existing probe results (suitable for previously probed scenarios)
-q, --quiet	No	Quiet mode: suppress console decorative output, only output pure JSON to stdout (suitable for Agent/script calls). [debug] output is disabled by default in this mode
--debug	No	Enable debug output: print [debug] details of API responses (with full response body) to stderr. Disabled by default in quiet mode (-q)
--summary	No	Summary mode: only output aggregated statistics (skip GetProbeResult detailed resource list), significantly reducing output size
--keep-attributes	No	Retain the original attributes field for each resource (stripped by default to reduce output size)
--related-resource-id	No	Resource ID for querying related resources (GetProbeRelatedItems). When specified, skips full probe and only queries related resources for this resource (e.g., i-xxx for ECS, sg-xxx for security group, vpc-xxx for VPC)
--related-types	No	Comma-separated resource type filter for --related-resource-id (e.g., security_group,vswitch). Empty means return all related resource types

Usage Examples

1. Basic Probe

python scripts/probe.py

2. Export JSON Results

python scripts/probe.py --output probe_result.json

3. Skip Probe and Pull Existing Results

If you have previously probed (resource data is cached on the server side), you can directly pull existing results without creating a new task:

python scripts/probe.py --skip-probe
python scripts/probe.py --skip-probe --output result.json
python scripts/probe.py --skip-probe --list-types ecs,vpc

4. Agent Mode (Quiet Mode)

Use the -q flag to suppress all human-readable output, printing only pure JSON to stdout for easy Agent or script parsing. By default, the attributes field of each resource is stripped to reduce output size:

python scripts/probe.py -q
python scripts/probe.py -q --output result.json
python scripts/probe.py -q --regions cn-hangzhou --list-types ecs,vpc

To retain full attributes, use --keep-attributes:

python scripts/probe.py -q --keep-attributes

Output Auto-Degradation (Agent Mode)

When the JSON output is large (e.g., 127 security groups with Permissions arrays + 100 vswitches), the script automatically degrades output to prevent terminal truncation:

Output Size	Behavior
≤ 50KB	Normal JSON (with indentation) to stdout
50KB ~ 200KB	Compact JSON (no indentation) to stdout, warning on stderr
> 200KB	Full data written to file, stdout outputs a summary JSON only

When output exceeds 200KB, stdout returns a summary like:

{
  "status": "output_too_large",
  "output_file": "/path/to/file.json",
  "total_resources": 246,
  "by_type": {"security_group": 127, "vswitch": 100, "ecs": 19},
  "hint": "Read the output file for full results"
}

Recommendation: Always use --output to write full results to a file, then read the file for parsing. This avoids any output size issues:

python scripts/probe.py -q --output result.json

5. Summary Mode (Aggregated Statistics Only)

When there are many resources, use --summary to skip the detailed resource list and only output aggregated statistics by region/type, suitable for dashboard analysis:

python scripts/probe.py --summary
python scripts/probe.py -q --summary
python scripts/probe.py --summary --output summary.json

6. Query Related Resources for a Specific Resource

Query all resources related to a specific resource (e.g., ECS instance's security groups, vswitches, etc.):

python scripts/probe.py --related-resource-id i-xxx --related-types security_group,vswitch --output related_items_ecs.json

This calls the GetProbeRelatedItems API, which returns all resources topologically related to the specified resource in the probe results. It does NOT create a new probe task. The attributes field of each related resource is stripped by default (same as probe results), which significantly reduces output size for resources like security groups that contain large Permissions arrays.

Two-step workflow (get a resource ID first, then query its related resources):

# Step 1: Get resource list (use --skip-probe if already probed, -q for machine-readable output)
python scripts/probe.py --skip-probe -q --list-types ecs --regions cn-hangzhou

# Step 2: Pick any instance ID from Step 1 output, then query its related resources
python scripts/probe.py --related-resource-id i-bp1xxxxx --related-types security_group,vswitch --output related_items_ecs.json

Supported --related-types values (same as --list-types resource type keys):

• security_group — Security Group
• vswitch — VSwitch
• vpc — Virtual Private Cloud
• disk — Cloud Disk
• eni — Elastic Network Interface
• eip — Elastic IP Address
• ... (any resource type key listed in the supported resource types)

Leave --related-types empty to return all related resource types:

python scripts/probe.py --related-resource-id i-xxx --output all_related.json

Output format (when using --related-resource-id):

{
  "mode": "related-items",
  "query_time": "2025-04-17 16:00:00",
  "resourceId": "i-bp1xxxxx",
  "serviceTypes": ["security_group", "vswitch"],
  "related_items": { ... }
}

How It Works

The script implements resource probing through a two-phase flow using the BPStudio SDK:

Phase 1: Submit Probe Task

Call ExecuteOperationSync API:

• ServiceType = probe
• Operation = CreateOneClickJob
• Attributes = {"regions": [...]}

The API returns an operation_id (task ID).

Phase 2: Poll for Results

Call ExecuteOperationSync API:

• ServiceType = probe
• Operation = GetOneClickStatus
• Attributes = {"jobId": <jobId>}
• Poll every 10 seconds, timeout after 30 minutes, until the task completes
• When status == "SUCCESS", the arguments field contains the probed resource data (JSON)
• When status == "FAILURE", the message field contains the error information

Execution Modes

• Default (full probe): CreateOneClickJob → poll GetOneClickStatus (every 10s, timeout 30min) until SUCCESS → GetProbeResourceSummary → GetProbeResult → GetLastProbeTime + GetProbeRegions (cache validation) → output
• --skip-probe: Skip CreateOneClickJob and polling, directly pull cached results via GetProbeResourceSummary → GetProbeResult → cache validation (cache_info included in output)
• --related-resource-id: Skip full probe, directly call GetProbeRelatedItems with resourceId and optional serviceTypes filter → output. Requires a prior probe run; empty result if no probe data exists.

Cache Validation (All Modes)

After fetching results, the script always calls GetLastProbeTime and GetProbeRegions to validate the cache state:

• cache_info (only in --skip-probe mode): Contains last_probe_minutes_ago (minutes since last probe, -1 if no valid job) and cached_regions (list of region IDs in cache)
• cache_warnings (all modes): Array of warning strings when:
  • No valid probe job exists (lastTime == -1)
  • --regions includes regions not present in the cache → results for those regions will be empty

Output Description

Console Output

======================================================================
CADT Resource Probe (BPStudio)
======================================================================
  Probe time:   2025-04-17 15:30:00
======================================================================

Submitting probe task...
  Task ID: op-xxxxxxxxxxxxx
  Waiting for probe to complete...
  Current status: Running (elapsed 5s)
  Current status: SUCCESS (elapsed 15s)
  Probe complete!

======================================================================
Probe Results
======================================================================
<Resource list details>
======================================================================

JSON Output

When using the --output flag, a full JSON file is written:

{
  "regions": ["cn-hangzhou", "cn-beijing"],
  "probe_time": "2025-04-17 15:30:00",
  "job_id": "12345",
  "progress": {"status": "SUCCESS"},
  "summary": { ... },
  "probe_result": { ... },
  "probe_result_filter": {
    "regions": "all",
    "serviceType": "all"
  }
}

--skip-probe Output (with cache_info)

When using --skip-probe, the output includes a cache_info field:

{
  "regions": ["cn-hangzhou"],
  "probe_time": "2025-04-17 16:00:00",
  "job_id": "",
  "progress": {},
  "cache_info": {
    "last_probe_minutes_ago": 45,
    "cached_regions": ["cn-hangzhou", "cn-beijing"]
  },
  "summary": { ... },
  "probe_result": { ... },
  "probe_result_filter": { ... }
}

--related-resource-id Output

When using --related-resource-id, the output structure is different from the full probe output:

{
  "mode": "related-items",
  "query_time": "2025-04-17 16:00:00",
  "resourceId": "i-bp1xxxxx",
  "serviceTypes": ["security_group", "vswitch"],
  "related_items": {
    "list": [
      { "resourceId": "sg-xxx", "resourceType": "security_group", ... },
      { "resourceId": "vsw-xxx", "resourceType": "vswitch", ... }
    ]
  }
}

Key fields:

• mode: Always "related-items" for this query type
• resourceId: The resource ID that was queried
• serviceTypes: The type filters applied (empty list means no filter)
• related_items: Raw response from GetProbeRelatedItems, containing a list of related resources

_relationships (Auto-extracted Relationship IDs)

Each resource in probe_result.list and related_items.list automatically includes a _relationships field (when applicable) that extracts cross-resource reference IDs from the nested attributes JSON. This field is added before attributes are stripped, so it is always available even without --keep-attributes.

Example — an ECS instance resource:

{
  "resourceId": "i-bp1xxxxx",
  "resourceType": "ecs",
  "_relationships": {
    "security_group_ids": ["sg-abc", "sg-def"],
    "vswitch_id": "vsw-123",
    "vpc_id": "vpc-456"
  }
}

Supported resource types and extracted relationships:

Resource Type	Extracted Fields
ecs	security_group_ids, vswitch_id, vpc_id
eni	security_group_ids, vswitch_id, vpc_id
rds, polardb, dds, kvstore, tair	vpc_id, vswitch_id
clb, slb, alb, nlb	vpc_id, vswitch_id (where applicable)
nat, vpn	vpc_id, vswitch_id (where applicable)
nas, hbase, clickhouse, lindorm, adb	vpc_id, vswitch_id
elasticsearch, kafka, emr, mse	vpc_id, vswitch_id (where applicable)
disk	instance_id (references ECS)
eip	instance_id (bound resource)

If a resource type is not in the table or no relationship fields are found, _relationships is omitted.

cache_warnings

When cache issues are detected, a cache_warnings array is added to the output (in all modes):

{
  "cache_warnings": [
    "Requested region(s) cn-shenzhen not in probe cache — results for these regions will be empty. Cached regions: cn-hangzhou, cn-beijing"
  ]
}

Observability

User-Agent

All Alibaba Cloud API calls made by this skill (SDK and CLI) include a custom User-Agent:

--user-agent AlibabaCloud-Agent-Skills/alibabacloud-cadt-probe/{session-id}

Rule: Every aliyun CLI command that calls a cloud API MUST include the --user-agent flag above. Local utility commands (e.g. configure, plugin, version) do not support this flag and should be excluded.

Session ID

• Source: environment variable SKILL_SESSION_ID
• Fallback: if SKILL_SESSION_ID is not set, a random 32-character hex UUID is generated
• Scope: one session ID per invocation (created when the BPStudio client is initialized)

Exit Codes

Exit Code	Meaning
0	Success
1	Failure (credentials not configured, probe task submission failed, API error, etc.)
2	Region cache miss — --regions includes regions not present in the probe cache. Results are still output, but the Agent should be aware that those regions returned empty data due to cache gaps, not because the regions have no resources.

Error Handling

Credentials Not Configured

Error: Unable to get credential. Please configure the default credential chain.

Probe Task Submission Failed

Check:

• Whether credentials are correct
• Whether the account has BPStudio operation permissions (see references/ram-policies.md for the full permission list)
• Whether the Cloud Config service has been enabled

If the error is a permission error, follow the Permission Failure Handling process described in the RAM Policy section above.

Probe Task Timeout

Default timeout is 30 minutes (1800 seconds). If it times out, the script outputs the current status and task ID, which can be used to query later.

Cache Warnings (stderr)

Cache warnings are printed to stderr and included in the JSON output as cache_warnings:

• No valid probe job found in cache — results may be empty or stale
• Requested region(s) <regions> not in probe cache — results for these regions will be empty. Cached regions: <cached>

Supported Resource Types

CADT probe supports 60+ Alibaba Cloud resource types. See resource_types.md for the full list.

File Structure

alibabacloud-cadt-probe/
├── SKILL.md                    # This file
├── references/
│   ├── ram-policies.md         # Required RAM permissions
│   └── resource_types.md       # Supported resource types reference
└── scripts/
    └── probe.py                # Core probe script

Dependencies

alibabacloud_bpstudio20210931==8.1.1
alibabacloud_credentials==1.0.5