alibabacloud-pai-node-management

name	alibabacloud-pai-node-management
domain	aiops
description	Manage Alibaba Cloud PAI compute Nodes — list nodes within a ResourceGroup or Quota, inspect a node's status, hardware spec (GPU / CPU / memory) and resource usage, and perform safety-gated maintenance operations (cordon, uncordon, drain) with explicit two-step confirmation. Use when the user asks to list / get / inspect PAI nodes, check node status, find a node's MachineGroup or ResourceGroup, or cordon / uncordon / drain a node. Out of scope: node power changes (start / stop / reboot), node replacement / migration, per-node GPU metrics, and per-node pod listing — these are platform-managed and require escalation to the platform Ops team / Lingjun ops ticket / account manager. Trigger phrases: "PAI node", "list nodes", "node status", "GPU node", "Lingjun node", "cordon node", "uncordon node", "drain node", "isolate node for maintenance", "PAI 节点", "查看节点", "隔离节点", "恢复节点".
required_permissions	["pai:ListNodes","pai:GetResourceGroup","pai:ListResourceGroups","pai:ListQuotas","pai:GetQuota","pai:UpdateResourceGroup","pai:UpdateQuota"]

PAI Node Management

0. Observability (MUST follow for every aliyun command)

Upon loading this skill, generate a random session ID (32-char hex string) once for the entire session. Use it as {session-id} below.

Rule: Every aliyun CLI command that calls a cloud API MUST include the --user-agent flag. Local utility commands (e.g. configure, plugin, version) do not support this flag and should be excluded.

--user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-node-management/{session-id}

Example (assuming session-id is a]1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6):

aliyun ecs describe-instances --biz-region-id cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-node-management/a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6

Do not skip, alter the format, or omit --user-agent on any aliyun command invocation.

Plugin freshness (one-time per session, optional system command — NO --user-agent):

aliyun plugin update

Every command omits the --user-agent flag for readability. The agent MUST append it to every actual business API invocation following the rules above. Missing or malformed --user-agent on a business API call is a workflow violation.

1. Scenario

Inspect and operate Alibaba Cloud PAI Nodes via the aliyun paistudio CLI plugin: enumerate nodes, single-node detail lookup, and cordon / uncordon / drain. Hierarchy: ResourceGroup → MachineGroup → Node(s).

Out of scope (refuse outright; do NOT route to alternative transports — see references/not-implementable.md and references/refusal-patterns.md):

• Node power (start / stop / reboot), replace, migration.
• MachineGroup deletion / release.
• Per-node metrics (get-node-gpu-metrics) and per-node pod listing (list-node-pods) — not in public CLI.

🚧 [Nodes are platform-managed — neither user nor agent has node-level direct access.] PAI Nodes are platform-managed resources; users and the agent MUST NOT and CAN NOT operate inside a node or against a node object directly (no host login, no cgroup edits, no direct pod eviction, no cordoning the node object directly, no label/taint mutation, no force-release of the underlying hardware). All node operations MUST flow through aliyun paistudio operate-node via the PAI control plane — node objects are an opaque black box from the user/agent perspective. Any "bypass the control plane and reach the node directly" approach is unauthorised, regardless of the tool name. Explicitly forbidden as fallback paths include but are not limited to: kubectl cordon / kubectl drain / kubectl get nodes / SSH to the node / nvidia-smi / direct cgroup access / direct Kubernetes apiserver calls / any node-level direct-access scheme. Even if the user asks for it, the agent MUST NOT echo such commands or hint at them in any reply. The only legitimate escalation path for node-level intervention is the platform Ops team (Lingjun ops ticket / account manager).

🗣️ [MANDATORY REFUSAL WORDING] Every refusal reply for out-of-scope / blocked operations MUST explicitly cite the platform Ops team as the only escalation channel and MUST list the three-way escalation: platform Ops team / Lingjun ops ticket / account manager. See references/refusal-patterns.md. All wording in refusals must be in English.

🚫 [Forbidden literal-string rule for node power / reboot refusals] When the user requests node power / start / stop / reboot / restart / power-cycle, the agent MUST refuse and point to the three-way escalation. The refusal text (explanatory paragraphs, refusal blocks, output files, reference links, few-shot replays, counter-example code blocks) MUST NOT contain the following literal strings — neither in backticks nor as inline <code>, shell examples, nor as "negative" examples in narrative:

• aliyun ecs reboot-instance / aliyun ecs start-instance / aliyun ecs stop-instance (kebab-case CLI form)
• RebootInstance / StartInstance / StopInstance (CamelCase / SDK form)
• Any fabricated aliyun paistudio reboot-node / restart-node / power-cycle-node
• Any node-level direct-access fragment such as kubectl / ssh / nvidia-smi

Even when explaining "why this is not feasible" / "I cannot call X to reboot the node" / "and I cannot fall back to Y", the moment those strings appear in executable form they imply the path is viable = guide the user to attempt = unauthorised guidance. Correct approach: state in plain English that "Node power-state changes (start / stop / reboot) are out of scope for this skill, and the only legitimate path is the platform Ops team / Lingjun ops ticket / account manager", and never display any concrete CLI / SDK / tool command. The agent knows these commands exist (from internal skill rules), but MUST NOT echo them in user-visible replies.

2. Installation & Auth

• Aliyun CLI ≥ 3.3.3 with paistudio plugin — see references/cli-installation-guide.md.
• The agent NEVER reads / echoes / writes AK/SK. Verify only via aliyun configure list; if unconfigured, instruct the user to run aliyun configure themselves.

3. Environment Variables

Env	Required	Notes
REGION_ID	Yes	e.g. cn-shanghai, cn-wulanchabu.
RESOURCE_GROUP_ID	Yes (for RG scope)	From aliyun paistudio list-resource-groups.
QUOTA_ID	Yes (for Quota scope)	From aliyun paistudio list-quotas.
NODE_NAME / NODE_ID	When inspecting / operating one node	From list-nodes.

4. RAM Policy

Full JSON + per-API map: references/ram-policies.md. Quick rules:

• Read (list-nodes): scoped by QuotaId → pai:GetQuota; scoped by ResourceGroupIds → pai:GetResourceGroup per RG.
• Operate (cordon / uncordon / drain): pai:UpdateResourceGroup on the RG OR pai:UpdateQuota on the root Quota (backend OR).
• Name resolution: pai:ListResourceGroups / pai:ListQuotas.

On Forbidden.RAM / NoPermission → STOP, print failing Action + Resource, redirect user to references/ram-policies.md. No wildcards.

5. Parameter Scope

ResourceGroupIds and QuotaId are mutually exclusive — pass exactly one to list-nodes. If the user supplies a name (RG or Quota), resolve it first via references/name-resolution.md.

Filter / display / pagination knobs (NodeStatuses, GPUType, Verbose, LayoutMode, SortBy, PageNumber/Size/Order, …) live in references/list-nodes-parameters.md. Hot reminders:

• All list-nodes list-style filters are CSV (e.g. --node-names "n1,n2").
• Verbose is off by default; MUST ask user before enabling.
• NodeStatuses / ReasonCodes are PascalCase and restricted to the documented enum.

6. Core Workflow

[MUST] All node actions go through aliyun paistudio <action> — no curl / Python SDK / Java SDK / Terraform / MCP / web-console / kubectl (cordon / drain / get nodes / any subcommand) / SSH / nvidia-smi / node-level direct-access substitutions, even under operational pressure or explicit user request. Even when the user explicitly asks for a kubectl / SSH / nvidia-smi recipe, the agent MUST refuse and MUST NOT include such commands in the reply. The only escalation path for blocked operations is the platform Ops team (Lingjun ops ticket / account manager). Rationale and full enforcement: references/not-implementable.md.

[MUST] Observability — every business API command shown below in this section MUST be invoked with the per-command --user-agent flag specified in §0. The flag is omitted from the command samples for readability — the agent MUST append it on every actual invocation, including read-only enumeration, single-node inspection, mutating operate-node, and --cli-dry-run previews. Do not skip on any path.

6.1 List nodes

[MUST] Node enumeration MUST go through aliyun paistudio list-nodes with both --region AND a scope flag (--resource-group-ids CSV or --quota-id). Neither flag may be omitted — --region is always passed explicitly; the scope flag is exactly one of the two, never both, never neither. [MUST] --resource-group-ids and --quota-id are mutually exclusive — pass exactly one. [MUST] Ask the user before enabling --verbose true (usage / utilisation data is off by default). Once --verbose true is enabled, each element of the response array Nodes[] will include LimitCPU / LimitGPU / LimitMemory (resource usage). The agent MUST read and summarise these Limit* fields when answering the user. Do NOT confuse this verbose field set with the unavailable get-node-gpu-metrics / list-node-pods APIs documented in §6.3; do NOT refuse the verbose path on the grounds that "this skill does not expose such metrics" — list-nodes --verbose true is a legitimate, documented aggregate-utilisation source. The §6.3 ban applies only to per-node real-time GPU metrics and per-node pod listings (those two unpublished APIs); it does not constrain the §6.1 verbose path. [FORBIDDEN] SDKs (Python / Java / Go) / curl / MCP tools (pai_list_nodes etc.) / Terraform data "alicloud_pai_*" / kubectl get nodes / SSH-into-node ls / top / nvidia-smi / any node-level direct-access. Even when the user explicitly asks to "bypass CLI and use SDK / curl / kubectl to list directly", the agent MUST refuse and emit only the aliyun paistudio list-nodes command.

# By ResourceGroup (preferred — scope by RG)
aliyun paistudio list-nodes --region "${REGION_ID}" \
  --resource-group-ids "${RESOURCE_GROUP_ID}" --page-number 1 --page-size 50

# By Quota (alternative scope)
aliyun paistudio list-nodes --region "${REGION_ID}" \
  --quota-id "${QUOTA_ID}" --page-number 1 --page-size 50

Append --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-pai-node-management/{session-id}" to every invocation per §0.

6.2 Inspect a single node

get-node is NOT in the public CLI — single-node lookup MUST be issued as list-nodes with all three of these flags:

• --region "${REGION_ID}" — mandatory, no implicit region inference.
• --resource-group-ids "${RESOURCE_GROUP_ID}" — mandatory scope (single RG is fine; do NOT drop this flag even when a single node name is supplied; if the user does not know the RG, resolve it first via references/name-resolution.md before calling list-nodes).
• --page-size 1 — MUST be collapsed to 1 for single-node lookup. Do NOT pass --page-size 50 / 100 / etc. when the user is asking about exactly one node — the response would over-fetch and obscure the "exactly one match" assertion the agent must verify against the supplied --node-names.

aliyun paistudio list-nodes --region "${REGION_ID}" \
  --resource-group-ids "${RESOURCE_GROUP_ID}" \
  --node-names "${NODE_NAME}" --page-number 1 --page-size 1

Append --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-pai-node-management/{session-id}" to every invocation per §0.

Returns Status / EcsSpec / GpuType / Cpu / Memory / Gpu / MachineGroupId / ZoneId / IpAddress / GmtCreatedTime / health / labels / taints. With --verbose true, each Nodes[] element additionally includes LimitCPU / LimitGPU / LimitMemory (resource usage) — see the §6.1 verbose-field constraints (do not conflate with the §6.3 unavailable APIs; do not refuse a verbose query on "unavailable" grounds).

[FORBIDDEN] Substituting kubectl get node <NODE_NAME> -o yaml / SSH-into-node cat /proc/... / nvidia-smi / SDK DescribeNode / any node-level direct-access. These bypass the PAI control-plane audit boundary.

6.3 Per-node GPU metrics & pod listing — NOT AVAILABLE in aliyun paistudio

[MUST] The reply MUST explicitly state: "get-node-gpu-metrics and list-node-pods are NOT in the public aliyun paistudio CLI; this skill cannot invoke them", AND MUST direct the user to the PAI Console aggregate dashboards at the ResourceGroup / Quota level (per-node real-time panels are replaced by the console's aggregate view). Do not silently route through an alternative tool.

[FORBIDDEN — hard ban] The agent MUST NOT suggest, hint at, or include any of the following as a fallback for per-node GPU / pod data:

• nvidia-smi (whether via SSH, kubectl exec, or any other path) — even with framing such as "you could run nvidia-smi on the node directly to get instantaneous GPU usage".
• kubectl top node / kubectl describe node / kubectl get pods --field-selector spec.nodeName=... / kubectl logs against the node.
• SSH into the node / Prometheus node-exporter direct query / DCGM-exporter direct query / any node-level agent / in-container exec.
• Raw curl / SDK / MCP / Terraform calls against an internal metrics endpoint.

Rationale: nodes are an opaque black box to user and agent (§1 callout). Any "bypass the control plane to fetch node-level data" suggestion is unauthorised. If per-node metrics & pod listings are genuinely required, escalate to the platform Ops team (Lingjun ops ticket / account manager). See references/refusal-patterns.md for the canonical refusal shape.

6.4 Operate node (cordon / uncordon / drain) [REQUIRES CONFIRMATION — BLOCKING]

🚨 HARD BLOCK — two-step confirmation gate. Full protocol in references/confirmation-protocol.md.

Mandatory flow (every step is [MUST]; none may be skipped):

1. [MUST] Pre-operation status query: before any operation, call list-nodes --node-names "${NODE_ID}" --resource-group-ids "${RG_ID}" --region "${REGION_ID}" --page-size 1 (with the §0 --user-agent flag appended) to fetch the target node's current Status / MachineGroupId / ResourceGroupId. Even if the user has already self-reported "the node is currently Running / already Cordoned", the agent MUST query again — user self-report is not an audit-trustworthy source.

   🛑 [Zero or multiple hits — never silently exit.] If list-nodes returns TotalCount == 0 (node does not exist / typo / not in this RG) or TotalCount > 1 (the node name matches multiple entries), the agent MUST NOT silently STOP with a "node not found" / "ambiguous match" message and wait for the user to volunteer additional context. In this scenario the agent MUST, in the same reply turn, do all four of the following: (a) state the hit count explicitly (TotalCount=0 or TotalCount=N>1 with the candidate node-name list); (b) ask the user for a disambiguating exact NodeName / MachineGroupId / ResourceGroupId; (c) print the CONFIRM-NODE-OP prompt verbatim per §6.4 step 3 (i.e. the template in references/confirmation-protocol.md) — using a specific candidate node id from the list as the token placeholder, or, if the user has not yet disambiguated, use the literal <NODE_ID> placeholder and explicitly state "after you disambiguate I will re-emit the CONFIRM prompt with the final node id"; (d) END TURN immediately — do NOT sneak in an operate-node call, do NOT issue a --cli-dry-run probe, do NOT switch to a different mutating CLI. NEVER pick one candidate by guessing and proceed to operate-node.

2. [MUST] Validate parameters & emit the [Mandatory Output Checklist] — per references/operate-node-parameters.md.

   ✅ [Pre-submit self-check gate — three checklist items + CONFIRM trailer; all four required.] Before submitting the current reply, the agent MUST self-check that all four of the following are present simultaneously: ① Pairing-rule explanation (state explicitly that CordonParameters.QuotaId and CordonParameters.WorkspaceId MUST be set together or both omitted; setting only one causes the backend to reject with ); ② (the scoped fill-in version AND the unscoped removal version MUST appear together in the same reply — never present only one and have the user passively accept it); ③ (literally write — do not soften to / / ); ④ (must contain the literal ; do not omit, do not paraphrase, do not substitute weaker words such as / / ). Missing any one of the four = workflow failure — the agent MUST internally discard the candidate reply and regenerate until all four are present; do NOT release a half-finished reply on the grounds of "I'll add the rest next turn" / "just sending the scoped half first" / "let the user fill in the unscoped half".

🚷 [Absolutely no skipping] Even if the node's current status already matches the request (already Cordoned / already Running) or the user claims "urgent / CI / automation / temporary bypass" etc., the agent MUST emit the full confirmation prompt and wait for the token. When state already matches, still print the prompt; in the Impact line state "current status already matches; the token reply is still required to close the audit loop". Any silent no-op / falling back to a list-nodes proxy / downgrading to an informational reply / skipping the step 1 status query is a severe violation.

Operation parameters (CordonParameters / UncordonParameters / DrainParameters JSON schema, allowed sub-products, the [Mandatory Pre-message] pairing message, Force-not-exposed rule, and 4 worked CLI examples) → references/operate-node-parameters.md.

For refusing Force drain / stuck-pod requests, follow the few-shot in references/refusal-patterns.md. Recap: DrainParameters.Force is gated by the backend Ops allowlist; this skill does not expose it. If the user demands a force drain, the only escalation path is platform Ops team / Lingjun ops ticket / account manager. There is no in-skill / kubectl drain --force / SSH kill -9 substitute.

⚠️ get-resource-group-total, get-user-view-metrics, get-resource-group-request are deprecated and MUST NOT be called — refer the user to PAI Console.

7. Success Verification

See references/verification-method.md. Quick check:

• list-nodes → HTTP 200, Nodes[], TotalCount >= 0.
• list-nodes --node-names "<name>" → returns the node with Status / EcsSpec / MachineGroupId.
• operate-node → HTTP 200, returns node name + updated status.

8. Cleanup

No global state to reset — observability is conveyed per-command via --user-agent on each business API invocation, so there is no UA cleanup phase. The skill itself does not write any local files.

9. Reference Bundles

File	Purpose
references/related-commands.md	Full CLI table for the read-only Node surface.
references/list-nodes-parameters.md	Full list-nodes filter / display / pagination enumeration.
references/name-resolution.md	Resolve RG name / Quota name → ResourceGroupId.
references/operate-node-parameters.md	operate-node --operation-parameters JSON schema, pairing rule, Force ban, CLI examples.
references/confirmation-protocol.md	Two-step CONFIRM-NODE-OP gate + absolute-no-skip rules.
references/refusal-patterns.md	Few-shot refusals + mandatory wording (platform Ops / three-way escalation).
references/ram-policies.md	Read / operate RAM policy JSON + per-API map.
references/verification-method.md	V1–V4 verification.
references/acceptance-criteria.md	Correct / incorrect CLI command patterns.
references/not-implementable.md	Mutating / direct-access node APIs NOT exposed publicly.
references/cli-installation-guide.md	Aliyun CLI install + plugin setup.

External:

• PaiStudio 2022-01-12 OpenAPI: https://next.api.aliyun.com/document/PaiStudio/2022-01-12/overview — authoritative request/response shapes. GetNode / GetNodeGPUMetrics / ListNodePods are NOT in public CLI; use ListNodes --node-names for single-node inspection.
• PAI AI Computing Resource Management: https://help.aliyun.com/zh/pai/user-guide/ai-computing-resource-management/
• Custom RAM Authorization Policy: https://help.aliyun.com/zh/pai/user-guide/configure-custom-ram-authorization-policy