managing-vertex-ai

name	managing-vertex-ai
description	Use when working with Vertex Ai — google Vertex AI platform management. Covers model management, training pipelines, prediction endpoints, feature store, experiments, datasets, and custom jobs. Use when managing ML models on GCP, deploying prediction endpoints, investigating training failures, or auditing Vertex AI resources.
connection_type	gcp
preload	false

Vertex AI Management Skill

Manage and monitor Google Cloud Vertex AI ML platform resources.

MANDATORY: Discovery-First Pattern

Always list existing resources before creating or modifying anything.

Phase 1: Discovery

#!/bin/bash

REGION="${CLOUDSDK_COMPUTE_REGION:-us-central1}"
PROJECT="${CLOUDSDK_CORE_PROJECT:-$(gcloud config get-value project 2>/dev/null)}"

echo "=== Project & Region ==="
echo "Project: $PROJECT"
echo "Region: $REGION"

echo ""
echo "=== Models ==="
gcloud ai models list --region="$REGION" --format="table(name.basename(), displayName, createTime.date())" 2>/dev/null | head -15

echo ""
echo "=== Endpoints ==="
gcloud ai endpoints list --region="$REGION" --format="table(name.basename(), displayName, createTime.date())" 2>/dev/null | head -15

echo ""
echo "=== Training Pipelines ==="
gcloud ai custom-jobs list --region="$REGION" --format="table(name.basename(), displayName, state, createTime.date())" 2>/dev/null | head -10

echo ""
echo "=== Feature Stores ==="
gcloud ai featurestores list --region="$REGION" --format="table(name.basename(), state, createTime.date())" 2>/dev/null | head -10

Core Helper Functions

#!/bin/bash

REGION="${CLOUDSDK_COMPUTE_REGION:-us-central1}"
PROJECT="${CLOUDSDK_CORE_PROJECT:-$(gcloud config get-value project 2>/dev/null)}"

# Vertex AI REST API helper
vertex_api() {
    local method="${1:-GET}"
    local endpoint="$2"
    local data="${3:-}"
    local base="https://${REGION}-aiplatform.googleapis.com/v1"
    local url="${base}/projects/${PROJECT}/locations/${REGION}/${endpoint}"
    local token
    token=$(gcloud auth print-access-token 2>/dev/null)

    if [ -n "$data" ]; then
        curl -s -X "$method" -H "Authorization: Bearer $token" \
            -H "Content-Type: application/json" "$url" -d "$data"
    else
        curl -s -X "$method" -H "Authorization: Bearer $token" "$url"
    fi
}

# gcloud AI wrapper
vai() {
    gcloud ai  --region= --format=json 2>/dev/null
}

Output Rules

• TOKEN EFFICIENCY: Target ≤50 lines per output
• Use --format=json with jq or --format=table for CLI
• Never dump full resource descriptions -- extract key fields
• Use resource basename instead of full resource path in displays

Common Operations

Model Registry and Versions

#!/bin/bash
echo "=== All Models ==="
vai models list | jq -r '.[] | "\(.name | split("/")[-1])\t\(.displayName)\t\(.createTime[0:16])"' | column -t

MODEL_ID="${1:-}"
if [ -n "$MODEL_ID" ]; then
    echo ""
    echo "=== Model Details ==="
    vai models describe "$MODEL_ID" | jq '{
        id: (.name | split("/")[-1]),
        display_name: .displayName,
        container: .containerSpec.imageUri,
        artifact_uri: .artifactUri,
        created: .createTime,
        deployed_models: [.deployedModels[]? | {endpoint: (.endpoint | split("/")[-1]), id: .deployedModelId}]
    }'

    echo ""
    echo "=== Model Versions ==="
    vai models list-versions "$MODEL_ID" | jq -r '
        .[] | "\(.name | split("/")[-1])\t\(.versionId)\t\(.createTime[0:16])"
    ' | column -t | head -10
fi

Endpoint Health and Traffic

#!/bin/bash
echo "=== Endpoint Summary ==="
vai endpoints list | jq -r '.[] | "\(.name | split("/")[-1])\t\(.displayName)\t\(.deployedModels | length) models"' | column -t

ENDPOINT_ID="${1:-}"
if [ -n "$ENDPOINT_ID" ]; then
    echo ""
    echo "=== Deployed Models on $ENDPOINT_ID ==="
    vai endpoints describe "$ENDPOINT_ID" | jq '{
        id: (.name | split("/")[-1]),
        display_name: .displayName,
        deployed_models: [.deployedModels[]? | {
            model_id: .id,
            display_name: .displayName,
            machine_type: .dedicatedResources.machineSpec.machineType,
            min_replicas: .dedicatedResources.minReplicaCount,
            max_replicas: .dedicatedResources.maxReplicaCount,
            traffic_split: .trafficPercentage
        }]
    }'
fi

Training Pipeline Status

#!/bin/bash
echo "=== Training Pipelines ==="
vai training-pipelines list | jq -r '
    .[] | "\(.name | split("/")[-1])\t\(.displayName)\t\(.state)\t\(.createTime[0:16])"
' | column -t | head -15

PIPELINE_ID="${1:-}"
if [ -n "$PIPELINE_ID" ]; then
    echo ""
    echo "=== Pipeline Details ==="
    vai training-pipelines describe "$PIPELINE_ID" | jq '{
        state: .state,
        error: .error,
        training_task: .trainingTaskDefinition,
        start_time: .startTime,
        end_time: .endTime,
        model_to_upload: .modelToUpload.displayName
    }'
fi

Experiment Tracking

#!/bin/bash
echo "=== Experiments ==="
vertex_api GET "metadataStores/default/contexts?filter=schema_title=%22system.Experiment%22" \
    | jq -r '.contexts[]? | "\(.name | split("/")[-1])\t\(.createTime[0:16])"' | column -t | head -15

EXPERIMENT="${1:-}"
if [ -n "$EXPERIMENT" ]; then
    echo ""
    echo "=== Runs in $EXPERIMENT ==="
    vertex_api GET "metadataStores/default/contexts?filter=schema_title=%22system.ExperimentRun%22%20AND%20parent_contexts=%22projects/${PROJECT}/locations/${REGION}/metadataStores/default/contexts/${EXPERIMENT}%22" \
        | jq -r '.contexts[]? | "\(.name | split("/")[-1])\t\(.metadata.state // "unknown")\t\(.createTime[0:16])"' | column -t | head -20
fi

Feature Store Management

#!/bin/bash
echo "=== Feature Stores ==="
vai featurestores list | jq -r '.[] | "\(.name | split("/")[-1])\t\(.state)\t\(.onlineServingConfig.fixedNodeCount // 0) nodes"' | column -t

FS_NAME="${1:-}"
if [ -n "$FS_NAME" ]; then
    echo ""
    echo "=== Entity Types in $FS_NAME ==="
    vai featurestores entity-types list --featurestore="$FS_NAME" \
        | jq -r '.[] | "\(.name | split("/")[-1])\t\(.createTime[0:16])"' | column -t
fi

Safety Rules

• NEVER undeploy models from production endpoints without explicit confirmation -- causes immediate prediction failures
• NEVER delete models that have active deployments -- undeploy first
• Always check traffic split before modifying endpoint configurations
• Cost awareness: GPU machines (n1-standard + NVIDIA T4/V100/A100) incur high costs -- verify machine type before launching
• Quota limits: Check regional quotas before launching large training jobs

Output Format

Present results as a structured report:

Managing Vertex Ai Report
═════════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

Shortcut	Counter	Why
"I'll skip discovery and check known resources"	Always run Phase 1 discovery first	Resource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"	Follow the full discovery → analysis flow	Quick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"	Audit configuration explicitly	Defaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"	Always check relevant metrics when available	API/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"	Try the command and report the actual error	Assumed permission failures prevent useful investigation; actual errors are informative

Common Pitfalls

• Region consistency: Models, endpoints, and datasets must be in the same region -- cross-region references fail
• Endpoint traffic split: Traffic percentages must sum to 100 -- partial updates cause deployment errors
• Custom container ports: Vertex AI expects predictions on port 8080 by default -- configure AIP_HTTP_PORT if different
• Service account permissions: Custom training jobs need the Vertex AI Service Agent role -- missing permissions cause silent failures
• Pipeline caching: Vertex Pipelines cache steps by default -- use enable_caching=False for non-deterministic steps
• Feature store ingestion: Online serving has eventual consistency -- writes may not be immediately readable
• Quota exhaustion: GPU quota is per-region -- training jobs queue indefinitely when quota is exhausted
• Model artifacts: Model must be in GCS -- local paths are not supported for deployment