| name | amp-diagnostics |
| version | 1.0.0 |
| last_updated | 2025-04-12 |
| description | Use this skill to investigate and troubleshoot Amazon Managed Service for Prometheus (AMP) problems by analyzing workspace configurations, remote write failures, query performance, alerting/ruler issues, ingestion problems, Prometheus-compatible API issues, IAM/SigV4 authentication, and collector configuration. Activate when: workspace creation failures, remote write errors, SigV4 authentication failures, ingestion rate limiting, high cardinality issues, query timeouts, PromQL errors, rule evaluation failures, SNS alert integration issues, ADOT collector problems, Prometheus server remote write configuration, EKS add-on issues, Grafana data source setup, VPC endpoint access, or the user says something is wrong with AMP without naming specific symptoms.
|
| compatibility | Requires AWS CLI with amp and cloudwatch permissions, promtool (optional), Prometheus or Grafana client for query validation, CloudTrail permissions. Optional: ADOT Collector CLI, kubectl (for EKS-based collectors).
|
AMP Diagnostics
When to use
Any AMP investigation where the console alone is insufficient — workspace lifecycle issues, remote write ingestion failures, SigV4 authentication errors, query performance problems, alerting/ruler configuration, collector troubleshooting, or integration with Grafana and third-party tools.
Investigation workflow
Step 1 — Collect and triage
# List all workspaces and their status
aws amp list-workspaces --query 'workspaces[*].{id:workspaceId,alias:alias,status:status.statusCode}'
# Describe specific workspace
aws amp describe-workspace --workspace-id <workspace-id>
# Check CloudWatch metrics for AMP workspace
# Key metrics: ActiveSeries, IngestionRate, IngestionErrors, QueryErrors
aws cloudwatch get-metric-statistics --namespace AWS/Prometheus \
--metric-name ActiveSeries --dimensions Name=WorkspaceId,Value=<workspace-id> \
--start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 300 --statistics Maximum
# Check ingestion errors
aws cloudwatch get-metric-statistics --namespace AWS/Prometheus \
--metric-name IngestionErrors --dimensions Name=WorkspaceId,Value=<workspace-id> \
--start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 300 --statistics Sum
Triage returns:
- Workspace state, alias, creation timestamp
- Active series count and ingestion rate
- Ingestion and query error counts
- Workspace endpoint URLs (remote write and query)
If the workspace status is not ACTIVE, that IS the root cause domain. Don't chase ingestion or query symptoms.
Step 2 — Domain deep dive (only if needed)
# Check remote write endpoint connectivity
curl -s -o /dev/null -w "%{http_code}" --aws-sigv4 "aws:amz:us-east-1:aps" \
--user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
https://aps-workspaces.<region>.amazonaws.com/workspaces/<workspace-id>/api/v1/remote_write
# Check query endpoint
curl -s --aws-sigv4 "aws:amz:<region>:aps" \
--user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
"https://aps-workspaces.<region>.amazonaws.com/workspaces/<workspace-id>/api/v1/query?query=up"
# List alert manager configuration
aws amp describe-alert-manager-definition --workspace-id <workspace-id>
# List ruler configuration
aws amp describe-rule-groups-namespace --workspace-id <workspace-id> --name <namespace>
aws amp list-rule-groups-namespaces --workspace-id <workspace-id>
# CloudTrail for AMP API events
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=aps.amazonaws.com --max-results 20
Read references/amp-guardrails.md before concluding on any AMP issue.
Step 3 — Detailed investigation (low-confidence cases only)
# CloudTrail for detailed API call history
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=aps.amazonaws.com \
--start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%S) --max-results 50
# VPC endpoint check (if using private connectivity)
aws ec2 describe-vpc-endpoints --filters Name=service-name,Values=com.amazonaws.<region>.aps-workspaces
# IAM role/policy verification for SigV4
aws iam simulate-principal-policy --policy-source-arn <role-arn> \
--action-names aps:RemoteWrite aps:QueryMetrics aps:GetSeries aps:GetLabels
# Check ADOT collector logs (if running on EKS)
kubectl logs -n opentelemetry -l app=adot-collector --tail=100
# Check Prometheus server remote write status (if using Prometheus)
kubectl logs -n monitoring -l app=prometheus --tail=100 | grep -i "remote"
Tool quick reference
| Tool / Command | When to use |
|---|
aws amp list-workspaces | List all workspaces, check status and aliases |
aws amp describe-workspace | Workspace details, endpoints, status, tags |
aws amp describe-alert-manager-definition | Alert manager YAML configuration |
aws amp list-rule-groups-namespaces | List all ruler namespaces |
aws amp describe-rule-groups-namespace | Ruler rules YAML for a namespace |
aws cloudwatch get-metric-statistics | AMP metrics: ActiveSeries, IngestionRate, IngestionErrors |
aws cloudtrail lookup-events | AMP API call history and errors |
aws iam simulate-principal-policy | Verify SigV4 IAM permissions |
aws ec2 describe-vpc-endpoints | VPC endpoint status for private connectivity |
curl --aws-sigv4 | Test remote write and query endpoints directly |
promtool check rules | Validate Prometheus rule files locally |
kubectl logs | ADOT collector or Prometheus server logs on EKS |
Gotchas: AMP
These are the mistakes commonly made during AMP troubleshooting.
- AMP is a managed Prometheus-compatible service — it is NOT self-hosted Prometheus. There is no Prometheus server to SSH into, no local TSDB, no Prometheus UI. You interact via remote write (ingestion) and query (PromQL) API endpoints only.
- Remote write to AMP requires SigV4 authentication — basic auth, bearer tokens, and other Prometheus-native auth methods do NOT work. Every request must be signed with AWS credentials using the SigV4 protocol.
- Each workspace has ingestion rate limits and active series limits. Exceeding these causes HTTP 429 (throttling) responses on the remote write endpoint. Check CloudWatch
IngestionRate and ActiveSeries metrics.
- The query endpoint is separate from the remote write endpoint. Remote write URL ends in
/api/v1/remote_write. Query URL uses /api/v1/query and /api/v1/query_range. Mixing them up causes 404 or method-not-allowed errors.
- AMP does NOT run a Prometheus server — there is no local storage, no Prometheus UI, no targets page, no service discovery configuration. AMP is a remote storage backend only. Use Grafana or the query API for visualization.
- Alerting in AMP uses the ruler service with SNS integration for notifications. It does NOT use a standalone Alertmanager with email/Slack/PagerDuty receivers directly — you route through SNS topics which then fan out to your notification channels.
- Metrics retention in AMP is 150 days and is NOT configurable. You cannot extend or reduce it. Plan data lifecycle accordingly.
- HA collector setups (multiple Prometheus servers or ADOT collectors scraping the same targets) send duplicate samples. AMP deduplicates based on labels, but you must ensure consistent external labels across replicas to enable proper deduplication.
- ADOT Collector and Prometheus server are both valid remote write clients, but they have different configuration formats. ADOT uses
exporters.prometheusremotewrite in its YAML config. Prometheus uses remote_write in prometheus.yml. Don't mix the config syntax.
- VPC endpoints are available for private connectivity to AMP (
com.amazonaws.<region>.aps-workspaces). Without a VPC endpoint, traffic goes over the public internet even from within AWS. This matters for security-sensitive workloads.
- Workspace deletion is permanent — there is no recovery, no recycle bin, no snapshot. All metrics data is lost immediately. Always confirm the workspace ID before deleting.
- Label and series cardinality is the #1 performance issue in AMP. High-cardinality labels (like request IDs, pod UIDs, or timestamps in label values) explode the active series count, leading to throttling, slow queries, and increased costs.
Anti-hallucination rules
- Always cite specific AWS CLI output, CloudWatch metrics, CloudTrail events, or collector logs as evidence.
- Never suggest accessing Prometheus server internals (TSDB, WAL, config reload endpoint) — AMP is fully managed with no server-side access.
- Never recommend basic auth, bearer tokens, or API keys for AMP authentication — only SigV4 signing is supported.
- Never claim metrics retention is configurable — it is fixed at 150 days.
- Never suggest using Prometheus UI, targets page, or service discovery configuration — AMP has no UI; use Grafana or the query API.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
26 runbooks
Runbooks are organized by failure domain. Use the appropriate runbook based on the symptom category.
| Category | IDs | Covers |
|---|
| A — Workspace | A1–A3 | Workspace creation failures, configuration issues, deletion/recovery |
| B — Ingestion | B1–B4 | Remote write failures, SigV4 auth errors, rate limiting, high cardinality |
| C — Query | C1–C3 | Query timeouts, PromQL errors, query performance |
| D — Alerting | D1–D3 | Rule evaluation failures, SNS integration, alert manager config |
| E — Collectors | E1–E3 | ADOT collector issues, Prometheus server remote write, EKS add-on |
| F — Security | F1–F2 | IAM/SigV4 configuration, VPC endpoint access |
| G — Integration | G1–G2 | Grafana data source, third-party tools |
| Z — Catch-All | Z1 | General AMP troubleshooting |
