ecs-observability

name

ecs-observability

description

Advise on Amazon ECS observability architecture — select the logs/metrics/traces stack (CloudWatch, Container Insights, X-Ray, ADOT/OpenTelemetry, Managed Prometheus/Grafana, FireLens to third-party) by compliance needs, existing tooling, scale, budget, and launch types (EC2, Fargate, Managed Instances, ECS Anywhere). Use for "how should we monitor our ECS services", "Container Insights or Prometheus for ECS", "are we losing ECS container logs", "set up tracing on Fargate", "ECS logging best practices", "Datadog vs CloudWatch for ECS", "GPU metrics for ECS tasks", or "plan live-debug access to an ECS task". Any ECS logging, metrics, tracing, or alerting design question qualifies even if "observability" is never said. Skip for EKS/Kubernetes (eks-* skills), deployment mechanics/CI-CD/deploy-failure diagnosis (ecs-devops; deploy-failure alerting stays here), security posture beyond observability audit logging (ecs-security), live-estate audits (ecs-operation-review), and FinOps audits of observability spend.

ECS Observability

Advisory skill for designing the observability architecture of Amazon ECS workloads — which logging pipeline, which metrics stack, which tracing path, driven by the customer's compliance posture, existing tooling, scale, budget, and the launch types they actually run. This is primarily a design-and-selection brain: it produces recommendations and decision rationale, with AWS documentation URLs for every load-bearing fact — and it carries the diagnostic facts (the log-loss gate, ECS Exec constraints) that troubleshooting asks need, without being a step-by-step runbook.

The accuracy bar (non-negotiable for this skill). Launch-type scope-bleed is the #1 error class in this domain — capabilities documented for one launch type get silently generalized to all four. Every capability claim in this skill is scoped to EC2 / Fargate / Managed Instances (MI) / ECS Anywhere (EXTERNAL), and where AWS docs are silent the claim is marked "undocumented — verify". Never state launch-type support you cannot cite to an AWS-published source. When you can't ground a claim, say so — do not synthesize.

When to Use This Skill

Activate when the user wants to:

• Choose between CloudWatch-native (Container Insights, X-Ray, Application Signals), open-source-flavored (ADOT, Amazon Managed Prometheus, Amazon Managed Grafana), or third-party (Datadog, Splunk, SIEM via FireLens) observability for ECS
• Design log delivery — awslogs vs FireLens, blocking vs non-blocking mode, buffer sizing, multi-destination routing
• Decide standard vs enhanced Container Insights, or how to get GPU telemetry from ECS tasks
• Pick a tracing path for new or existing ECS services (ADOT/OTel vs legacy X-Ray SDK/daemon)
• Understand what observability each launch type (EC2, Fargate, Managed Instances, ECS Anywhere) can and cannot support
• Plan live-debugging access (ECS Exec) including private-subnet VPC endpoint requirements
• Weigh compliance requirements (no log loss, session auditing, KMS encryption) against availability and cost

Do NOT use this skill for:

• EKS or any Kubernetes observability → use eks-best-practices (design guidance) or eks-operation-review (live-cluster audit), or the other eks-* skills
• ECS deployment mechanics — circuit breaker configuration, rollback strategy, CI/CD pipelines → use ecs-devops. Deployment-failure alerting and visibility (e.g., EventBridge on SERVICE_DEPLOYMENT_FAILED) stays here.
• ECS security posture, IAM hardening, compliance audits beyond observability's audit-logging angle → use ecs-security
• Scoring or auditing a live ECS estate against a rubric → use ecs-operation-review
• Overall ECS architecture — compute selection, networking, service design → use ecs-architect
• GenAI/LLM workload design on ECS (GPU serving stacks, model hosting) → use ecs-genai; come back here for the GPU telemetry question
• Cost optimization of an existing observability bill as a FinOps exercise — this skill flags cost levers during design but does not audit spend

The Decision Framework

Elicit these five criteria before recommending a stack — each one prunes the option space:

Criterion	Ask	Why it decides
Launch types in use	EC2, Fargate, Managed Instances, EXTERNAL — which, today and planned?	Hard eliminator. ADOT/AMP is unsupported on EXTERNAL; agentless GPU metrics are MI-only; Fargate forbids host agents; App Signals daemon strategy excludes Fargate. Check references/launch-type-matrix.md before every claim.
Compliance needs	Can any log record be lost? Are debug sessions audited? Customer-managed KMS?	Drives the blocking vs non-blocking log gate, FireLens filesystem buffering, ECS Exec session logging, and KMS endpoint/key-policy work.
Existing tooling	Grafana/PromQL estate? Datadog/Splunk/SIEM contract? OTel standardization?	Existing Grafana → ADOT+AMP+AMG. Third-party APM/SIEM → FireLens routing. No estate → CloudWatch-native is the lowest-friction default.
Scale	Log MB/s per task, task count, metric cardinality	High log throughput breaks default buffers (silent loss); container-level enhanced CI multiplies metric series; sidecar-per-task overhead compounds with task count.
Budget	Appetite for custom-metric, log-ingestion, and AMP/AMG spend	Free vended metrics + EventBridge events are the zero-cost floor; enhanced CI and high-cardinality Prometheus are the main cost levers. Link https://aws.amazon.com/cloudwatch/pricing/ — never quote prices.

Default recommendation shape

For a CloudWatch-centric customer with no contrary criteria (EC2/Fargate fleets — EXTERNAL needs a different shape, since ADOT and Container Insights are not documented there): awslogs (with an explicit, documented delivery-mode decision) + Container Insights enhanced + ADOT sidecar for traces + EventBridge alerting on service action ERROR events. Enhanced is AWS's recommended Container Insights tier (per https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-ECS-cluster.html) — but it adds container-level metric cardinality, a cost lever to surface (Advisory Rule 3; cost posture in references/metrics-stacks.md). For MI fleets the logs + metrics + EventBridge parts carry over, but the ADOT-sidecar tracing leg is not documented for Managed Instances as of 2026-07-10 (the ADOT integration page enumerates Fargate + EC2 only, per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/application-metrics-prometheus.html) — verify against the live doc before recommending it there (see the matrix's ADOT row). Deviate per the criteria above.

Logs — the decision gate every engagement must surface

Facts verified 2026-07-09 against https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-account-settings.html and https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_LogConfiguration.html

Since June 25, 2025, the ECS default log delivery mode is non-blocking — and non-blocking mode silently drops logs when its in-memory buffer (default max-buffer-size: 10 MiB) fills. No metric, no daemon log line, no signal (per https://aws.amazon.com/blogs/containers/preventing-log-loss-with-non-blocking-mode-in-the-awslogs-container-log-driver/). The alternative, blocking, preserves every record but can hang the application and fail health checks if the log pipeline backs up. This is a compliance-versus-availability decision, and since the 2025 default change the risky posture is what customers get by not deciding:

• Availability-first (user-facing services, tolerable loss): keep non-blocking, size max-buffer-size to ~25 MB for in-Region delivery at up to ~5 MB/s per container (AWS benchmark guidance — "no observed loss" in AWS's test runs, not a guarantee); above 6 MB/s AWS calls the driver's behavior "less predictable and consistent" — that becomes a FireLens conversation (see reference). Document the accepted loss risk.
• Completeness-first (audit/financial/security logs): set mode: blocking per container or set the defaultLogDriverMode account setting — a per-Region setting, repeat it in every Region running tasks — and accept that a CloudWatch disruption can stall the app.
• Both required: FireLens (Fluent Bit) with filesystem buffering — loss-resistant without blocking stdout/stderr.
• Always audit the effective account default first: aws ecs list-account-settings --name defaultLogDriverMode --effective-settings.

FireLens is also the usual routing layer when the destination is not CloudWatch (Splunk, Datadog, OpenSearch, S3, Firehose, SIEMs) — though the native splunk log driver is documented on both Fargate and EC2 as a FireLens-free path to Splunk. FireLens is documented for Linux tasks on Fargate and EC2 only; MI and EXTERNAL support is undocumented — verify before advising.

For buffer benchmarks, FireLens rules, and high-throughput patterns, see: Log Delivery Reference

Metrics — Container Insights tiers and the GPU trap

Facts verified 2026-07-09 against https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html and https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-enhanced-observability-metrics-ECS.html

• Standard Container Insights (containerInsights=enabled): task/service/cluster-level metrics. Enhanced (enhanced, released Dec 2024): adds container-level metrics, per-TaskId dimensions, and health-status metrics — AWS's recommended default, at the cost of more metric series (cardinality is the main cost lever — link https://aws.amazon.com/cloudwatch/pricing/, never quote prices). Documented launch types: EC2 (agent ≥ 1.29), Fargate, Managed Instances; EXTERNAL is not listed — don't claim it.
• GPU telemetry is the sharpest launch-type trap: agentless GPU/DCGM metrics (utilization, memory, power, temperature at container/task/instance level) are Managed Instances-ONLY and require enhanced CI (per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/monitoring-managed-instances.html). The EC2 launch type gets only the free GPU reservation metric — GPU utilization there means customer-managed DCGM/CloudWatch-agent tooling. Never generalize in either direction.
• Prometheus/Grafana route: ADOT sidecar remote-writes to Amazon Managed Service for Prometheus, visualized in Amazon Managed Grafana — documented for Fargate and EC2 only; EXTERNAL explicitly unsupported, MI not documented as of 2026-07-10 (see the ADOT row of references/launch-type-matrix.md for the source quote) — verify before advising MI. Alternative keeping data in CloudWatch: the CloudWatch agent's Prometheus scrape support.
• Zero-cost floor available everywhere: free vended CPU/memory metrics + EventBridge service/task/deployment events + container health checks.

For metric tables, instance-level collection, and AMP vs CloudWatch-agent-Prometheus selection, see: Metrics Stacks Reference

Traces — OTel is the path; X-Ray SDK/daemon is legacy

Facts verified 2026-07-10 against https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-daemon-timeline.html

• The X-Ray SDKs and daemon entered maintenance mode on February 25, 2026 (security fixes only); AWS recommends OpenTelemetry. Steer all new tracing to the ADOT sidecar collector (public.ecr.aws/aws-observability/aws-otel-collector), which still delivers traces to the X-Ray backend — the maintenance-mode timeline names only the SDKs and daemon, not the backend service (inference from https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-daemon-timeline.html, which covers SDKs/daemon only; the migration guide keeps X-Ray as the trace destination).
• CloudWatch Application Signals on ECS is custom setup with no autodiscovery — service names are wired via env vars. Sidecar strategy: EC2 + Fargate, requires awsvpc. Daemon strategy: one agent per cluster, excludes Fargate. MI: not documented for either strategy as of 2026-07-10; EXTERNAL: sidecar structurally excluded (awsvpc required), daemon undocumented (see the App Signals rows of the matrix).
• Placement rule: Fargate → sidecar only (no host access); EC2 → sidecar or daemon-scheduled collector; MI → neither pattern is documented as of 2026-07-10 (the ADOT page enumerates Fargate + EC2 only — per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/application-metrics-prometheus.html; verify against the live doc before advising, see the matrix's ADOT row); EXTERNAL → ADOT unsupported.

For migration gotchas, IAM, and Application Signals strategy details, see: Tracing and Signals Reference

Live debugging — ECS Exec

Facts verified 2026-07-09 against https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html

• Works on all four launch types (one of the few capabilities that does), and is the sole shell path on Managed Instances (no SSH there).
• Private-subnet precision, for tasks in a VPC (EC2/Fargate/MI): ECS Exec needs the ssmmessages interface VPC endpoint — not the full ssm/ec2messages trio (per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) — plus a KMS endpoint only when sessions are encrypted with a customer-managed key. On EXTERNAL (ECS Anywhere) the host must reach ssm + ec2messages + ssmmessages regardless — an SSM Agent registration prerequisite (per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html).
• Compliance anchors: session logging via cluster executeCommandConfiguration, CloudTrail ExecuteCommand events, and an IAM deny on ssm:StartSession to close the unlogged side door. Commands run as root; readonlyRootFilesystem unsupported.

For full constraints and audit setup, see: Native Visibility and ECS Exec Reference

Launch-Type Discipline

Before asserting any capability, check references/launch-type-matrix.md — the per-capability EC2/Fargate/MI/EXTERNAL matrix with a source URL per row. The matrix is the single source of truth for these facts; the recurring traps (restated here per the matrix, verified 2026-07-10 — re-check the matrix row before load-bearing use):

• Agentless GPU/DCGM → MI only (EC2 gets reservation-only).
• ADOT/AMP → documented for Fargate + EC2; EXTERNAL explicitly unsupported; MI undocumented — verify before advising.
• FireLens → Linux Fargate/EC2 documented; MI/EXTERNAL undocumented — verify before advising.
• Application Signals daemon → no Fargate; sidecar → requires awsvpc (which EXTERNAL cannot run).
• EXTERNAL supports bridge/host/none networking only — anything demanding awsvpc is structurally out.
• Fargate has no host — every agent is an in-task sidecar; MI's AMI is AWS-owned — nothing can be baked into it.

How to Use the References

This skill uses progressive disclosure — essential decision guidance is in this file; detailed reference material is loaded on demand:

Reference	Load when the task is about…
log-delivery.md	blocking vs non-blocking details, buffer-size benchmarks, awslogs options, FireLens/Fluent Bit configuration, filesystem buffering, high-throughput logging, routing to third-party destinations (Datadog, Splunk, SIEMs) and awslogs-vs-FireLens selection
metrics-stacks.md	Container Insights standard vs enhanced metric tables, GPU/DCGM scoping, instance-level metrics, ADOT+AMP+AMG vs CloudWatch-agent Prometheus, metric cost levers
tracing-and-signals.md	X-Ray maintenance-mode details and migration, ADOT collector setup and IAM, Application Signals sidecar vs daemon, language support
launch-type-matrix.md	any "does X work on launch type Y" question — check before every capability claim
native-visibility-and-exec.md	EventBridge events, service events, container health checks, ECS Exec constraints and VPC endpoints, monitoring-plan baseline

Advisory Rules

1. Scope every capability claim by launch type. If the matrix row says "undocumented — verify", say exactly that to the customer and check the live AWS doc.
2. Always surface the log delivery-mode gate in any logging discussion — the customer must consciously choose a failure mode; the post-2025 default chooses silent loss for them.
3. Never quote prices or assert the enhanced-CI pricing model for ECS — link https://aws.amazon.com/cloudwatch/pricing/ instead.
4. Do not recommend X-Ray SDK/daemon for new builds — ADOT/OTel, always, since the 2026-02-25 maintenance date.
5. Recommend the zero-cost floor first (vended metrics, EventBridge ERROR-event alerting, health checks) — it survives every stack decision and every launch type.
6. Cite as you advise — every load-bearing recommendation carries its AWS documentation URL, dated where the fact is volatile.

Sources

• ECS account settings — default log driver mode · LogConfiguration API reference · Using the awslogs driver · Preventing log loss with non-blocking mode (AWS Containers blog)
• Using FireLens · High-throughput log configuration
• Container Insights for ECS · Setting up Container Insights on ECS · Enhanced observability metrics for ECS · Container Insights enhanced observability What's New (Dec 2024)
• Monitoring Managed Instances (incl. agentless DCGM) · ECS Managed Instances · Free vended CloudWatch metrics
• ADOT/AMP for ECS application metrics · ECS trace data via ADOT · ADOT ECS setup · AMP query onboarding · CloudWatch agent Prometheus for ECS
• X-Ray daemon on ECS · X-Ray SDK/daemon maintenance timeline · X-Ray to OTel migration
• Application Signals on ECS · · ·

---

This skill is provided as sample code for educational and demonstration purposes only. Recommendations should be validated against current AWS documentation before acting on them. See the project's README and LICENSE for full terms.