dd-aws-integration

name	dd-aws-integration
description	Set up the Datadog AWS integration with Terraform - creates the cross-account IAM role Datadog assumes (external ID, no stored credentials), attaches the permission policies Datadog publishes, and registers the account through datadog_integration_aws_account so AWS metrics, the resource catalog, and CSPM findings start flowing. Use when the user has AWS resources they want to monitor, wants to connect an AWS account to Datadog, asks to set up or repair the AWS integration, or needs the Datadog IAM role and external ID provisioned. Does not set up log forwarding.
metadata	{"version":"1.0.0","author":"datadog-labs","repository":"https://github.com/datadog-labs/agent-skills","tags":"datadog,aws,integration,terraform,iam,cloud","alwaysApply":"false","tools":"terraform"}

Datadog AWS Integration

You are helping a user set up the Datadog AWS integration using Terraform.

The integration creates an IAM role in the customer's AWS account that Datadog assumes via cross-account role delegation. Datadog's AWS account is granted sts:AssumeRole with an external ID for security. No long-lived credentials are stored - Datadog assumes the role on demand.

This is a hands-on setup: run the commands yourself as part of the conversation rather than handing the user a list, keep them in the loop, and pause for confirmation before terraform apply.

Phase 0: Preflight

Terraform or OpenTofu. Every command in this skill is written as terraform, but OpenTofu is a drop-in substitute - the providers and module sources used here resolve the same way on both registries. Check which binary the user actually has before Phase 1:

command -v terraform tofu

If only tofu is on the PATH, read every terraform <subcommand> below as tofu <subcommand>. If both are present, ask which one the user wants rather than guessing.

Datadog credentials. Load DD_SITE / DD_API_KEY / DD_APP_KEY from the environment (falling back to .env.local / .env) and validate both keys - they fail independently:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
: "${DD_SITE:=datadoghq.com}"
echo "DD_SITE=${DD_SITE}"
echo "DD_API_KEY=$([ -n "${DD_API_KEY:-}" ] && echo set || echo UNSET)   DD_APP_KEY=$([ -n "${DD_APP_KEY:-}" ] && echo set || echo UNSET)"
printf 'DD-API-KEY: %s\n' "$DD_API_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "validate:     HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v1/validate"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n'   \
  | curl -sS --max-time 20 -o /dev/null -w  \
      -H @- 

Result	Meaning	What to do
Both 200	Keys are good for this site	Continue to Phase 1
validate is 403	The API key is invalid, or belongs to a different region than DD_SITE	Ask which site the key belongs to, fix DD_SITE, re-check
validate 200, current_user 403	The app key is wrong or from another region - not the API key	Get one from <APP_BASE>/organization-settings/application-keys
Either key unset	Nothing to validate	On a commercial site, run the dd-account-setup skill, then come back. On ddog-gov.com or us2.ddog-gov.com, ask the user for the keys directly - that skill validates DD_SITE against a list that excludes both government sites and will reject them

App URL. The Datadog app host is not app.${DD_SITE} for every site. It is https://app.datadoghq.com (US1), https://app.datadoghq.eu (EU1), https://app.ddog-gov.com (Gov), and for every other site it is https://${DD_SITE} itself - https://us3.datadoghq.com, https://us5.datadoghq.com, https://ap1.datadoghq.com, https://ap2.datadoghq.com, https://uk1.datadoghq.com. Resolve it once and substitute it wherever <APP_BASE> appears below. Full list: https://docs.datadoghq.com/getting_started/site/

Remember the resolved DD_SITE. Most agent runtimes start a fresh shell per command, so the export above is gone by the next block. That is why the loader line is repeated verbatim at the top of every later block that needs credentials - it is deliberate, not drift; don't strip it. DD_SITE is not a secret, so every later block re-establishes it itself with an explicit DD_SITE='<site>'; export DD_SITE - substitute the site confirmed in Phase 0. It is a plain assignment rather than : "${DD_SITE:=...}" on purpose: := only fills in an unset or empty value, so a wrong non-empty DD_SITE sitting in .env would survive it and every call would go to the wrong region. The keys are guarded with :? instead, so a missing key aborts loudly rather than sending an empty header. Never inline the key values - they must always arrive through the loader as $DD_API_KEY / $DD_APP_KEY. Two consequences follow, and both are deliberate:

• Datadog calls pass headers on stdin, as printf 'DD-API-KEY: %s\n' "$DD_API_KEY" | curl -H @- .... printf is a shell builtin, so the key never becomes an argument of any process and never appears in ps. Writing -H "DD-API-KEY: $DD_API_KEY" instead would put it in curl's argv. (-H @- needs curl 7.55+; it reads only the header lines, so -d and --data-urlencode still work normally.)
• Terraform never receives the keys as values at all for AWS, Azure, and GCP: the Datadog provider reads DD_API_KEY / DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan. (OCI is the exception - its module needs them as inputs, so there they travel as TF_VAR_*.)

Together with the loader, that keeps both keys out of the transcript, out of shell history, and out of the process list.

Tools. terraform is required; the aws CLI is optional (it only looks up the account ID):

command -v terraform || echo "MISSING terraform - https://developer.hashicorp.com/terraform/install"
command -v aws >/dev/null 2>&1 && echo "aws: available" || echo "aws: not installed"

The snippets here are POSIX shell. Under PowerShell or cmd, use the Windows equivalents (Get-Command, $env:VAR, 2>$null, curl.exe) - same calls, same order.

Phase 1: Determine Scope

Ask the user for:

• Their AWS account ID (12-digit number).
• Which AWS regions they want to monitor. Default to all regions if they have no preference.

If they don't know their account ID:

If the aws CLI is available:

aws sts get-caller-identity --query "Account" --output text

Otherwise, offer to install it - it's the fastest way to look this up automatically:

I can read your AWS account ID directly if you install the AWS CLI. On macOS: brew install awscli. Otherwise: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html. After install, run aws configure (or set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars). Want to install it now, or paste your account ID yourself?

If the user prefers to find it manually: AWS Console top-right account menu, or My Account at https://console.aws.amazon.com/billing/home#/account.

Also determine the correct Datadog trusted AWS account ID. It depends on two things - the Datadog site and the AWS partition the monitored account lives in - and getting it wrong writes an sts:AssumeRole trust for the wrong account, so do not guess.

For a commercial AWS account (partition aws):

DD_SITE	Trusted Account ID
ap1.datadoghq.com	417141415827
ap2.datadoghq.com	412381753143
ddog-gov.com	392588925713
uk1.datadoghq.com	117348461845
datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, datadoghq.eu	464622532012

us2.ddog-gov.com (US2-FED) is deliberately not in that table, and neither is any GovCloud or China AWS account. Datadog publishes a different id for a GovCloud-partition account than for a commercial account on the same government site, and this skill does not carry those values. In any of those cases, stop and read the id from Datadog's AWS manual setup guide with the DATADOG SITE selector on that page set to the user's site - the page renders the id per site, and distinguishes the commercial value from the GovCloud one. Never fall back to the commercial default for a site or partition that isn't listed above.

Phase 2: Generate and Apply Terraform

Check what already exists - local Terraform first, then Datadog

Do this before generating or applying anything, and in this order. Local state first, because it decides whether an existing integration is something you can update or something you must not touch:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print
# A project is "present" if it has configuration - .terraform/ may simply not exist yet on a fresh clone
# with a remote backend, and terraform.tfstate does not exist at all when state is remote.
if [ -n "$(find . -maxdepth 1 -type f \( -name '*.tf' -o -name '*.tf.json' \) -print -quit)" ]; then
  terraform init -input=false >/dev/null || { echo "terraform init failed - resolve that before concluding anything about existing state"; exit 1; }
  out=$(terraform state list 2>&1); rc=$?
  if [ "$rc" -ne 0 ]; then
    case $out in
      *'No state file'*|*'no state'*|*'Backend initialization required'*)
        echo "project is initialized but has no state yet - treat as a clean install" ;;
      *)
        printf '%s\n' "$out"
        echo "could not read state (backend or credentials problem) - do NOT treat this as 'nothing exists'"; exit 1 ;;
    esac
  elif [ -z "$out" ]; then
    echo "state is empty - treat as a clean install"
  else
    printf '%s\n' "" | grep -F  ||  
  

   

Match the exact resource address datadog_integration_aws_account, not a loose grep -i datadog: unrelated Datadog resources, or cloud IAM left behind by a partial apply, would otherwise read as a managed integration.

Then ask Datadog what it already has:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET -H @- "https://api.${DD_SITE}/api/v2/integration/aws/accounts")
code=$(printf   |  -1); body=$(   | sed )
[  =  ] || {  ;   ;  1; }
  

Now reconcile the two answers before doing anything:

• Not in Datadog, nothing in local state - a clean install. Continue.

• In Datadog and present in local state - this is the update/repair case, not a duplicate. Continue into the Terraform below as a change to the existing resources, and let the plan show what it will alter.

• In Datadog but absent from local state (the AWS account ID from Phase 1 already appears) - stop. Applying would either create a duplicate or fight with whatever manages it. Say so plainly and offer the options: import the existing object into this project (terraform import datadog_integration_aws_account.datadog_integration "<config-id>", where the config id is not the AWS account id - get it from the List all AWS integrations endpoint, querying by AWS account id), manage it where it is already managed, or delete it in Datadog first. Only continue if the user picks one and confirms. Two things about importing, in this order. Generate the configuration first (the Terraform below, adapted to the identity that already exists - same role/app/service-account name), because terraform import binds an existing object to a configured resource address and fails without one. And importing the Datadog registration alone is not enough: the cloud-side identity (the IAM role, the app registration, the service account) is still outside state, so either import those too or reference them with data sources, or the next apply will try to create them again and collide.

  # 1. get the Datadog-side config id (NOT the AWS account id)
  for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed \'\'); [ -n  ] &&  ; ; 
  DD_SITE=;  DD_SITE   
  : ; : 
      \
    | curl -sS -H @- 
  
  terraform import datadog_integration_aws_account.datadog_integration 
  

Check for Existing Terraform

Before generating a new Terraform configuration, check if the user already has a Terraform project in the current directory or nearby:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print

If existing .tf files are found:

• Read them to understand what providers and resources are already configured.
• If a datadog provider already exists, reuse its configuration - do not create a duplicate.
• If an aws provider already exists, reuse it.
• Only add the new resources needed (IAM role, policies, datadog_integration_aws_account) to the existing project. Do not regenerate providers, variables, or terraform blocks that already exist.
• If the user has a modular layout (e.g., separate files per concern), create a new file like datadog-aws-integration.tf for the Datadog resources.

If no existing Terraform is found, generate a standalone configuration.

The full HCL template - providers, the trust policy, the dynamically-fetched permission set with its 6144-character chunking, the role, and the datadog_integration_aws_account registration - is in references/terraform.md. Read it now and emit it with the placeholders filled in.

The template leaves logs_config.lambda_forwarder empty (metrics only). Forwarding AWS logs to Datadog is a separate follow-on that deploys the Datadog Forwarder Lambda and registers its ARN here; point the user at https://docs.datadoghq.com/logs/guide/forwarder/ once the integration is live. Don't set up the forwarder inline.

Applying the Terraform

1. Replace all <PLACEHOLDER> values:

   • <DD_SITE> from Phase 0
   • <DATADOG_TRUSTED_ACCOUNT_ID> from the table in Phase 1
   • <AWS_ACCOUNT_ID> from Phase 1

2. Ensure the user has AWS credentials configured (aws configure or environment variables), and that they belong to the account being registered. If they don't, Terraform creates the IAM role in one account while registering a different one with Datadog - the apply succeeds and the integration is broken:

   want='<AWS_ACCOUNT_ID>'   # from Phase 1
   have=$(aws sts get-caller-identity --query Account --output text) || { echo "no usable AWS credentials"; exit 1; }
   [ "$have" = "$want" ] || { echo "ambient AWS credentials are for account $have, but you are registering $want - stop and fix this"; exit 1; }
   echo "AWS credentials match account $want"
   

   Without the aws CLI, the template enforces the same thing with a lifecycle.precondition on the IAM role, which fails the apply. (A check block would only warn - Terraform does not stop an apply for a failed check.)

3. Run terraform init to install providers.

4. Plan, and save the plan to a file. The Datadog provider reads DD_API_KEY / DD_APP_KEY straight from the environment, so there are no root variables and no -var= arguments - nothing secret ends up in the plan file, in state, or on a command line:

   for f in .env.local .env; do [ -f "" ] || ;  k  DD_SITE DD_API_KEY DD_APP_KEY;    && ; v=$(grep -E   |  -1 |  -d= -f2- | sed \'\'); [ -n  ] &&  ; ; 
   DD_SITE=;  DD_SITE   
   
   : ; : 
    077          
   terraform plan -out=tfplan
   

Getting the Most Out of Your Integration

Once terraform apply completes successfully, congratulate the user and let them know metrics typically arrive within 5-10 minutes. Then check for early metrics and show a widget.

Checking for Metrics

Give it a few seconds, then query the metrics API - substitute the account ID from Phase 1 for <AWS_ACCOUNT_ID>:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
sleep 10
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -G -H @- "https://api.${DD_SITE}/api/v1/query" \
  --data-urlencode  \
  --data-urlencode  \
  --data-urlencode )
code=$(   |  -1); body=$(   | sed )
[  =  ] || {  ;   ;  1; }
  

The response carries the series/pointlist JSON the widget rendering below expects:

Rendering the Widget

If the series array is non-empty, render an ASCII chart from the real data:

• Use the pointlist values to plot the line, scaling Y-axis to actual min/max.
• Use box-drawing characters (╭, ╰, ─, │, ┤) for the line.
• List the host names from each series scope at the bottom.
• Show the top 3 series by average value if multiple are returned.

If the series array is empty, show this static preview instead and let the user know metrics are still propagating:

┌─────────────────────────────────────────────────────────┐
│  aws.ec2.cpuutilization          ▂▃▅▆▇▆▅▃▂▁▂▃▅▆▇█▇▅▃  │
│  100% ┤                                          ╭──╮   │
│   75% ┤                    ╭───╮              ╭──╯  │   │
│   50% ┤              ╭────╯   ╰──╮     ╭────╯     │   │
│   25% ┤    ╭────────╯            ╰────╯           │   │
│    0% ┤────╯                                       │   │
│       └────────────────────────────────────────────┘   │
│                                                         │
│  Metrics are on their way - check back in a few minutes │
└─────────────────────────────────────────────────────────┘
Metrics Explorer: <APP_BASE>/metric/explorer?exp_metric=aws.ec2.cpuutilization

Confirm to the user that their integration is configured and data will appear shortly. All links use DD_SITE - construct them as <APP_BASE>/....

• AWS Integration tile: <APP_BASE>/integrations/amazon-web-services - access the pre-built dashboard and verify the integration is active.
• Metrics Explorer: <APP_BASE>/metric/explorer?exp_metric=aws.ec2.cpuutilization - confirm data is flowing.
• Each AWS service (EC2, RDS, Lambda, S3, ECS, EKS, etc.) has its own dashboard that activates automatically when metrics for that service are detected.

Recommended Monitors - suggest creating monitors at <APP_BASE>/monitors/create:

• EC2 CPU utilization exceeding a threshold

• RDS free storage space running low

• Lambda error rate spikes

• ELB unhealthy host count

• Cloud Security: <APP_BASE>/security/compliance - review security posture findings across AWS resources.

• Resource Catalog: <APP_BASE>/infrastructure/catalog - browse EC2 instances, RDS databases, Lambda functions, and more.

• Infrastructure Map: <APP_BASE>/infrastructure/map - visualize AWS infrastructure.

Explore more Datadog products:

• Log Management: <APP_BASE>/logs - centralized log search and alerting.
• APM & Traces: <APP_BASE>/apm/getting-started - distributed tracing for applications on Lambda, ECS, EKS, or EC2.
• Notebooks: <APP_BASE>/notebook - shareable investigations combining metrics, logs, and events.

Important Notes

• The user must have IAM permissions to create roles, policies, and policy attachments in their AWS account.
• The external ID is generated by Datadog and included automatically in the terraform - it should not be hardcoded.
• The IAM permissions are fetched dynamically from Datadog via datadog_integration_aws_iam_permissions - they may change over time as Datadog adds new integrations.
• Permissions are automatically split into multiple policies to stay under the AWS 6144-character IAM policy size limit.
• The Datadog API and app keys are never passed to Terraform as values: the datadog provider reads DD_API_KEY and DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan. Don't declare key variables, and don't write the keys into a committed .tfvars file or any other persistent file.
• Never run terraform apply without showing the plan to the user first.