dd-gcp-integration

name	dd-gcp-integration
description	Set up the Datadog Google Cloud integration with Terraform - creates a service account in the host project, lets Datadog's delegate principal impersonate it via roles/iam.serviceAccountTokenCreator (no service-account keys), enables the required APIs, grants the monitoring roles across the chosen projects and folders, and registers the account through datadog_integration_gcp_sts. Use when the user wants to monitor GCP resources such as Compute Engine, Cloud SQL, GKE, Cloud Run, or Pub/Sub, wants to connect a GCP project or folder or organization to Datadog, or asks to set up or repair the GCP integration. Does not set up log forwarding.
metadata	{"version":"1.0.0","author":"datadog-labs","repository":"https://github.com/datadog-labs/agent-skills","tags":"datadog,gcp,google-cloud,integration,terraform,cloud","alwaysApply":"false","tools":"terraform"}

Datadog GCP Integration

You are helping a user set up the Datadog GCP integration using Terraform.

The integration creates a GCP service account in the customer's project and grants Datadog's delegate principal the ability to impersonate it via roles/iam.serviceAccountTokenCreator. This avoids long-lived keys entirely.

This is a hands-on setup: run the commands yourself as part of the conversation rather than handing the user a list, keep them in the loop, and pause for confirmation before terraform apply.

Phase 0: Preflight

Terraform or OpenTofu. Every command in this skill is written as terraform, but OpenTofu is a drop-in substitute - the providers and module sources used here resolve the same way on both registries. Check which binary the user actually has before Phase 1:

command -v terraform tofu

If only tofu is on the PATH, read every terraform <subcommand> below as tofu <subcommand>. If both are present, ask which one the user wants rather than guessing.

Datadog credentials. Load DD_SITE / DD_API_KEY / DD_APP_KEY from the environment (falling back to .env.local / .env) and validate both keys - they fail independently:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
: "${DD_SITE:=datadoghq.com}"
echo "DD_SITE=${DD_SITE}"
echo "DD_API_KEY=$([ -n "${DD_API_KEY:-}" ] && echo set || echo UNSET)   DD_APP_KEY=$([ -n "${DD_APP_KEY:-}" ] && echo set || echo UNSET)"
printf 'DD-API-KEY: %s\n' "$DD_API_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "validate:     HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v1/validate"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n'   \
  | curl -sS --max-time 20 -o /dev/null -w  \
      -H @- 

Result	Meaning	What to do
Both 200	Keys are good for this site	Continue to Phase 1
validate is 403	The API key is invalid, or belongs to a different region than DD_SITE	Ask which site the key belongs to, fix DD_SITE, re-check
validate 200, current_user 403	The app key is wrong or from another region - not the API key	Get one from <APP_BASE>/organization-settings/application-keys
Either key unset	Nothing to validate	On a commercial site, run the dd-account-setup skill, then come back. On ddog-gov.com or us2.ddog-gov.com, ask the user for the keys directly - that skill validates DD_SITE against a list that excludes both government sites and will reject them

App URL. The Datadog app host is not app.${DD_SITE} for every site. It is https://app.datadoghq.com (US1), https://app.datadoghq.eu (EU1), https://app.ddog-gov.com (Gov), and for every other site it is https://${DD_SITE} itself - https://us3.datadoghq.com, https://us5.datadoghq.com, https://ap1.datadoghq.com, https://ap2.datadoghq.com, https://uk1.datadoghq.com. Resolve it once and substitute it wherever <APP_BASE> appears below. Full list: https://docs.datadoghq.com/getting_started/site/

Remember the resolved DD_SITE. Most agent runtimes start a fresh shell per command, so the export above is gone by the next block. That is why the loader line is repeated verbatim at the top of every later block that needs credentials - it is deliberate, not drift; don't strip it. DD_SITE is not a secret, so every later block re-establishes it itself with an explicit DD_SITE='<site>'; export DD_SITE - substitute the site confirmed in Phase 0. It is a plain assignment rather than : "${DD_SITE:=...}" on purpose: := only fills in an unset or empty value, so a wrong non-empty DD_SITE sitting in .env would survive it and every call would go to the wrong region. The keys are guarded with :? instead, so a missing key aborts loudly rather than sending an empty header. Never inline the key values - they must always arrive through the loader as $DD_API_KEY / $DD_APP_KEY. Two consequences follow, and both are deliberate:

• Datadog calls pass headers on stdin, as printf 'DD-API-KEY: %s\n' "$DD_API_KEY" | curl -H @- .... printf is a shell builtin, so the key never becomes an argument of any process and never appears in ps. Writing -H "DD-API-KEY: $DD_API_KEY" instead would put it in curl's argv. (-H @- needs curl 7.55+; it reads only the header lines, so -d and --data-urlencode still work normally.)
• Terraform never receives the keys as values at all for AWS, Azure, and GCP: the Datadog provider reads DD_API_KEY / DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan. (OCI is the exception - its module needs them as inputs, so there they travel as TF_VAR_*.)

Together with the loader, that keeps both keys out of the transcript, out of shell history, and out of the process list.

Tools. terraform is required, plus jq for Phase 1. The gcloud CLI is optional for discovery, but the google provider needs ambient GCP credentials either way (gcloud auth application-default login, or GOOGLE_APPLICATION_CREDENTIALS):

command -v terraform || echo "MISSING terraform - https://developer.hashicorp.com/terraform/install"
command -v jq || echo "MISSING jq - needed to read the delegate email in Phase 1"
command -v gcloud >/dev/null 2>&1 && echo "gcloud: available" || echo "gcloud: not installed"

If jq isn't available, read data.attributes.delegate_account_email out of the raw JSON response in Phase 1 yourself instead of piping through jq.

The snippets here are POSIX shell. Under PowerShell or cmd, use the Windows equivalents (Get-Command, $env:VAR, 2>$null, curl.exe) - same calls, same order.

Phase 1: Gather Datadog Delegate Principal

Datadog impersonates the customer's service account through a per-org delegate service account. Get its email - GET first, and only POST if the org has no delegate yet, so an existing one is reused rather than re-created:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET \
      -H @- "https://api.${DD_SITE}/api/v2/integration/gcp/sts_delegate")
code=$(   |  -1); body=$(   | sed )
 [  =  ]; 
   
  resp=$(    \
    | curl -sS -w  -X POST -H  -d  \
        -H @- )
  code=$(   |  -1); body=$(   | sed )
 [  !=  ]; 
   
    
   
   
   
   1

[  =  ] || {  ;   ;  1; }
email=$(   | jq -r )
  
  *@*.iam.gserviceaccount.com)   ;;
  *)  ;   ;  1 ;;

Remember the printed email as DATADOG_PRINCIPAL_ID - the Terraform below needs it. It is org-specific and cannot be hardcoded.

Never continue on an empty or null email. The guards above exist because curl -s | jq -r on an error response prints null, which would silently become member = "serviceAccount:null" in the Terraform and produce an integration that cannot authenticate.

If jq is unavailable, run the block exactly as written except the email=$(... | jq -r ...) line - do not remove the printf ... | curl -H @- pipeline, which is what supplies authentication. Read data.attributes.delegate_account_email out of $body yourself and apply the same rule: refuse to proceed unless it looks like a ...@....iam.gserviceaccount.com address.

Phase 2: Determine Scope

Ask the user if they already know which GCP project IDs and/or folder IDs they want Datadog to monitor.

If they do, collect:

• Which GCP project should host the Datadog service account (the "host project")?
• The list of project IDs and/or folder IDs to monitor.

If they don't know or want help figuring it out:

If the gcloud CLI is available, offer to discover their GCP environment. Explain that you will use gcloud to list their organizations, folders, and projects so they can pick which ones to monitor. This is best-effort - run each command independently and work with whatever succeeds:

Organizations:

gcloud organizations list --format="table(displayName, name)"

Folders (uses the REST API to search all active folders across the org; also requires gcloud for the access token):

printf 'Authorization: Bearer %s\n' "$(gcloud auth print-access-token)" \
  | curl -sS -H @- -H "Content-Type: application/json" \
      -d '{"query": "lifecycleState=ACTIVE"}' \
      "https://cloudresourcemanager.googleapis.com/v2/folders:search"

If the response contains a nextPageToken, paginate by adding "pageToken": "<token>" to the request body until all folders are retrieved.

Expand nested folders yourself. The search above returns every active folder in the org, so use it to build the full descendant set for whatever the user picks: for each chosen folder, collect every folder whose parent chain leads back to it, then list the projects of all of them:

# replace these with the chosen folder and every folder nested beneath it
set -- 123456789012 987654321098
for folder do
  gcloud projects list --filter="parent.id=${folder} AND lifecycleState=ACTIVE AND NOT projectId:sys*" \
    --format="value(projectId)"
done | sort -u

Put that expanded list into the Terraform's project_ids in addition to keeping the chosen folders in folder_ids. The reason is in the template: folder-level IAM is inherited by every descendant project, but API enablement is not, and the google_projects data source matches on immediate parent only. Folders in folder_ids cover IAM (including projects created later); the expanded project_ids is what actually enables the required APIs in each existing project. Skipping the expansion produces an integration that looks configured but silently collects nothing from projects in sub-folders.

Projects (active only, excluding system projects):

gcloud projects list --filter="lifecycleState=ACTIVE AND NOT projectId:sys*" --format="table(projectId, name, parent.id)"

If any individual command fails (e.g., the user lacks permission to list organizations or folders), inform the user which command failed and why, but continue with whatever information was successfully retrieved. For example, if they can list projects but not folders, proceed with project-level setup.

Otherwise (gcloud not installed or not authenticated), ask the user to gather the IDs from the GCP Console:

• Project IDs: console.cloud.google.com → project picker (top bar) → each project's ID is shown next to its name.
• Folder IDs (optional): console.cloud.google.com → IAM & Admin → Manage Resources → expand the org tree; folder IDs are visible in the resource manager.
• Every project inside those folders, including nested sub-folders - expand the whole subtree in Manage Resources and collect each project ID. This is not optional busywork: the Terraform enables the required APIs per project, so any project you don't list gets folder-inherited IAM and no API enablement, and collects nothing. If the user won't enumerate them, tell them folder-scoped setup needs gcloud (which can expand the subtree automatically) and offer project-scoped setup instead.
• Host project: ask which project the Datadog service account should live in.

Present whatever results were gathered in a readable format and let the user choose:

• Specific projects: list of project IDs
• Folders: list of folder IDs (Datadog will discover all active projects within them)
• Both: a combination of explicit projects and folders

Then ask which project should host the Datadog service account.

Phase 3: Generate and Apply Terraform

Check what already exists - local Terraform first, then Datadog

Do this before generating or applying anything, and in this order. Local state first, because it decides whether an existing integration is something you can update or something you must not touch:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print
# A project is "present" if it has configuration - .terraform/ may simply not exist yet on a fresh clone
# with a remote backend, and terraform.tfstate does not exist at all when state is remote.
if [ -n "$(find . -maxdepth 1 -type f \( -name '*.tf' -o -name '*.tf.json' \) -print -quit)" ]; then
  terraform init -input=false >/dev/null || { echo "terraform init failed - resolve that before concluding anything about existing state"; exit 1; }
  out=$(terraform state list 2>&1); rc=$?
  if [ "$rc" -ne 0 ]; then
    case $out in
      *'No state file'*|*'no state'*|*'Backend initialization required'*)
        echo "project is initialized but has no state yet - treat as a clean install" ;;
      *)
        printf '%s\n' "$out"
        echo "could not read state (backend or credentials problem) - do NOT treat this as 'nothing exists'"; exit 1 ;;
    esac
  elif [ -z "$out" ]; then
    echo "state is empty - treat as a clean install"
  else
    printf '%s\n' "" | grep -F  ||  
  

   

Match the exact resource address datadog_integration_gcp_sts, not a loose grep -i datadog: unrelated Datadog resources, or cloud IAM left behind by a partial apply, would otherwise read as a managed integration.

Then ask Datadog what it already has:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET -H @- "https://api.${DD_SITE}/api/v2/integration/gcp/accounts")
code=$(printf   |  -1); body=$(   | sed )
[  =  ] || {  ;   ;  1; }
  

Now reconcile the two answers before doing anything:

• Not in Datadog, nothing in local state - a clean install. Continue.

• In Datadog and present in local state - this is the update/repair case, not a duplicate. Continue into the Terraform below as a change to the existing resources, and let the plan show what it will alter.

• In Datadog but absent from local state (a service account for the host project from Phase 2 is already registered) - stop. Applying would either create a duplicate or fight with whatever manages it. Say so plainly and offer the options: import the existing object into this project (terraform import datadog_integration_gcp_sts.datadog_integration "<integration-uuid>" - the uuid comes from GET /api/v2/integration/gcp/accounts, not the project id or service-account email), manage it where it is already managed, or delete it in Datadog first. Only continue if the user picks one and confirms. Two things about importing, in this order. Generate the configuration first (the Terraform below, adapted to the identity that already exists - same role/app/service-account name), because terraform import binds an existing object to a configured resource address and fails without one. And importing the Datadog registration alone is not enough: the cloud-side identity (the IAM role, the app registration, the service account) is still outside state, so either import those too or reference them with data sources, or the next apply will try to create them again and collide.

  # the uuid comes from the accounts listing, not the project id or service-account email
  terraform import datadog_integration_gcp_sts.datadog_integration "<integration-uuid>"
  

• Present in local state but absent from Datadog - a partial or rolled-back install. The cloud-side resources may exist while the registration does not. Do not start from scratch: run the plan and let it show what is missing, and expect it to re-create only the registration.

Check for Existing Terraform

Before generating a new Terraform configuration, check if the user already has a Terraform project in the current directory or nearby:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print

If existing .tf files are found:

• Read them to understand what providers and resources are already configured.
• If a datadog provider already exists, reuse its configuration - do not create a duplicate.
• If a google provider already exists, reuse it.
• Only add the new resources needed (service account, IAM bindings, datadog_integration_gcp_sts) to the existing project. Do not regenerate providers, variables, or terraform blocks that already exist.
• If the user has a modular layout, create a new file like datadog-gcp-integration.tf for the Datadog resources.

If no existing Terraform is found, generate a standalone configuration.

The full HCL template - providers, API enablement, the service account, the token-creator binding for Datadog's delegate, the project and folder role bindings, and the datadog_integration_gcp_sts registration - is in references/terraform.md, along with the projects-only and folders-only variants. Read it now and emit it with the placeholders filled in.

Applying the Terraform

1. Replace all <PLACEHOLDER> values in the template with the actual values gathered:

   • <USER_FOLDER_IDS> and <USER_PROJECT_IDS> from Phase 2
   • <HOST_PROJECT_ID> from Phase 2
   • <DATADOG_PRINCIPAL_ID> from Phase 1
   • <DD_SITE> from Phase 0

2. If the user selected only projects (no folders), remove the folder_ids local, the google_projects.folder_projects data source, the google_folder_iam_member resource, and simplify all_project_ids to just local.project_ids.

3. If the user selected only folders (no explicit projects), still put the expanded descendant project list from Phase 2 into project_ids - do not set it to []. folder_ids grants IAM (and covers projects created later), but API enablement is per-project, so an empty project_ids leaves every existing project in those folders without the required APIs.

4. Run terraform init to install providers.

5. Plan, and save the plan to a file. The Datadog provider reads DD_API_KEY / DD_APP_KEY straight from the environment, so there are no root variables and no -var= arguments - nothing secret ends up in the plan file, in state, or on a command line:

   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 |  -d= -f2- | sed \'\'); [ -n  ] &&  ; ; 
   DD_SITE=;  DD_SITE   
   
   : ; : 
    077          
   terraform plan -out=tfplan
   

Getting the Most Out of Your Integration

Once terraform apply completes successfully, congratulate the user and let them know metrics typically arrive within 5-10 minutes. Then check for early metrics and show a widget.

Checking for Metrics

Give it a few seconds, then make a single query to the metrics API - substitute a project you actually put in monitoring scope for <MONITORED_PROJECT_ID> - not necessarily the host project, which the template allows to be out of scope:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
sleep 10
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -G -H @- "https://api.${DD_SITE}/api/v1/query" \
  --data-urlencode  \
  --data-urlencode  \
  --data-urlencode )
code=$(   |  -1); body=$(   | sed )
[  =  ] || {  ;   ;  1; }
  

Rendering the Widget

If the series array is non-empty, render an ASCII chart from the real data:

• Use the pointlist values to plot the line, scaling Y-axis to actual min/max.
• Use box-drawing characters (╭, ╰, ─, │, ┤) for the line.
• List the instance names from each series scope at the bottom.
• Show the top 3 series by average value if multiple are returned.

If the series array is empty, show this static preview instead and let the user know metrics are still propagating:

┌─────────────────────────────────────────────────────────┐
│  gcp.gce.instance.cpu.utilization   ▂▃▅▆▇▆▅▃▂▁▂▃▅  │
│  100% ┤                                          ╭──╮   │
│   75% ┤                    ╭───╮              ╭──╯  │   │
│   50% ┤              ╭────╯   ╰──╮     ╭────╯     │   │
│   25% ┤    ╭────────╯            ╰────╯           │   │
│    0% ┤────╯                                       │   │
│       └────────────────────────────────────────────┘   │
│                                                         │
│  Metrics are on their way - check back in a few minutes │
└─────────────────────────────────────────────────────────┘
Metrics Explorer: <APP_BASE>/metric/explorer?exp_metric=gcp.gce.instance.cpu.utilization

Confirm to the user that their integration is configured and data will appear shortly. All links use DD_SITE - construct them as <APP_BASE>/....

• GCP Integration tile: <APP_BASE>/integrations/google-cloud-platform - access the pre-built dashboard and verify the integration is active.
• Metrics Explorer: <APP_BASE>/metric/explorer?exp_metric=gcp.gce.instance.cpu.utilization - confirm data is flowing.
• Each GCP sub-integration (Cloud SQL, Cloud Run, Pub/Sub, GKE, etc.) has its own dashboard that activates automatically when metrics for that service are detected.

Recommended Monitors - suggest creating monitors for common GCP health signals at <APP_BASE>/monitors/create:

• Compute Engine CPU exceeding a threshold
• Cloud SQL connection counts approaching limits
• GKE nodes entering NotReady state
• Pub/Sub dead-letter queue growth

If resource collection was enabled:

• Resource Catalog: <APP_BASE>/infrastructure/catalog - browse Compute instances, Cloud SQL databases, GKE clusters, and more.
• Infrastructure Map: <APP_BASE>/infrastructure/map - visualize GCP infrastructure.

Explore more Datadog products:

• Log Management: <APP_BASE>/logs - stream GCP logs for centralized search and alerting. Setup: https://docs.datadoghq.com/integrations/google_cloud_platform/#log-collection
• APM & Traces: <APP_BASE>/apm/getting-started - distributed tracing for applications on Cloud Run, GKE, or Compute Engine.
• Notebooks: <APP_BASE>/notebook - shareable investigations combining metrics, logs, and events.

Important Notes

• Always confirm project IDs, folder IDs, and the host project with the user before generating Terraform.
• The user needs permission to create service accounts, enable services, and set IAM policy on every project and folder in scope: roles/resourcemanager.projectIamAdmin / roles/resourcemanager.folderIamAdmin, roles/iam.serviceAccountAdmin on the host project, and roles/serviceusage.serviceUsageAdmin on every project where the config enables an API (the host project included - it gets iam.googleapis.com and iamcredentials.googleapis.com). roles/serviceusage.serviceUsageConsumer, which the service account itself receives, is not enough to enable an API.
• The Datadog delegate principal ID is org-specific and must be fetched from the API - it cannot be hardcoded.
• The Datadog API and app keys are never passed to Terraform as values: the datadog provider reads DD_API_KEY and DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan. Don't declare key variables, and don't write the keys into a committed .tfvars file or any other persistent file.
• Never run terraform apply without showing the plan to the user first.