notifly-sqs-dlq-alarm-checking

name	notifly-sqs-dlq-alarm-checking
description	Check whether a Notifly SQS DLQ has CloudWatch alerting configured, verify it live in AWS, and add the missing alarm via Terraform + PR workflow.
version	1.0.0
author	Hermes Agent
license	MIT
metadata	{"hermes":{"tags":["notifly","sqs","dlq","cloudwatch","terraform","github","alarms"]}}

Notifly SQS DLQ Alarm Checking

Use when asked whether a Notifly SQS DLQ has alerting, or to add missing DLQ alarms in team-michael/notifly-event.

If the user asks why an existing DLQ alarm fired rather than whether alerting exists, use notifly-alert-live-investigation as the umbrella and its references/sqs-dlq-root-cause.md checklist. In particular, do not stop at alarm history: inspect/preserve the DLQ message, decode Notifly base64+gzip bodies, compare SQS Received vs Deleted, and correlate VisibilityTimeout/maxReceiveCount with the DLQ-visible timestamp.

Source of truth

• Terraform root: infra/terraform/prod/ap-northeast-2/sqs
• Queue inventory: infra/terraform/prod/ap-northeast-2/sqs/queues.tf
• DLQ queue definitions live under local.inventory_queues
• CloudWatch alarm definitions live under local.inventory_metric_alarm_details
• Actual alarms are generated by:
  • resource "aws_cloudwatch_metric_alarm" "metric_alarms"
  • for_each = local.inventory_metric_alarm_details

Important: a queue appearing in cloudwatch/dashboards.tf only means it is graphed on a dashboard. That does not imply an alarm exists.

Alarm pattern used in this repo

Existing DLQ alarms follow this exact pattern:

• Alarm name: <queue-name> has been created
• Metric: AWS/SQS / ApproximateNumberOfMessagesVisible
• Threshold: GreaterThanOrEqualToThreshold, 1
• Period: 60
• Statistic: Sum
• evaluation_periods = 1
• datapoints_to_alarm = 1
• SNS action: data.aws_sns_topic.topics["cloudwatch-incidents-notifly"].arn

Despite the name, has been created means "DLQ has visible messages", not queue resource creation.

Investigation workflow

1. Check Terraform first

Search queues.tf for the queue name in both sections:

• queue definition under inventory_queues
• alarm definition under inventory_metric_alarm_details

If the queue exists but there is no matching "<queue>-dlq has been created" entry, Terraform is missing the alert.

2. Check live AWS

Use Python via terminal, not execute_code, so shell AWS credentials are available.

Check queue attributes with boto3 SQS:

• ApproximateNumberOfMessages
• RedrivePolicy
• RedriveAllowPolicy

Check live alarms with boto3 CloudWatch:

• describe_alarms()
• filter on Dimensions[].Name == "QueueName" and the exact queue name

If Terraform lacks the alarm and CloudWatch returns zero matching alarms, the DLQ is not alerting.

Change workflow

3. Add the missing alarm

Insert a new entry near the top of local.inventory_metric_alarm_details in queues.tf:

"cafe24-worker-queue-dlq has been created" = {
  "actions_enabled" = true
  "alarm_actions" = [
    data.aws_sns_topic.topics["cloudwatch-incidents-notifly"].arn,
  ]
  "comparison_operator" = "GreaterThanOrEqualToThreshold"
  "datapoints_to_alarm" = 1
  "dimensions" = [
    {
      "name"  = "QueueName"
      "value" = "cafe24-worker-queue-dlq"
    },
  ]
  "evaluation_periods"        = 1
  "extended_statistic"        = null
  "insufficient_data_actions" = []
  "metric_name"               = "ApproximateNumberOfMessagesVisible"
  "namespace"                 = "AWS/SQS"
  "ok_actions"                = []
  "period"                    = 60
  "statistic"                 = "Sum"
  "threshold"                 = 1.0
  "treat_missing_data"        = "missing"
  "alarm_description"         = null
}

Adapt queue name as needed for other DLQs.

Validation

Run from repo root:

terraform fmt infra/terraform/prod/ap-northeast-2/sqs/queues.tf
terraform -chdir=infra/terraform/prod/ap-northeast-2/sqs init -backend=false
terraform -chdir=infra/terraform/prod/ap-northeast-2/sqs validate
tflint --chdir=infra/terraform/prod/ap-northeast-2/sqs

Pitfall: .terraform.lock.hcl

terraform init -backend=false may modify infra/terraform/prod/ap-northeast-2/sqs/.terraform.lock.hcl by adding an extra provider hash for the current platform. Do not commit that incidental lockfile change unless intended.

If you revert .terraform.lock.hcl after init, terraform validate may fail because the cached provider checksum no longer matches the reverted lock file. In that case, re-run terraform init -backend=false before validate/tflint.

PR workflow notes for this repo

• Use semantic commit subjects only; do not prepend non-standard markers.
• Use git author:
  • Andrej Karpathy <[e-mail]>
• After opening the PR, check:
  • review comments
  • issue comments
  • checks / CI results
• Do not wait for Cloudflare preview unless the user explicitly asks.

Live verification notes

For AWS/GitHub access in this environment:

• AWS creds are typically available in shell env; use terminal + boto3.
• aws CLI may be unavailable.
• GitHub token can be read from ~/.hermes/.env if gh is unavailable.
• Do not rely on parsing an authenticated git remote for the token; remotes may be redacted.