godebug

name	godebug
description	Stateless CLI debugger for Go applications using Delve. Use when debugging Go programs via command line, setting breakpoints, inspecting variables, stepping through code, or analyzing goroutines. Each command outputs JSON and exits - perfect for AI agents.

godebug - Stateless Go Debugger CLI

A single-command CLI debugger for Go applications. Each invocation runs one command, outputs structured JSON, and exits. Designed for AI agent tool calling.

Quick Start

# 1. Start Delve in background (recommended for programs with output)
dlv debug ./myapp --headless --api-version=2 --listen=:4445 --accept-multiclient &
sleep 2

# 2. Connect godebug
godebug connect localhost:4445

# 3. Set breakpoint
godebug --addr localhost:4445 break main.go:42

# 4. Run to breakpoint
godebug --addr localhost:4445 continue

# 5. Inspect state
godebug --addr localhost:4445 locals
godebug --addr localhost:4445 stack

# 6. End session
godebug --addr localhost:4445 quit

Note: For programs without stdout output, you can use godebug start ./myapp instead. See "Starting Delve" section for details on when to use each approach.

Prerequisites

Debug Symbols Required

Binaries must be compiled with debug symbols for breakpoints to work. Without debug symbols, breakpoints will not be hit and the program will run to completion.

# CORRECT - includes debug symbols, disables optimizations
go build -gcflags="all=-N -l" -o ./myapp .

# CORRECT - default go build includes symbols
go build -o ./myapp .

# WRONG - strips symbols, breakpoints won't work!
go build -ldflags "-s -w" -o ./myapp .

Flags explained:

• -gcflags="all=-N -l": Disables optimizations (-N) and inlining (-l) for all packages
• -ldflags "-s -w": Strips debug symbols (-s) and DWARF info (-w) - avoid this for debugging

If breakpoints are not being hit, rebuild the binary with debug symbols.

Starting Delve

There are two ways to start a debug session. Choose based on whether your program produces output.

Recommended: Manual Delve Start (Programs with Output)

Use this approach for most Go programs. Any program that uses fmt.Println, log.Println, or writes to stdout/stderr will cause SIGPIPE (exit code 13) with godebug start.

# 1. Start Delve in background (keeps stdout connected to terminal)
dlv debug ./myapp --headless --api-version=2 --listen=:4445 --accept-multiclient &
sleep 2  # Wait for server to start

# 2. Connect with godebug
godebug connect localhost:4445

# 3. Debug as normal
godebug --addr localhost:4445 break main.main
godebug --addr localhost:4445 continue

For pre-compiled binaries:

dlv exec ./myapp --headless --api-version=2 --listen=:4445 --accept-multiclient &
sleep 2
godebug connect localhost:4445

For tests:

dlv test ./pkg/... --headless --api-version=2 --listen=:4445 --accept-multiclient &
sleep 2
godebug connect localhost:4445

Key flags explained:

• --headless: Run without terminal UI (required for godebug)
• --api-version=2: Use Delve API v2 (required)
• --listen=:4445: Port to listen on (choose any free port)
• --accept-multiclient: Allow reconnection if connection drops
• &: Run in background so stdout stays connected to terminal

Alternative: godebug start (Silent Programs Only)

Use godebug start only for programs that produce no output:

# Only for programs without any fmt.Println, log.Println, etc.
godebug start ./myapp
# Returns: {"data": {"addr": "127.0.0.1:58656", ...}}

godebug --addr 127.0.0.1:58656 break main.main
godebug --addr 127.0.0.1:58656 continue

Why this limitation exists: godebug start launches Delve as a subprocess. When the command returns, the pipe for the target program's stdout is closed. Any subsequent fmt.Println in the target causes SIGPIPE (exit code 13), terminating the program before breakpoints are hit.

Quick Reference

Program Type	Start Method
Has fmt.Println, log.*, stdout writes	dlv debug ... & + godebug connect
Silent (no output)	godebug start
Tests	dlv test ... & + godebug connect
Pre-compiled binary	dlv exec ... & + godebug connect
Remote debugging	dlv debug --listen=0.0.0.0:4445 on remote

Cleanup

When done debugging, clean up the Delve process:

# End the debug session
godebug --addr localhost:4445 quit

# If Delve is still running in background
pkill -f "dlv debug"
# or
pkill -f "dlv exec"

Detection Criteria

Use this skill when:

• Debugging Go applications from command line
• Setting breakpoints and stepping through code
• Inspecting variables, stack traces, goroutines
• Attaching to running Go processes
• Testing with conditional breakpoints
• The user mentions godebug or stateless debugging

Debugging Workflow Order

The debugger is a tool of last resort, not first resort. Before starting a debug session, use faster tools in this order:

1. Run the program     →  Observe error messages, panic traces, output
2. Static analysis     →  Read the code, look for known antipatterns
3. Race detector       →  go run -race / go test -race (for concurrency bugs)
4. Targeted logging    →  Add temporary log statements at key points
5. Debugger            →  Only when above tools don't reveal the issue

When Simpler Tools Are Faster

Situation	Better Tool	Why
Concurrency bug suspected	go test -race	Pinpoints exact lines with no stepping
Panic with stack trace	Read the trace	Location already provided
Known bug pattern	Code review	Pattern recognition beats stepping
Need to see one value	Temporary log.Printf	Faster than breakpoint setup
Regression after change	git diff + review	Compare what changed

When the Debugger IS the Right Tool

Use godebug when:

• Runtime state inspection: You need to see actual variable values at specific moments
• Complex interactions: Multiple goroutines, timing-dependent behavior that's hard to reason about
• Unclear reproduction: Bug is intermittent or behavior doesn't match code reading
• No obvious pattern: You've reviewed code and used race detector but can't spot the issue
• Exploration: Understanding unfamiliar code paths by stepping through execution

AI Debugging Strategy

The Scientific Protocol

Stop "flailing" (randomly changing code). Follow this cycle:

1. Observation: Gather raw data (logs, panic traces) without interpretation
2. Hypothesis: Formulate a falsifiable theory (e.g., "The goroutine hangs because the channel is unbuffered")
3. Prediction: Define what must happen if the hypothesis is true
4. Experiment: Change one variable only. Run the test
5. Analysis: If prediction failed, revert the change. You eliminated one possibility

Symptom → Command Sequence

Symptom	First Command	If Result Shows	Then
Wrong variable value	locals	unexpected value	eval parent struct, stack for call path
Program crashes/panics	break on error line	stack trace	frame N + locals for each frame
Program hangs	goroutines	blocked goroutines	goroutine N + stack to find blocker
Race condition suspected	go test -race first	race location	break both locations
Wrong control flow	break at branch point	wrong branch taken	eval condition expression
Test fails	start --mode test	test location	break in test, then step
Intermittent bug	break --cond	specific state	locals + goroutines

Bug Type → Command Sequence

Data Bug (wrong value):

1. break <file>:<line> at assignment
2. continue
3. locals to see current state
4. eval "expr" for specific expressions
5. stack to trace where value came from
6. frame N + locals to inspect caller's state

Control Flow Bug (wrong branch):

1. break <file>:<line> at decision point
2. continue
3. eval "condition" to see boolean result
4. step to confirm which branch executes

Concurrency Bug (race/deadlock):

1. First: go test -race ./... outside debugger
2. break on sync primitive or shared variable
3. goroutines to see all goroutine states
4. goroutine N → stack → locals for each relevant goroutine
5. Look for: missing locks, wrong order, blocked channels

Deadlock Investigation:

1. goroutines - identify which are in runtime.gopark
2. goroutine N → stack for each blocked goroutine
3. Draw dependency: "Goroutine A holds Lock 1, waiting for Lock 2. Goroutine B holds Lock 2, waiting for Lock 1"
4. The cycle is the deadlock

Memory/Resource Leak:

1. Use pprof first: go tool pprof http://localhost:6060/debug/pprof/heap
2. break at allocation site identified by pprof
3. stack to see who allocates
4. eval "len(slice)" or eval "cap(slice)" to check growth

Debugging Do's and Don'ts

Mindset & Strategy

Category	DON'T	DO
Bias	Confirmation bias: "I know X is fine" → ignore X's logs	Falsification: "If it's NOT X, then this must return Y. Verify."
Focus	Tunnel vision: 4 hours in one function	Divide & conquer: verify inputs/outputs at boundaries first
Process	Shotgun: change 3 things at once	Isolation: change ONE thing, revert if no fix, try next
Ego	"The library is broken"	"What assumption have I made that is incorrect?"

Go-Specific Tactics

Category	DON'T	DO
Observability	fmt.Println("HERE 1") spam	godebug break + locals - no code changes
Concurrency	Stare at code imagining races	Run go test -race - let runtime prove it
Logging	log.Println("Error happened")	Structured: slog.Error("failed", "userID", id, "error", err)
Regressions	Manual checkout of random commits	git bisect run go test -run TestBroken ./...
Testing	Happy path only	Table-driven with nil, empty, edge cases
Deadlocks	Random time.Sleep() calls	Stack dump → identify runtime.gopark → draw lock graph

Global Flags

Flag	Description	Default
--addr	Delve server address (host:port)	Required for all commands except start
--output	Output format: json or text	json
--timeout	Operation timeout (e.g., 10s, 1m)	30s

Command Reference

Session Management

start - Start Debug Session

Starts a Delve debug server and returns the connection address.

# Debug mode (default) - compile and debug
godebug start ./cmd/myapp

# Test mode - debug tests
godebug start --mode test ./...

# Exec mode - debug pre-compiled binary
godebug start --mode exec ./binary

# With program arguments
godebug start ./cmd/myapp -- -port 8080

Flags:

• --mode: Debug mode: debug (default), test, or exec

Output:

{
  "success": true,
  "command": "start",
  "data": {
    "addr": "127.0.0.1:58656",
    "mode": "debug",
    "pid": 87833,
    "target": "./testdata/debugme"
  },
  "message": "Debug server started"
}

connect - Connect to Existing Server

Connect to a manually started Delve server.

# First, start Delve manually
dlv debug ./myapp --headless --api-version=2 --listen=:2345

# Then connect
godebug connect localhost:2345

Output:

{
  "success": true,
  "command": "connect",
  "data": {
    "addr": "127.0.0.1:2345",
    "running": false
  },
  "message": "Connected to debug server"
}

status - Show Debug State

godebug --addr 127.0.0.1:2345 status

Output:

{
  "success": true,
  "command": "status",
  "data": {
    "exited": false,
    "running": false
  },
  "message": "Process paused"
}

restart - Restart Program

Restarts the debugged program from the beginning. Rebuilds the binary if source changed.

godebug --addr 127.0.0.1:2345 restart

Output:

{
  "success": true,
  "command": "restart",
  "data": { ... },
  "message": "Program restarted"
}

quit - End Session

godebug --addr 127.0.0.1:2345 quit

Output:

{
  "success": true,
  "command": "quit",
  "message": "Debug session terminated"
}

Breakpoints

break - Set Breakpoint

# Set breakpoint at file:line
godebug --addr 127.0.0.1:2345 break main.go:36

# Set breakpoint at function
godebug --addr 127.0.0.1:2345 break main.innerFunc

# Conditional breakpoint - only stops when condition is true
godebug --addr 127.0.0.1:2345 break --cond "i > 2" main.go:42

Flags:

• --cond: Condition expression (e.g., "x > 10", "name == \"test\"")

File Path Resolution:

Breakpoint locations can be specified as:

• Short filename (e.g., main.go:42) - works if the file is unique in loaded sources
• Relative path (e.g., pkg/handler/main.go:42) - when multiple files share the same name
• Absolute path (e.g., /home/user/project/main.go:42) - always unambiguous

If you get "could not find file", use godebug sources to see how files are referenced.

Shell Quoting for Method Names:

Function names with special characters (parentheses, asterisks) must be quoted:

# WRONG - shell interprets ( and * as glob/subshell
godebug --addr $ADDR break sync.(*WaitGroup).Wait

# CORRECT - quote the function name
godebug --addr $ADDR break "sync.(*WaitGroup).Wait"
godebug --addr $ADDR break "sync.(*Mutex).Lock"
godebug --addr $ADDR break "bytes.(*Buffer).Write"

Output (standard breakpoint):

{
  "success": true,
  "command": "break",
  "data": {
    "file": "/path/to/main.go",
    "function": "main.innerFunc",
    "id": 1,
    "line": 36
  },
  "message": "Breakpoint 1 set"
}

Output (conditional breakpoint):

{
  "success": true,
  "command": "break",
  "data": {
    "condition": "i > 2",
    "file": "/path/to/main.go",
    "function": "main.processItems",
    "id": 1,
    "line": 42
  },
  "message": "Breakpoint 1 set"
}

breakpoints - List Breakpoints

godebug --addr 127.0.0.1:2345 breakpoints

Output:

{
  "success": true,
  "command": "breakpoints",
  "data": {
    "breakpoints": [
      {
        "enabled": true,
        "file": "/path/to/main.go",
        "function": "main.innerFunc",
        "id": 1,
        "line": 36
      }
    ],
    "count": 1
  },
  "message": "1 breakpoints"
}

clear - Remove Breakpoint

godebug --addr 127.0.0.1:2345 clear 1

Output:

{
  "success": true,
  "command": "clear",
  "data": {
    "file": "/path/to/main.go",
    "id": 1,
    "line": 36
  },
  "message": "Breakpoint 1 cleared"
}

Execution Control

continue - Resume Execution

godebug --addr 127.0.0.1:2345 continue

Output:

{
  "success": true,
  "command": "continue",
  "data": {
    "breakpoint": {
      "file": "/path/to/main.go",
      "id": 1,
      "line": 36
    },
    "exited": false,
    "goroutine": {
      "id": 1
    },
    "location": {
      "file": "/path/to/main.go",
      "function": "main.innerFunc",
      "line": 36
    },
    "running": false
  },
  "message": "Stopped at breakpoint"
}

next - Step Over

Execute next line, stepping over function calls.

godebug --addr 127.0.0.1:2345 next

Output:

{
  "success": true,
  "command": "next",
  "data": {
    "exited": false,
    "goroutine": {"id": 1},
    "location": {
      "file": "/path/to/main.go",
      "function": "main.middleFunc",
      "line": 31
    },
    "running": false
  },
  "message": "Stepped to next line"
}

step - Step Into

Step into function calls.

godebug --addr 127.0.0.1:2345 step

Output:

{
  "success": true,
  "command": "step",
  "data": {
    "exited": false,
    "goroutine": {"id": 1},
    "location": {
      "file": "/path/to/main.go",
      "function": "main.middleFunc",
      "line": 32
    },
    "running": false
  },
  "message": "Stepped into function"
}

stepout - Step Out

Step out of current function.

godebug --addr 127.0.0.1:2345 stepout

Output:

{
  "success": true,
  "command": "stepout",
  "data": {
    "exited": false,
    "goroutine": {"id": 1},
    "location": {
      "file": "/path/to/main.go",
      "function": "main.outerFunc",
      "line": 26
    },
    "running": false
  },
  "message": "Stepped out of function"
}

Variable Inspection

locals - Show Local Variables

godebug --addr 127.0.0.1:2345 locals

Output:

{
  "success": true,
  "command": "locals",
  "data": {
    "count": 3,
    "variables": [
      {
        "children": [
          {"name": "", "type": "int", "value": "2"},
          {"name": "", "type": "int", "value": "4"},
          {"name": "", "type": "int", "value": "6"}
        ],
        "name": "result",
        "type": "[]int",
        "value": ""
      },
      {"name": "i", "type": "int", "value": "3"},
      {"name": "item", "type": "int", "value": "4"}
    ]
  },
  "message": "3 local variables"
}

args - Show Function Arguments

godebug --addr 127.0.0.1:2345 args

Output:

{
  "success": true,
  "command": "args",
  "data": {
    "arguments": [
      {"name": "x", "type": "int", "value": "25"},
      {"name": "~r0", "type": "int", "value": "0"}
    ],
    "count": 2
  },
  "message": "2 arguments"
}

eval - Evaluate Expression

godebug --addr 127.0.0.1:2345 eval "x"
godebug --addr 127.0.0.1:2345 eval "x * 2"
godebug --addr 127.0.0.1:2345 eval "len(result)"

Output:

{
  "success": true,
  "command": "eval",
  "data": {
    "expression": "x",
    "name": "x",
    "type": "int",
    "value": "25"
  }
}

Stack Navigation

stack - Show Stack Trace

# Full stack trace
godebug --addr 127.0.0.1:2345 stack

# Limited depth
godebug --addr 127.0.0.1:2345 stack --depth 3

Flags:

• --depth: Maximum number of frames to show

Output:

{
  "success": true,
  "command": "stack",
  "data": {
    "count": 4,
    "frames": [
      {"file": "/path/to/main.go", "function": "main.innerFunc", "index": 0, "line": 36},
      {"file": "/path/to/main.go", "function": "main.middleFunc", "index": 1, "line": 31},
      {"file": "/path/to/main.go", "function": "main.outerFunc", "index": 2, "line": 26},
      {"file": "/path/to/main.go", "function": "main.main", "index": 3, "line": 7}
    ],
    "goroutineId": 1
  },
  "message": "4 frames"
}

frame - Switch Stack Frame

godebug --addr 127.0.0.1:2345 frame 1

Output:

{
  "success": true,
  "command": "frame",
  "data": {
    "file": "/path/to/main.go",
    "function": "main.middleFunc",
    "index": 1,
    "line": 31
  },
  "message": "Switched to frame 1"
}

Goroutine Management

goroutines - List All Goroutines

godebug --addr 127.0.0.1:2345 goroutines

Output:

{
  "success": true,
  "command": "goroutines",
  "data": {
    "count": 6,
    "goroutines": [
      {
        "id": 1,
        "location": {
          "file": "/path/to/main.go",
          "function": "main.innerFunc",
          "line": 36
        },
        "selected": true
      },
      {
        "id": 2,
        "location": {
          "file": "/usr/local/go/src/runtime/proc.go",
          "function": "runtime.gopark",
          "line": 461
        },
        "selected": false
      }
    ],
    "selectedId": 1
  },
  "message": "6 goroutines"
}

goroutine - Switch Goroutine

godebug --addr 127.0.0.1:2345 goroutine 2

Output:

{
  "success": true,
  "command": "goroutine",
  "data": {
    "id": 2,
    "location": {
      "file": "/usr/local/go/src/runtime/proc.go",
      "function": "runtime.gopark",
      "line": 461
    }
  },
  "message": "Switched to goroutine 2"
}

Source Code

list - Show Source Code

# Show source at current location
godebug --addr 127.0.0.1:2345 list

# Show with custom context (lines before/after)
godebug --addr 127.0.0.1:2345 list --context 3

Flags:

• --context: Number of lines before and after current line (default: 5)

Output:

{
  "success": true,
  "command": "list",
  "data": {
    "currentLine": 36,
    "file": "/path/to/main.go",
    "function": "main.innerFunc",
    "lines": [
      {"content": "func innerFunc(x int) int {", "current": false, "lineNumber": 35},
      {"content": "\treturn x * x // Breakpoint here", "current": true, "lineNumber": 36},
      {"content": "}", "current": false, "lineNumber": 37}
    ]
  },
  "message": "/path/to/main.go:36"
}

sources - List Source Files

godebug --addr 127.0.0.1:2345 sources

Output:

{
  "success": true,
  "command": "sources",
  "data": {
    "count": 310,
    "sources": [
      "/path/to/main.go",
      "/path/to/other.go",
      "/usr/local/go/src/fmt/print.go"
    ]
  },
  "message": "310 sources"
}

Core Workflows

Basic Debugging Workflow

# 1. Start session
godebug start ./cmd/myapp
# Save the addr from output: 127.0.0.1:58656

# 2. Set breakpoints
godebug --addr 127.0.0.1:58656 break main.go:42
godebug --addr 127.0.0.1:58656 break pkg/handler.go:100

# 3. Run to breakpoint
godebug --addr 127.0.0.1:58656 continue

# 4. Inspect state
godebug --addr 127.0.0.1:58656 locals
godebug --addr 127.0.0.1:58656 args
godebug --addr 127.0.0.1:58656 stack

# 5. Step through code
godebug --addr 127.0.0.1:58656 next    # Step over
godebug --addr 127.0.0.1:58656 step    # Step into
godebug --addr 127.0.0.1:58656 stepout # Step out

# 6. Continue or quit
godebug --addr 127.0.0.1:58656 continue
godebug --addr 127.0.0.1:58656 quit

Conditional Breakpoint Workflow

Use conditional breakpoints to stop only when specific conditions are met:

# Start session
godebug start ./myapp
# addr: 127.0.0.1:58656

# Break only when loop variable exceeds threshold
godebug --addr 127.0.0.1:58656 break --cond "i > 100" main.go:50

# Break only for specific user
godebug --addr 127.0.0.1:58656 break --cond "user.ID == 42" handlers.go:75

# Break when slice is empty
godebug --addr 127.0.0.1:58656 break --cond "len(items) == 0" process.go:30

# Continue - will only stop when condition is true
godebug --addr 127.0.0.1:58656 continue

Test Debugging Workflow

# 1. Start in test mode
godebug start --mode test ./pkg/...
# Save addr: 127.0.0.1:58656

# 2. Set breakpoint in test or code under test
godebug --addr 127.0.0.1:58656 break pkg/handler_test.go:25
godebug --addr 127.0.0.1:58656 break pkg/handler.go:50

# 3. Run tests to breakpoint
godebug --addr 127.0.0.1:58656 continue

# 4. Debug as normal
godebug --addr 127.0.0.1:58656 locals

Goroutine Debugging Workflow

# 1. Start session and run to interesting point
godebug start ./concurrent-app
godebug --addr 127.0.0.1:58656 break worker.go:30
godebug --addr 127.0.0.1:58656 continue

# 2. List all goroutines
godebug --addr 127.0.0.1:58656 goroutines

# 3. Switch to specific goroutine
godebug --addr 127.0.0.1:58656 goroutine 5

# 4. Inspect that goroutine's state
godebug --addr 127.0.0.1:58656 stack
godebug --addr 127.0.0.1:58656 locals

# 5. Switch back to main goroutine
godebug --addr 127.0.0.1:58656 goroutine 1

Remote Debugging Workflow

# On remote machine: start Delve server
dlv debug ./myapp --headless --api-version=2 --accept-multiclient --listen=0.0.0.0:2345

# Locally: connect to remote server
godebug connect remote-host:2345

# Debug as normal with --addr
godebug --addr remote-host:2345 break main.go:42
godebug --addr remote-host:2345 continue

Race Condition Debugging Workflow

Race conditions in concurrent Go code can be challenging to debug because they may execute faster than breakpoints can catch them. Use this workflow:

Step 1: Confirm the Race with Go's Race Detector

Before using the debugger, confirm the race condition exists:

# Run with race detector
go run -race ./myapp

# Or test with race detector
go test -race ./...

The race detector will report data races with stack traces showing where they occur.

Step 2: Set Breakpoints on Sync Primitives

For WaitGroup, Mutex, or channel issues, set breakpoints on the sync package methods:

# Start debug session
godebug start ./myapp

# Breakpoints on WaitGroup methods (note: must quote due to special chars)
godebug --addr $ADDR break "sync.(*WaitGroup).Add"
godebug --addr $ADDR break "sync.(*WaitGroup).Done"
godebug --addr $ADDR break "sync.(*WaitGroup).Wait"

# Breakpoints on Mutex methods
godebug --addr $ADDR break "sync.(*Mutex).Lock"
godebug --addr $ADDR break "sync.(*Mutex).Unlock"

# Breakpoints on channel operations (runtime)
godebug --addr $ADDR break "runtime.chansend1"
godebug --addr $ADDR break "runtime.chanrecv1"

Step 3: Set Breakpoints BEFORE Goroutine Creation

For race conditions involving goroutine startup, set breakpoints before the go statement, not inside the goroutine:

// Example buggy code:
for i := 0; i < 10; i++ {
    go func(id int) {
        wg.Add(1)  // BUG: Add() inside goroutine
        // ...
    }(i)
}
wg.Wait()  // May return immediately!

# Set breakpoint BEFORE the for loop, not inside the goroutine
godebug --addr $ADDR break main.go:15  # Line with 'for'

# Use step to watch goroutine creation
godebug --addr $ADDR step

Step 4: Use Conditional Breakpoints for Specific States

# Break when WaitGroup counter might be wrong
godebug --addr $ADDR break --cond "counter == 0" main.go:50

# Break on specific goroutine count
godebug --addr $ADDR break --cond "len(goroutines) > 5" worker.go:30

Step 5: Inspect All Goroutines

When stopped, examine all goroutines to understand the race:

# List all goroutines
godebug --addr $ADDR goroutines

# Switch to each goroutine and inspect
godebug --addr $ADDR goroutine 5
godebug --addr $ADDR stack
godebug --addr $ADDR locals

Common Race Patterns

Pattern	Symptom	Debug Strategy
WaitGroup.Add inside goroutine	Wait() returns early	Break on sync.(*WaitGroup).Add, check call location
Missing mutex lock	Data corruption	Break on shared variable access
Channel send/recv mismatch	Deadlock or panic	Break on channel operations
Closure capturing loop var	Wrong values	Break inside goroutine, check captured values

When Races Are Too Fast

If the race always wins and breakpoints never hit:

1. The bug is deterministic - analyze the code statically
2. Add time.Sleep() temporarily to slow down the race
3. Use GOMAXPROCS=1 to serialize goroutine execution:

   GOMAXPROCS=1 godebug start ./myapp
   

4. Set breakpoint at main.main and use step instead of continue

Best Practices

1. Track the Address

The --addr flag is required for all commands after start. Always capture and reuse the address:

# Good: Save the address
ADDR=$(godebug start ./myapp | jq -r '.data.addr')
godebug --addr $ADDR break main.go:42

# Or extract from JSON manually
# {"data": {"addr": "127.0.0.1:58656"}} -> use 127.0.0.1:58656

2. Use Conditional Breakpoints for Loops

Don't step through 1000 iterations - use conditions:

# Bad: Will stop on every iteration
godebug --addr $ADDR break main.go:42

# Good: Only stop when interesting
godebug --addr $ADDR break --cond "i == 999" main.go:42
godebug --addr $ADDR break --cond "err != nil" main.go:42

3. Check Status Before Commands

# Verify session state before continuing
godebug --addr $ADDR status
# If running: true, wait or use another command
# If exited: true, session ended

4. Use Stack Depth for Large Call Stacks

# Limit output for deep recursion
godebug --addr $ADDR stack --depth 5

5. Clean Up Sessions

Always quit when done to release resources:

godebug --addr $ADDR quit

Exit Codes

There are two types of exit codes to understand:

1. godebug CLI exit codes - The exit code returned by the godebug command itself
2. Target process exit status - The exit status of the debugged program (in JSON data.exitStatus)

godebug CLI Exit Codes

These are the exit codes returned by godebug commands. Use echo $? after running a command to check.

Code	Constant	Meaning	JSON Error Code
0	ExitSuccess	Command completed successfully	-
1	ExitGenericError	Unspecified error	INTERNAL_ERROR, EVAL_FAILED
2	ExitUsageError	Invalid arguments or flags	INVALID_ARGUMENT
3	ExitConnectionError	Cannot connect to Delve server	CONNECTION_FAILED, CONNECTION_REFUSED
4	ExitNotFound	Resource not found (breakpoint, goroutine, frame)	NOT_FOUND
124	ExitTimeout	Operation timed out (GNU timeout convention)	TIMEOUT
125	ExitProcessError	Target process error	PROCESS_EXITED

JSON Error Codes Reference:

Error Code	Description
CONNECTION_FAILED	Cannot reach the Delve server
CONNECTION_REFUSED	Server actively refused the connection
TIMEOUT	Operation exceeded time limit
INVALID_ARGUMENT	Bad input from user
NOT_FOUND	Requested resource doesn't exist
PROCESS_EXITED	Target program terminated
EVAL_FAILED	Expression evaluation failed
INTERNAL_ERROR	Unexpected internal error

Example:

godebug --addr 127.0.0.1:2345 break main.go:999
echo $?  # Returns 4 if line doesn't exist (NOT_FOUND)

godebug --addr 127.0.0.1:9999 status
echo $?  # Returns 3 if server not running (CONNECTION_REFUSED)

Target Process Exit Status (data.exitStatus)

When the debugged program exits, the exitStatus field in the JSON response shows how it terminated. This is separate from the CLI exit code.

Common values:

Status	Meaning
0	Program completed successfully
1	General error (often os.Exit(1) or log.Fatal())
2	Panic without recovery, or deadlock detected
n	Value passed to os.Exit(n)

Signal-based exits (128 + signal number):

Status	Signal	Meaning
130	SIGINT (2)	Interrupted (Ctrl+C)
131	SIGQUIT (3)	Quit with core dump
134	SIGABRT (6)	Aborted
137	SIGKILL (9)	Killed forcefully
139	SIGSEGV (11)	Segmentation fault
143	SIGTERM (15)	Terminated

Go-specific patterns:

Scenario	Exit Status	Notes
panic() without recovery	2	Stack trace printed to stderr
os.Exit(n)	n	Deferred functions NOT called
log.Fatal()	1	Calls os.Exit(1) after logging
Deadlock detected	2	"fatal error: all goroutines are asleep"
Race detector found race	66	When running with -race

Example JSON Response

When the target process exits:

{
  "success": true,
  "command": "continue",
  "data": {
    "exitStatus": 0,
    "exited": true,
    "running": false
  },
  "message": "Process exited"
}

Key fields:

• exited: true - The target process has terminated
• exitStatus - The exit code of the target process (not godebug)
• The godebug CLI itself returns exit code 0 (success) because the command worked

Distinguishing the Two

# Run godebug and capture both exit codes
OUTPUT=$(godebug --addr 127.0.0.1:2345 continue)
CLI_EXIT=$?

# CLI exit code (was the godebug command successful?)
echo "CLI exit: $CLI_EXIT"

# Target process exit status (how did the debugged program end?)
TARGET_EXIT=$(echo "$OUTPUT" | jq -r '.data.exitStatus // empty')
echo "Target exit: $TARGET_EXIT"

Troubleshooting

"Connection refused"

The Delve server isn't running or wrong address:

# Verify server is running
ps aux | grep dlv

# Try starting fresh
godebug start ./myapp

"No such file or directory"

Build the target first or use correct path:

# Ensure target exists
go build ./cmd/myapp
godebug start --mode exec ./cmd/myapp

"Could not attach to pid"

On macOS, you may need to codesign Delve:

# Check if dlv is codesigned
codesign -d -v $(which dlv)

Breakpoints Never Hit / Program Exits Immediately

This is often caused by missing debug symbols or race conditions.

Check 1: Debug symbols present?

# Rebuild with debug symbols
go build -gcflags="all=-N -l" -o ./myapp .

# Verify symbols exist (should show DWARF info)
go tool objdump ./myapp | head -20

Check 2: Is binary stripped?

# If built with -ldflags "-s -w", symbols are stripped
# Rebuild WITHOUT those flags
go build -o ./myapp .

Check 3: Race condition causing early exit?

# Set breakpoint at main.main first
godebug --addr $ADDR break main.main
godebug --addr $ADDR continue

# Then use step instead of continue
godebug --addr $ADDR step
godebug --addr $ADDR step

Check 4: Code path not executed?

# List sources to verify file is loaded
godebug --addr $ADDR sources | grep myfile

# Check breakpoints are actually set
godebug --addr $ADDR breakpoints

Breakpoint on Wrong Line

Compiler optimizations can move code. Disable optimizations:

go build -gcflags="all=-N -l" -o ./myapp .

"could not find function" Error

Function name may need quoting or different format:

# For methods with pointer receivers, quote the name
godebug --addr $ADDR break "sync.(*WaitGroup).Wait"

# For generic functions, include type parameters
godebug --addr $ADDR break "slices.Sort[int]"

Timeout Errors

Increase timeout for slow operations:

godebug --addr $ADDR --timeout 60s continue

Program Exits with Code 13 (SIGPIPE)

Exit code 13 means the program received SIGPIPE when trying to write to stdout/stderr. This is the most common issue when using godebug start.

Root cause: godebug start closes the stdout pipe after returning the server address. When the target program calls fmt.Println or similar, it gets SIGPIPE.

Solution: Use manual Delve start instead:

# Instead of: godebug start ./myapp
# Use:
dlv debug ./myapp --headless --api-version=2 --listen=:4445 --accept-multiclient &
sleep 2
godebug connect localhost:4445

See "Starting Delve" section above for full details.

Quick diagnosis:

# If you see this pattern, it's SIGPIPE:
godebug --addr $ADDR continue
# {"data": {"exitStatus": 13, "exited": true}, ...}

Output Interpretation

When continue Returns

JSON Field	Value	Meaning	Next Action
exited	true	Program ended	Check exitStatus, may need restart
exited	false	Stopped at breakpoint	locals, stack, eval to inspect
exitStatus	0	Clean exit	Bug may be logic, not crash
exitStatus	2	Panic or deadlock	Check stderr, goroutines
exitStatus	66	Race detected	Run with -race outside debugger
breakpoint.id	present	Stopped at breakpoint	Inspect state
running	true	Still executing	Wait or status to poll

When locals Returns

Pattern	Interpretation	Next Action
value: "<nil>"	Nil pointer/interface	Check initialization, stack for caller
value: "" (string)	Empty string	May be intentional or missing assignment
children: []	Empty slice/map	Check if expected, eval "cap(x)"
Complex nested struct	Large data	eval "x.SpecificField" for targeted inspection

When goroutines Returns

Pattern	Interpretation	Next Action
Many at runtime.gopark	Blocked waiting	Check what they wait for
At sync.(*Mutex).Lock	Waiting for lock	Find who holds lock
At runtime.chanrecv	Waiting on channel	Find sender
Only 1 at user code	Others blocked	Likely deadlock
Growing count	Goroutine leak	pprof goroutine profile

Error Recovery

error.code	Meaning	Recovery
CONNECTION_REFUSED	Server not running	start new session
NOT_FOUND	Invalid breakpoint/goroutine	breakpoints or goroutines to list valid
PROCESS_EXITED	Target ended	restart or quit + new start
TIMEOUT	Operation too slow	Increase --timeout, check for infinite loop
EVAL_FAILED	Bad expression	Check variable exists in scope

Multi-Tool Integration

godebug + Race Detector

Race detector finds locations; godebug inspects state.

# 1. Find race locations
go test -race ./... 2>&1 | tee race.log
# Extract: "main.go:42" and "main.go:67"

# 2. Debug with godebug
ADDR=$(godebug start ./myapp | jq -r '.data.addr')
godebug --addr $ADDR break main.go:42
godebug --addr $ADDR break main.go:67
godebug --addr $ADDR continue

# 3. Check goroutine states
godebug --addr $ADDR goroutines

# 4. Inspect each racing goroutine
godebug --addr $ADDR goroutine N
godebug --addr $ADDR locals
godebug --addr $ADDR stack

godebug + pprof (Memory Leaks)

pprof identifies allocation sites; godebug inspects context.

# 1. Capture heap profile
go tool pprof http://localhost:6060/debug/pprof/heap
(pprof) top10
# Identifies: "cache.go:89 allocates 500MB"

# 2. Debug allocation
ADDR=$(godebug start ./myapp | jq -r '.data.addr')
godebug --addr $ADDR break cache.go:89
godebug --addr $ADDR continue
godebug --addr $ADDR stack    # Who called this?
godebug --addr $ADDR locals   # What's allocated?

godebug + pprof (Goroutine Leaks)

# 1. Get goroutine profile
curl http://localhost:6060/debug/pprof/goroutine?debug=2 > goroutines.txt
# Look for many goroutines with same stack

# 2. Debug the leak
ADDR=$(godebug start ./myapp | jq -r '.data.addr')
godebug --addr $ADDR break worker.go:50  # Creation site
godebug --addr $ADDR continue
godebug --addr $ADDR locals    # What state causes leak?
godebug --addr $ADDR stack     # Full context

godebug + git bisect (Regressions)

# 1. Find breaking commit
git bisect start
git bisect bad HEAD
git bisect good v1.0.0
git bisect run go test -run TestBroken ./...
# Result: "abc123 is first bad commit"

# 2. Debug that commit
git checkout abc123
ADDR=$(godebug start ./myapp | jq -r '.data.addr')
godebug --addr $ADDR break <changed_file>:<line>
godebug --addr $ADDR continue
godebug --addr $ADDR locals

git bisect reset

Output Format

All commands output JSON with this structure:

{
  "success": true|false,
  "command": "command-name",
  "data": { ... },      // Command-specific data
  "message": "Human-readable summary",
  "error": {            // Only on failure
    "code": "ERROR_CODE",
    "message": "Error description"
  }
}

Use --output text for human-readable output instead of JSON.