| name | pulumi-debug-failed-operation |
| version | 1.1.1 |
| description | Debug a Pulumi update or preview that failed: read the failure Pulumi already
recorded, find what caused it, and fix it. Load this skill when the user asks
to debug, diagnose, or fix a failed update or preview, or points at a failing
`pulumi up` or `pulumi preview`. Don't load it for authoring new
infrastructure, migrations, or provider upgrades; those have their own skills.
|
Debug a failed Pulumi operation
A Pulumi operation has failed. Find what caused it and fix it. The user usually
points you at it, so start by working out which operation to debug, and confirm it
with the user before doing anything else. Pulumi recorded the error when the
operation failed, so once you know which operation it is you can read the error from
that record without running anything again.
The commands below reach Pulumi Cloud with pulumi api, a subcommand of the Pulumi
CLI that you run in your shell. Each one targets a stack by the explicit
{orgName}/{projectName}/{stackName} path you pass it, so you do not need that
stack selected locally to read its record. Selecting the stack matters later, when
you go to apply a fix.
Start from the operation the user gave you
The user usually supplies the operation as a set of fields: the org, project, stack,
and update version (or preview id) — most often stated in prose, for example "debug
update 161 of vvm-dev". You need these to address the API: {orgName},
{projectName}, {stackName}, and the version or preview id.
Fill any missing field from context. Take the org, project, or stack from the
currently selected stack (pulumi stack --show-name, pulumi stack ls) or
Pulumi.yaml. A missing version means the most recent update on that stack.
Briefly confirm which operation you landed on, its version or preview id and the
stack, before reading further. Keep it lightweight; they already told you.
Read what failed
A failed update and a failed preview both record engine events, and the error is in
the diagnostic messages inside those events. Using the fields you settled on above,
fetch the events and pull the messages out.
For a failed update, use the update path with the version number:
pulumi api /api/stacks/{orgName}/{projectName}/{stackName}/update/<version>/events \
| jq -r '.events[].diagnosticEvent | select(. != null) | "[\(.severity)] \(.message)"' \
| sed 's/<{%reset%}>//g'
For a failed preview, use the preview path with the preview id:
pulumi api /api/stacks/{orgName}/{projectName}/{stackName}/preview/<preview-id>/events \
| jq -r '.events[].diagnosticEvent | select(. != null) | "[\(.severity)] \(.message)"' \
| sed 's/<{%reset%}>//g'
Read every message, not only the ones tagged severity == "error". A provider
error carries that error tag, but a program error, which is the common case when
a preview fails, arrives as a stderr diagnostic tagged info#err. The trailing
sed strips terminal color codes that Pulumi embeds in the text, which otherwise
show up as <{%reset%}>.
Find the cause and where the fix belongs
An operation can fail with errors from more than one resource, so read all of the
diagnostics first, then work through each error. Trace every error back to the
resource that raised it (its URN and type), to where that resource is declared in
the program, and to the inputs that feed it.
The error text tells you what kind of problem it is, and that points to where the
fix belongs. A Pulumi fix lands in one of three places, and naming the right one
keeps you from editing code that was never the problem.
- The program. The code is wrong: a bad reference, a wrong type, an input the
provider rejected, or a value used before it had resolved. This is what a failed
preview usually reports, because the plan could not be built. Fix it by editing
the code.
- The state. The code is correct, but the stored state and the real cloud
resources disagree. Reconcile drift with
pulumi refresh, and bring a resource
that already exists outside the state under management with pulumi import
rather than recreating it. Note that an operation which failed partway through
applying may have already changed some resources, so check the current state
before you decide.
- The environment. The problem is outside Pulumi: credentials, permissions,
OIDC, or a quota. Fix the role, the ESC environment, or the capacity that the
provider rejected, rather than the resource code.
When a diagnostic is empty or too thin to act on, the real error usually isn't in
the record — it's in the log of whatever the resource shelled out to. Read it
there. If reaching it needs access you don't have (a token, a run, a not-found),
stop and tell the user what you're blocked on and the one thing you need from them.
Fix the cause
Make the smallest change that addresses the root cause. How you confirm the fix,
and how you deliver it, whether as a local edit or as a pull request, follow your
mode's workflow, not this skill.
If the user didn't say which operation
When the user gives you nothing to go on, debug their most recent operation on the
stack. The update list does not record who ran each update, so find it through the
API:
- Run
pulumi whoami to get the current user's login.
- Read the latest update and who requested it with
pulumi api /api/stacks/{orgName}/{projectName}/{stackName}/updates/latest, and
compare its requestedBy.githubLogin to the login from step 1.
- If they match, that update is the one to debug. If they do not, walk back one
version at a time with
pulumi api /api/stacks/{orgName}/{projectName}/{stackName}/updates/<n> until
requestedBy.githubLogin matches the user.
Tell the user which operation you landed on, its version, kind, and result, and
confirm it is the one they mean before going further.
