Prompt file imported from Danathar/zfs-kinoite-complex (
.github/prompts/diagnose-build-failure.prompt.md). Copyright stays with the author.
Diagnose a failed build
Goal: name the cause with evidence, and say which fix applies. Not the goal: make the build green by any means available.
A red run here is usually a fail-closed check doing exactly its job. Several of the checks below have no "fix" in this repository at all — the correct outcome is to stop and report what upstream did.
0. Did it fail, or was it cancelled?
gh run list --workflow build.yml --limit 10 --json createdAt,conclusion,eventThe build badge shows the most recent completed run, and a cancelled run
renders as failing exactly like a real failure. build.yml sets
cancel-in-progress: true, so back-to-back merges to main each cancel the
run before them.
A column of cancelled with no failure is a hypothesis, not a conclusion.
Check the event and the timing before reporting it: push runs bunched a few
minutes apart are a merge burst, but schedule and workflow_dispatch runs can
also be cancelled, and gh run cancel means a person stopped the production
workflow on purpose. Reporting "someone was merging quickly" when an operator
cancelled a run deliberately buries the more interesting question.
1. Which workflow, and does it publish?
gh run list --limit 20 --json databaseId,name,event,conclusion,headBranchBuild And Promote Main Image (build.yml) is the only one that signs and
promotes :latest. build-pr.yml and build-branch.yml do not promote, and
build-branch.yml publishes unsigned. If the red run is one of those two,
nothing reached :latest and the urgency is lower — say so.
2. Find the step, not the job
gh run view <run-id> --log-failed | tail -60Every workflow step that decides anything runs python3 -m ci_tools.cli <command>, and a known refusal exits 1 with one line on stderr. That line is
the diagnosis. Read it before forming a theory.
3. Match the message to the guard
These are the fail-closed refusals. Each names a real condition, and the response to each is different:
| The message says | What actually happened | What to do |
|---|---|---|
does not provide a kmod-zfs for primary kernel <k> at ZFS version <v> even after a rebuild |
The akmods cache resolves its own OpenZFS version independently of this repo, and the two disagree. Continuing would label the image org.zfs-kinoite-complex.zfs-version=<v> while shipping something else. |
Upstream problem. Establish whether the ZFS line supports that kernel yet. Do not loosen the match. |
REQUIRE_MATCH=true needs AKMODS_IMAGE_PINNED, the digest ref pin-akmods-cache published. |
The "Verify the rebuilt cache" step in .github/actions/prepare-main-akmods/action.yml ran without the digest the "Resolve shared akmods cache digest" step should have handed it. The rebuilt cache is already on the shared tag, but nothing has been built from it, signed or promoted. |
A repository problem: look at how action.yml passes steps.pin_akmods.outputs.akmods_image_pinned into that step. Do not drop REQUIRE_MATCH or let the check read the shared tag instead: the tag can point at a different image than the one this run signs (AGENTS.md section 0 rule 1). |
Refusing to check akmods cache ref <ref>: expected ghcr.io/<org>/<repo>@sha256:<64 hex digits>, the digest ref pin-akmods-cache publishes for this repository. |
The pinned cache ref the verify step was given is not a digest in this repository's own akmods cache — a tag, another repository or owner, or a truncated digest. Nothing was read from the registry and, as above, nothing has been built, signed or promoted. | A repository problem in whatever produced that ref, normally pin-akmods-cache. Do not widen the accepted pattern: checking a tag or another repository's image would vouch for something other than what this run builds from (AGENTS.md section 0 rule 1). |
Promoted digest mismatch: ... produced <a>, expected <b> |
skopeo copy did not preserve the digest. The copy already happened — promote_stable.py copies, then re-reads the destination — so if the destination named is :latest, the tag has already moved to the unexpected digest. |
Check which destination the message names. For an audit tag, stop and investigate. For :latest, treat it as live and escalate immediately: report what :latest currently resolves to, what the last known-good signed digest was, and that machines tracking it will pull whatever is there now. Restoring the tag is a promotion, so it is the maintainer's to perform — propose it, do not do it (AGENTS.md section 0 rule 6). Never bypass the guard itself. |
Refusing to promote run <n>: run <m> has already promoted a newer image to latest. |
The promote job of an older build ran after a newer run had already promoted — almost always a re-run of a finished run, which keeps its old run number. Moving :latest now would move it backwards, and an older image may be unable to import pools a newer one upgraded. The check runs before either copy, so no tag moved; :latest is still run <m>'s image. |
Nothing to fix and nothing to re-run. Say that the re-run was refused and that :latest did not move. Never delete stable-* tags to get past it (AGENTS.md section 0 rule 1). A deliberate rollback is a new run, which gets a higher run number, and is the maintainer's call (AGENTS.md section 0 rule 6). If a new run is refused too, promotion is running under a workflow whose run count is lower than the existing stable-* tags — escalate. |
Missing tag list in skopeo list-tags output for <ref> |
The registry answered the tag listing with no tag list, so the promote job could not tell whether a newer run had already promoted. It fails closed before any tag moves. | Usually a transient registry failure (see step 4). |
GITHUB_RUN_NUMBER is not a run number: <value> |
The promote job's run number was not a plain number, so it cannot be compared with the stable-<run>-<sha> tags. No tag moved. |
A repository problem: look at how build.yml sets the promote job's environment. |
Failed to resolve digest for <ref> / Missing digest in skopeo inspect output for <ref> |
A registry read returned a manifest with no digest — usually a transient registry or CDN failure, sometimes a deleted tag. | Check whether the ref still exists. If it does, this is very likely transient (see step 4). |
SIGNING_SECRET is empty; cannot sign published image. |
The signing job ran without its secret, typically because the run came from a context that does not get one. | Confirm the trigger. A branch or fork run is not supposed to sign. |
Missing required verification key file: <path> |
cosign.pub is not where the signing or promotion step expects it. |
A repository problem, not upstream. |
Replay lock file not found: <path> |
Replay mode was requested with no lock file. | See replay-a-build.prompt.md. |
4. Rule out a third-party service before proposing anything
Every workflow here pulls from quay.io, pushes to ghcr.io, and bootstraps
cosign from Sigstore. Any of the three can turn a run red with nothing wrong
in this repository, and all three shapes have been seen:
| Message | What happened |
|---|---|
happened during read: unexpected EOF |
A quay.io CDN blob transfer died mid-download. |
writing blob: … received unexpected HTTP status: 500 |
ghcr.io server error on a layer upload. |
Error: trusted root is required when using new bundle format |
Sigstore's TUF CDN was unavailable — look for failed to download https://tuf-repo-cdn.sigstore.dev/…root.json, http status code: 403 just above it. |
The third one is a trap, twice over. It names a trusted root and a bundle
format, so it reads like a signing problem here. It is not — it fails while
installing cosign, not while using it.
But do not then conclude that nothing ran. install-signing-tools is invoked in
three separate build.yml jobs — preflight, sign-akmods-cache and
promote-stable — and again inside publish-native-image, which runs after
the transient image has been pushed. A 403 in preflight's copy means almost
nothing happened; the same 403 in promote-stable's means the candidate was
already built, signed and published. Identify which job's installer failed
before describing the state of the run.
For any of them: say so, name the job and the step, and note that a re-run is
the maintainer's call — this repo does not retry automatically, and a re-run of
build.yml publishes.
5. Distinguish "the pipeline is green" from "the image is good"
If you re-check after a re-run, say which run you read. A successful run proves the build completed. It is not evidence the image is safe to boot (AGENTS.md section 0 rule 4).
6. What the answer looks like
State, in this order:
- Which run and which step, with the link.
- The guard message, quoted.
- Whether the run was cancelled rather than failed, or the cause is upstream, transient, or in this repository.
- The fix — or that there is no fix here and what has to change upstream.
If item 3 is anything other than "in this repository", stop there. Proposing a code change to get past a guard that is correctly refusing is the failure mode this repository cares most about.
