gpustack-operator-e2e

Imported from gpustack/gpustack-operator (.agents/skills/gpustack-operator-e2e/SKILL.md). Install upstream with npx skills add gpustack/gpustack-operator --skill gpustack-operator-e2e. Copyright stays with the author.

GPUStack Operator — local E2E verification

Deploy the operator to a Kubernetes cluster — the user's own by default, or one provisioned for the run and destroyed afterwards — and verify the scheduling chain end to end:

NFD labels nodes → DeviceManager detects accelerators → Worker profiles capacity
  → NodeFlavor/InstanceType reconcilers → ResourceFlavor
  → InstanceType (real CRD; .status = four-view EX/SH/SL/PT)
  → ClusterQueue (exactly one isolated CQ per pool — NO Cohort) → LocalQueue
  ⊢ node-devices AdmissionCheck gates per-card feasibility

• Chain detail → architecture.md (overview), then scheduling-chain.md for the reconcilers and admission.md for the five gates / four-view math these cases assert; the unified-pool refactor this suite tracks → specs/2026-06-29-instancetype-unified-pool-refactor.md.
• Accelerated cases run GPU-less by approximation: mock a fake accelerator NodeFeature + a per-card Devices ledger. The derivation and the four-view/AdmissionCheck math are NOT mocked — that is the verification.

Orchestration

Run as a test-orchestration lead (main agent) coordinating read-only domain specialists (Agent tool). Read orchestration.md before Phase 0 — it owns roles, phases, parallelism/rendezvous rules, report layout, and the fix-and-retest loop. Below are the operator-e2e specifics.

Hard rules (full list in orchestration.md; operator-e2e specifics):

• Pin the run to a user-confirmed context — confirm the active context is the intended cluster. Never switch on your own judgement. When the confirmed cluster is not the active context, do not switch: take its isolated kubeconfig with bash ../_e2e-lib/scripts/kube-context.sh <ctx> and prefix every command of the run with the KUBECONFIG=<path> it prints (details in orchestration.md). On the user's explicit say-so you may switch instead — then record the context to return to as an outstanding environment mutation and restore it in Phase 8. Adopting the context a cluster this run provisioned merged in itself needs no such record.
• Prefer the user's own cluster; provisioning is a separate opt-in and a cluster this run provisioned must be destroyed in Phase 8 — see cluster-provisioning.md.
• Offer a context checkpoint at the trigger points (see Flow); compacting is the user's action, and the lead hands over a focus block that carries the teardown obligation.
• Build locally when the nodes share this machine's image store; otherwise build remote and push — build-load.sh keeps PACKAGE_PUSH=false locally, and switches to a builder host + registry push when E2E_BUILDER_SSH / E2E_IMAGE_NAMESPACE are set inline (never written into a file).
• Touch only what this run creates — the Helm release, injected labels/NodeFeatures, mocked Devices, test workloads. Never touch the user's pre-existing resources.
• Confirm every mutating step (build-load.sh, deploy.sh, mutating cases, teardown.sh); read-only steps (preflight.sh, assert-core.sh, CASE 1) run unprompted.
• Specialists are read-only — the lead is the sole writer.

Layout:

• ../_e2e-lib/scripts/ — preflight.sh, build-load.sh <TAG>, deploy.sh <NS> <TAG> (installs only — it refuses an existing release; upgrade in place with helm upgrade, or teardown.sh first), assert-core.sh <NS>, teardown.sh <NS> (custom-resource drain, test artifacts, release uninstall and a CRD-drain verdict; the drain and the cleanup are both delegated to the chart's files/drain.sh and files/cleanup.sh), plus the cluster-lifecycle trio cluster-auth.sh <modality> (read-only), provision.sh <modality> and destroy.sh <modality>, and kube-context.sh <ctx> (read-only; targets a context that is not the current one).
• cases/case-N.sh <NS> — one scenario each; ends in a STATUS | CHECK | OBJECT table, exits non-zero on any FAIL.
• cases/_quota-lib.sh — sourced by the cases that short a pool by occupying it (50, 95): workload_charge reads what Kueue charged one replica's Workload, and pool_replica_bound turns it into an upper bound on the replicas the queue's quota holds, summed over every flavor. Not a case; never run it directly.
• cases/_rows-lib.sh — print_rows, the printer for that table. It splits a row on its first two | only, so an OBJECT that carries | prints whole; source it rather than writing another loop. Not a case; never run it directly.
• cases/_partition-lib.sh — sourced by the hardware-partition cases (24–32, 34) for node correlation, the MIG_NODE_SSH gate, profile/key discovery and pod plumbing. Not a case; never run it directly.
• cases/run-partition-block.sh <RAW_DIR> [NS] [CASES...] — runs that family in its required order, writing each case's raw log and exit code. Drive 24–32/34 with it rather than by hand: the ordering is a real constraint and the block outlives a context, so a step held only in conversation is one a compaction drops.
• cases/_rdma-lib.sh — sourced by the RDMA cases (80–85) for the precondition gate, the host-probe Pod and the ledger facts. Not a case; never run it directly.
• cases/run-rdma-block.sh [--report-only] <RAW_DIR> [NS] [CASES...] — --report-only prints, without touching the cluster, which of those readings this cluster can answer and what each unmet requirement is missing. A run prints the same matrix, then each case's exit code and whether its output carries NOTHING WAS VERIFIED — which is how a skip is told from a pass, since both exit 0.
• cases/_topology-tas-lib.sh — sourced by the TAS cases (87) for the zone-shape measurement, the apply and name-scoped teardown of a case-prefixed Topology/ResourceFlavor/ClusterQueue/LocalQueue set, the Workload admission readers, and the Workload→owner-Pod→ModelDeployment join. Not a case; never run it directly.
• cases/_serving-lib.sh — sourced by the serving cases (90, 91, 94): serving_profile <vendor> <engine> is the one place a vendor or engine difference is written — where a transfer is counted, how the leg is pinned, which routers render it — so aligning a vendor is an edit there, not in each case. Not a case; never run it directly.
• references/ — drain-recycle.md (per-case rationale + mock recipes), packaged-image-deploy.md (image-ref ↔ chart-values contract), rdma-host-shapes.md (which host shape answers which RDMA reading, and the one nothing answers); shared ../_e2e-lib/references/{orchestration,troubleshooting}.md.

Cases (locked titles)

Each case is self-contained; its header (see Case header contract) states goal / environment (incl. auto-skip) / inputs (MOCKED vs real) / expected / cleanup. Needs = the hardware/tooling the case requires; the exact auto-skip conditions live in the case's own Environment: field, which is what the script actually prints.

Each note below is something the lead must act on before or around a run. What a case does — its goal, environment, inputs, assertions and cleanup — lives in its own header, which the Case header contract below requires to be readable on its own; the index never restates it.

The Needs column is the one thing the index duplicates from a case header, so it is the one thing that can go stale. Change what a case requires and you must change that column in the same commit — nothing checks it. make lint docs verifies links, page structure and index labels; it never reads a Needs cell against the case it describes. Measured: case-45 was rewritten to name a Binding that deliberately does not exist, and its row went on demanding a Ready KVCachePoolBinding named by E2E_MD_BINDING until someone happened to read both. An index row is an assertion about code, and this one has no guard.

• Two disjoint accelerator families, two disjoint card populations. Logical slicing (software; the vendor preload library) is <base>.sliced plus its sizing keys, served only by a card that is not in a hardware partitioning mode. Physical partitioning (hardware; NVIDIA MIG) is <base>.partitioned plus one <base>.partitioned.<kind>-<profile> key per profile, served only by a card that is. A card serves exactly one family — which is why InstanceType carries four views (Accelerator(EX/SH/SL/PT)), and why a case deploying a logical slice must select a pool with a non-zero logical slice count, never merely a "sliceable" one. Normative reference: docs/accelerator-requests.md.

• Ask whether a pool BACKS the node; never match its rendered name. The webhook stamps each InstanceType with its own schedule labels — nodefeature.PoolScheduleLabels, via pkg/worker/webhooks/worker/instance_type.go — expressly "so it is selectable by the same discriminators its Devices and ResourceFlavors carry". So the pool backing a node is the one whose every label the node also carries, and the recipe is one predicate:

  # every discriminator the pool carries is a label the node carries too
  d = {k: v for k, v in pool_labels.items() if not k.startswith('schedule.gpustack.ai/')}
  backs = bool(d) and all(node_labels.get(k) == v for k, v in d.items())

  That covers the whole identity at once — kubernetes.io/os / arch, feature.gpustack.ai/acceleratable, acceleratable.feature.gpustack.ai/<accelerator group> and, only under instance-type-aware-cpu-manufacturer, general.feature.gpustack.ai/<general group>. Enumerating spec fields instead means getting that last one conditionally right: with CPU awareness off every pool's spec.generalGroup is generic, and with it on the pools split by CPU key, so a hand-written tuple is wrong in one mode or the other. schedule.gpustack.ai/* is the pool's own bookkeeping and never a node label, so it is excluded. Add the accelerator group the case targets — as an anchored suffix, acceleratorGroup.endswith('-' + groupID) — only to choose among the pools of a node carrying more than one accelerator group. The per-node group id from Devices.spec.groups[].id cannot carry the match on its own: it has no manufacturer, no os/arch and no CPU group.

  Three ways a looser match picks the wrong pool, all silent:

  • A substring of the InstanceType name. Real product tokens nest — a10 inside a100, l40 inside l40s, h20 inside h200 — so an A10 node's group id also matches the A100 pool's name. Which one wins then depends on API list order, not on anything the case controls. The anchored suffix above is what closes this.
  • Ignoring os/arch. The name carries -<os>-<arch> and the group id does not, so on a mixed-arch cluster with one accelerator model a node's group id matches both arch pools, and linux-amd64 sorts ahead of linux-arm64.
  • Ignoring the general group. Under CPU awareness two nodes with the same accelerator and os/arch but different CPUs are different pools, and a tuple that stops at the accelerator matches both.

  The damage is not a clean error: the claim is submitted through the wrong pool's entrance LocalQueue, whose ResourceFlavor selects other nodes, so a node-pinned Pod is simply never admitted and the case fails on a timeout that reads like a placement defect. Worse where a ceiling comes from the pool — CASE 11 reads its over-commit limit from Status.Detail.Memory — because a mismatched pool with larger cards makes a real over-commit pass vacuously.

• spec.os/spec.arch has no case of its own — CASE 1 (cpu pool) and CASE 6 (accelerated) assert it inline.

• CASE 4 is safe on a real-accelerator cluster: its mock uses a fake product key (nvidia-e2emock) that never collides with a real GPU pool.

• CASES 23–32, 34 and 39 need a node address you must ask the user for. 23–32 and 34 read MIG_NODE_SSH=<user@host>; CASE 39 reads PPU_NODE_SSH=<user@host>, and only when it is not already running on the node. Each exits 2 (input required), going no further, when its address is unset — ask for it and pass it inline at run time, never hardcoded. All auto-skip (exit 0) when the hardware itself is missing.

• Vendor cases carry a vendor image, because every in-container reading is that vendor's own tool: CASE 35 a CANN-family image (a bare one exits 127 on an Ascend slice), CASE 38 a ROCm-family image, CASE 39 any image shipping ppu-smi. Each takes an E2E_*_IMAGE=<ref> override, and all are large — pre-pull on the node, or the first claim spends its whole wait pulling.

• CASE 23 owns the whole MIG mode transition — enabling MIG moves a card out of the logical family and into the partition one entirely — and is self-recovering: its trap restores the card's original mode on pass AND fail.

• CASES 24–32 and 34 are the hardware-partition family, sharing cases/_partition-lib.sh (node correlation, the SSH gate, profile/key discovery, pod plumbing, the record idiom). Its leading underscore marks it not a case; reuse it when adding another partition case rather than copying the discovery block.

  • Each partitions a card only if none is partitioned yet and restores exactly the card it toggled, so one up-front nvidia-smi -i 0 -mig 1 plus a Device Manager rollout restart lets most of them run back to back — but CASE 27 requires the opposite and skips if any card is already partitioned. That ordering — 27 first, then 24, then the rest — is what run-partition-block.sh encodes; drive the family with it.
  • Optional environment, all with defaults: MIG_NODE_NAME, MIG_NODE_SSH_OPTS, MIG_GPU_INDEX (0), MIG_SSH_TIMEOUT (90), IMAGE, MIG_MIXED_INDEXES / MIG_MIXED_ROUNDS (CASE 24), MIG_WRITE_IDLE_WINDOW (CASE 25), MIG_MAX_FILL (CASE 27), F11_IMAGE / F11_EXPECT (CASE 28), MIG_RECLAIM_BOUND (CASE 31), MIG_OOB_WINDOW (CASE 32).

• CASE 24 is the headline regression guard for the failure the two-family split exists to remove: a single token pool used to let the kubelet hand a partition request a token from a card that cannot be partitioned, and the Pod died with a terminal UnexpectedAdmissionError.

• CASE 14 and CASES 11/22/41 want opposite POOL shapes. CASE 14's over-budget assertion needs a pool of a single logically sliceable card — with a free sibling the third slice simply lands there instead of being held — so it auto-skips on a multi-card pool. CASES 11, 22 and 41 need two or more. The incompatibility is between pools, not nodes: one node carrying a single-card pool of one accelerator model beside a multi-card pool of another satisfies both. But CASE 14 takes the first sliceable pool it finds rather than seeking a single-card one, so even that node can leave it skipping — check which pool it picked before reading the skip as a hardware verdict. What genuinely cannot be arranged is hiding a card: the *_VISIBLE_DEVICES variables are what the allocators inject into a workload container, not a filter on what the device manager detects.

• CASE 40's accelerator sub-check cannot execute on its own Instance. The case creates a CPU Instance on the general pool, which holds no accelerator, so the accelerator-family label set — including mode — always skips, on any hardware. Reading it needs an accelerated Instance, and no case does it on this surface: CASE 14 asserts the mode spelling on the Instance metrics subresource, but the Prometheus accelerator-label path is uncovered — CASE 41 and CASE 37 read neither mode nor those labels.

• CASE 40's "exactly one target" only bites on a multi-vendor node. A node running one device manager satisfies it trivially; the rule it guards — that a node's Instances are published by one device manager, never by each — is only exercised where two manufacturers are present. Read a PASS there as "not double-counted on this node", not as "the election was tested".

• CASES 25, 31, 32 and 34 record observations, not thresholds — accepted consequences with stated containment, measured and printed as a copyable block for the design record; each still carries one hard assertion where a real regression would hide. CASE 28 prints the same block but is a guard, and only an explicit F11_EXPECT=observe demotes its verdict to a recording.

• CASES 49–51 carry a placeholder image, and it is plumbing rather than a mock. A ModelDeployment role names no image of its own; the operator synthesizes one from what the InstanceType has observed, and a CPU-only InstanceType has observed no accelerator — so on the cluster these three are meant to run on, every role would be refused by the CONTROLLER for a reason none of them is about (names no image and none could be synthesized). Each role therefore carries an explicit image, E2E_MD_IMAGE overrides it, and no assertion in any of the three reads anything a replica runs. Cases 47 and 48 take the same escape with a real Mooncake image, because theirs read the rendered Pod spec.

• Run CASES 49–51 in a namespace that carries the pool's entrance LocalQueue, which is what routes a group into its ClusterQueue — kubectl -n <NS> get localqueue before deciding. It is not created everywhere: on a fresh docker-desktop the operator materialized one in default, kube-public and kube-node-lease and none in gpustack-system, which is the namespace

Truncated - read the full file at https://github.com/gpustack/gpustack-operator/blob/8d7d8ca0945afb74133f0c7cde9a7123cd2f89de/.agents/skills/gpustack-operator-e2e/SKILL.md.