generate-contract-tests

Imported from Ryuking1228/ai-dev-codex-version (skills/generate-contract-tests/SKILL.md). Install upstream with npx skills add Ryuking1228/ai-dev-codex-version --skill generate-contract-tests. Copyright stays with the author.

Contract Test Generation

GraphQL surfaces use GraphQlTester and schema inspection rather than an OpenAPI validator. Select ExecutionGraphQlServiceTester, WebGraphQlTester, HttpGraphQlTester, WebSocketGraphQlTester, or RSocketGraphQlTester from the approved transport design.

Desired Outcome

Executable tests that hold the code to the contract. This is the difference between a contract that is documented and one that is enforced: acceptance criteria in a skill file are instructions to a model, and a model that drifts from them produces plausible code that reads fine. A failing assertion does not read fine.

Generate, per service:

• Contract validation tests — every request and response the suite exercises validated in-process against the OpenAPI document
• Problem Details conformance tests — every registered problem type reachable, correctly shaped
• Authorization tests — unauthenticated, wrong role, and (where an ownership predicate applies) another principal's or another tenant's object
• Idempotency tests — replay, key reuse with a different body, missing required key
• Indeterminate-commit tests — the UnknownTransactionStatusException contract, which is the one most often generated wrong
• ArchUnit rules — the layering the API layer specification declares

Stack

Per @rules/api-contract-fidelity.md §7, and read from reports/07_test-specs/contract-test-specs.md where generate-test-specs recorded the selection:

--stack overrides the recorded selection for one run. The default is not optional for a Spring service — in-process validation of every exercised request and response is what makes the rest of the suite meaningful. For a non-Spring or non-JVM service, use the framework's equivalent in-process validator and record which one.

Decision Criteria

• Assert against the specification, not against the code. A test whose expected values were read out of the implementation passes whatever the implementation does, which is the failure mode this skill exists to remove. Expected status codes, shapes, and problem types come from the OpenAPI document and the problem type registry.
• Use the contract's own examples as fixtures. They are in the specification for this reason (@rules/api-contract-fidelity.md §3). An operation with no example gets a shape test only, and the missing example is reported.
• Never weaken a test to make it pass. A generated test that fails is the intended outcome when the code is wrong; report the failure, do not relax the assertion or mark it disabled.
• Cover the negative space. A suite that only exercises declared success paths cannot detect an endpoint the contract does not declare — the undeclared-endpoint check comes from api-contract-map.json, and it is generated as an assertion, not left to review.

Prerequisites

Steps

1. Resolve the stack and the output root — the recorded selection, overridden by --stack; --out, else the service's src/test/ under the contract map's source_root.

2. Pin the contract into the test tree — copy the specification to src/test/resources/contract/openapi/<service>.yaml — and api-contract-map.json beside it, which the inventory assertion needs for out_of_scope_handlers — and load both from the classpath. The design copy lives under reports/, which is git-ignored: a committed test that reads it there passes for its author and fails the CI contract stage on a fresh checkout (@rules/api-contract-fidelity.md §7).

3. Generate the validation harness — the OpenAPI validator wired into the test slice, so every request and response in the suite is checked without per-test boilerplate. Resolve the validator coordinate per @rules/api-contract-fidelity.md §7: the module is -mockmvc (there is no -spring-mvc artifact), and its JDK baseline must be at or below the project's target release — the 3.x line is Java 21 and will not compile into a Java 17 service. For GraphQL, copy the approved SDL to test resources, configure the selected GraphQlTester, inspect the executable schema, and derive resolver inventory independently. Test success shape, nullability, validation, registered errors[].extensions.type, authorization, tenant isolation, N+1 query count, query limits, and subscription controls per @rules/graphql-security-checks.md.

4. Generate per-operation tests from contract-test-specs.md: request validation, response conformance, error conformance, authorization, idempotency, indeterminate commit.

5. Generate the inventory assertion — derived from the code, not read from the map. Walk the source tree for mapped routes and assert every one is declared by the specification or listed in the map's out_of_scope_handlers. An assertion that only re-reads api-contract-map.json passes whenever the map is wrong, which is exactly when it needs to fail: the first real run of this pipeline produced a map claiming nothing unmapped while ten routes — including an unauthenticated /api/admin/users — were reachable and undeclared (@rules/api-security-checks.md API9).

   The unit of comparison is (HTTP method, normalized path), and normalization renames path variables positionally. Both halves matter, and getting either wrong under-counts silently:

   • Rewrite each path variable to its position — /api/orders/{id} and /orders/{orderId} both become /api/orders/{1} and /orders/{1}. Rewriting to a fixed name instead makes genuinely different paths collide and vanish into a dedup.
   • Key on the method too. GET /orders and POST /orders are two operations; deduping on path alone counts them once.

   An independent reviewer of that same first run counted ten undeclared operations where the generated assertion reported six — the whole gap was these two mistakes. State the counting rule in the generated test so a reader can check it, and report the count alongside the list.

6. Generate ArchUnit rules when selected — layer direction as api-layer-spec.md states it for its layering_style (ddd: controller -> application service -> domain -> repository; clean: controllers depend on usecase/ boundaries only and never on an *Interactor, usecase/ never depends on api/ or infrastructure/, api.presenter and the input/output data records import no domain type, exactly one input boundary and one interactor per operationId, and each output boundary implemented exactly once, in api.presenter), the transaction-opening rule stated by package (only application.. / usecase.. calls the runner — never by class-name suffix, which the first shared collaborator breaks), no persistence or ScalarDB type in a controller signature, no domain type in a response DTO, and the testability rules of @rules/tdd-workflow.md §4: no Instant.now() / LocalDate.now() / UUID.randomUUID() call in domain/ or the application package (application/, or usecase/ under clean), no transaction type (resolve its FQN from the code — application.tx or usecase.tx) held by a domain class, and domain tests importing nothing from infrastructure/. When an ArchitectureTest already exists, merge into it; two rule classes drift.

7. Wire the build — a test task the quality gate can invoke by name (a separate contractTest source set is the right shape when the contract stack's JUnit platform version cannot share the test classpath — jqwik pinning JUnit 5 against Spring 7's JUnit 6 is the known case — with contractTest depending on the ArchUnit task so one name still covers both) (@rules/ai-code-quality-gate.md stage 3), pinning versions per @rules/dependency-versions.md.

8. Report — operations covered, operations with no example, assertions skipped and why, and the run's counts and exit code as they were. A red suite is not a failed run of this skill: the skill completes, the failures are listed with their cause (code defect fixed, contract contradiction, spec drift → Open Question), and nothing was weakened to get to green.

--dry-run reports the plan and coverage without writing tests.

Output

Write reports in the language configured in work/pipeline-progress.json (options.output_language). Test code and identifiers stay in English.

Acceptance Criteria

• Every operationId in the contract map has at least one generated test, or is listed as uncovered with the reason
• Every GraphQL resolver-bound field coordinate has a generated test or an explicit uncovered entry; SDL fields and source resolvers have no silent unmapped entries
• The generated suite compiles and runs, and the task the gate invokes actually matched tests — a filter matching nothing is a green task and an ungated build (@rules/ai-code-quality-gate.md)
• The specification the tests load is inside the test tree, not under reports/
• Every expected value in a generated test traces to the specification or the problem type registry — none read from the implementation
• Every registered problem type has a conformance assertion
• The UnknownTransactionStatusException contract is asserted for every operation that can raise it, including which of the two responses applies. For GraphQL execution, assert HTTP 200 and errors[].extensions.type; protected mutations require same-key retry fields, while unprotected mutations require reconcile-only fields. Assert forbidden retry fields, Retry-After, and raw transaction identifiers are absent in the opposite branch (@rules/api-error-standard.md §3.2)
• The inventory assertion exists and fails when api-contract-map.json reports anything unmapped
• The build exposes a named task the quality gate can invoke, and it was verified to run
• No generated test was disabled or weakened to make the suite green

Related Skills