eval

Imported from TulipFarm/tulipfarm (apps/eval/AGENTS.md). Install upstream with npx skills add TulipFarm/tulipfarm --skill eval. Copyright stays with the author.

apps/eval

Offline eval harness. Runs a versioned Corpus of Eval Cases through the real Context assembler and the real Tool loop, scores deterministic Expectations, and prints a Scorecard.

Manually triggered before a release. Never runs in the product; nothing here is reachable at runtime.

Every noun above is defined in metadata/terminologies.md → Offline eval, which is binding. Two words are easy to get wrong: an eval check is an Expectation, never an "assertion" (Memory owns that), and a Sweep is never an "eval run" (a Trial contains real Runs).

Read on / Skip

Read on if your task touches: eval Cases, Expectation kinds, the Corpus hash, Sweep execution, Scorecard output, or wiring a new model binding.

Skip if you are changing the loop, the assembler or a Tool — this app consumes them unchanged. Only its own Cases need updating when their observable behaviour moves.

Map

Rules

The reasoning behind each of these — and thirty more — is in README.md. Read it before changing how a Case is scored, run or compared. These are the ones that bite hardest:

• The Eval Soul is loaded, never constructed, and its hash is folded into corpusHash.
• The assembler stays in the path. runTrial calls assembleSystemPrompt itself; a runner feeding hand-written prompts to the loop would never catch a Context-assembly regression.
• Expectations are data, never functions — that is what makes the Corpus hashable.
• A vendor failure is not a verdict. model_* loop failures count as errored, never failed.
• corpusHash is the unit of comparability. A delta across two Corpora, or two models, is refused rather than computed.
• Nothing becomes the Baseline on its own. Promotion is --promote, from a clean tree, from a complete and unregressed Sweep only.
• A content Expectation must be grounded in what the model was given, or declare ungrounded. A File's declared content counts as given; its id and filename do not.
• attachments is what this Turn sent; readable is what file_read can go and get. A File in both would be in the prompt from the first step, so a re-read Case asserting prompt_attaches would pass with the whole mechanism removed. Declaring it in neither is refused for the same reason.
• Name a shipped Tool with platformTools; only hand-declare a Tool an operator would author. A copy measures the model against a description no deployment sends, and cannot assert the properties that live in the declaration. The resolved declaration is in corpusHash, so rewording one retires every Baseline. An unresolvable name fails the load.
• Stateful L3 Tools run for real, routed by name in routeTools. MCP setup/get and reviewed Tools use the production domain and live account authority; Soul/Resource authoring, Record mutations, and file_create use the real writer, loader, Resource service, and File store. Never script their successful result. MCP fixture account choices and consent are host state, not model arguments; the Corpus rejects scripted MCP results in every Turn.
• MCP revokeGrantBeforeApproval removes a durable grant while the real approval wait is pending. MCP reads must use mcpIntegrationsFromBundle, never authored-loader state.
• mcp_provider_call_count counts actual external tools/call requests, not dispatcher attempts or model claims. Missing observations fail even when the expected count is zero.
• run_event_emitted.count pins an exact positive event count; repeated approvals must emit distinct durable events, not merely complete two provider calls.
• nativeRoutine starts from a persisted, claimed native inbox event, not an HTTP webhook. native_admission_equals observes admission rows and real account resolution, never model text. Its lease fault changes the inbox fence after Run insertion to prove the shared transaction rolls back.
• A saved generated File's audience comes from agentRoles, never the soul's roles:, which is advisory and writes no role_assignments. Pair generated_file_readable_by with a generated_file_not_readable_by for a Role the Agent lacks, or the Case passes just as well against an audience that shares every File with everyone.
• Every Case carries a script, so the whole Corpus runs free in ordinary CI.
• L3 preserves model streaming. run_event_text_omits reads concatenated durable participant text, not the final Message or operator evidence; missing observations fail.
• L3 claims each Run before execution, so State writes use the real lease-generation fence.
• Both tiers collect guard refusals. L3 reads durable guardrail.decision events. Use stage-specific guardrail_blocked so safe model refusals can be reported as unexercised.
• Two models are a control, not a contest, and they never run at once.
• A Judge failure errors the Trial; it never scores low.
• A fault is L3-only and breaks a dependency before or during a Turn. The checkpoint fault resumes a real approval checkpoint before failing; the output-limit fault uses the production model completion guard, not a synthetic error.
• Persisted Message Expectations read eval_messages, never Run events or recomputed history. Failure Cases must create their evidence through scripted model and Tool rounds before the fault.
• Provider File Expectations read splitPrompt binary parts against immutable Case Files.
• provider_prompt_contains reads the first splitPrompt projection, excluding assistant Messages. DOCX fixtures require a single L3 Chat Turn; text fixtures ground their final paragraph in content.
• model_not_called requires an observed zero invocation count, not missing usage. DOCX refusal variants contain real defective archives, never scripted parser verdicts or invented content.
• PDF Cases compare original binary bytes and all page estimates. A readable PDF's replaceAfterRead changes fixture storage after the real Tool succeeds; never script that Tool's result.
• Checkpoint replay is one L2 fault seam. It crashes after the first Tool result and retries the same input; do not turn it into a general workflow fixture.
• This workspace is CJS-by-default — no import.meta.

Commands