science-workflow-experiment

Imported from ccy5123/sci-adk (src/sci_adk/templates/research-workspace/.claude/skills/science-workflow-experiment/SKILL.md). Install upstream with npx skills add ccy5123/sci-adk --skill science-workflow-experiment. Copyright stays with the author (Apache-2.0).

science-workflow-experiment — Record + Derive (Stage 3)

The experiment-stage procedure: execute the frozen MethodPlan, append an honest append-only Evidence record, then derive revisable Claims from that record via the frozen DecisionRule. For the discipline (record vs belief, invariants, verbs, halts) load Skill("science-foundation-rigor"); this skill is the HOW.

Quick Reference (30 seconds)

• Two sub-stages, sequential (3b reads 3a's record):
  1. expert-experimentalist runs the MethodPlan → appends Evidence (null/negative included), filling bears_on[] per the Spec's pre-registered mapping.
  2. expert-statistician reads the Evidence → applies the frozen DecisionRule → derives Claims; authors verdicts for non-numeric checkpoints; loops sci-adk resolve.
• The record is honest or it is worthless: append nulls and negatives exactly; suppressing a null is the most common way a research record lies.
• No post-hoc bearing: bears_on[] was fixed at pre-registration; transcribe it, never invent it (anti-HARKing).
• Follow emergent questions: when a genuine NEW question surfaces mid-experiment ("has anyone already measured / refuted this?"), search it and RECORD the decision — sci-adk inquiry <run_dir> --question "<q>" --searched <dois...> (→ LITERATURE + INQUIRY_DECISION) or --skip --reason "..." (a recorded null). Agent-judgment, not a periodic prompt; the emergent discovery decision belongs in the record like every other.

Implementation Guide (5 minutes)

Sub-stage 3a — Collect Evidence (expert-experimentalist)

Run the FROZEN MethodPlan — do not redesign the method to chase a result (that would be HARKing and breaks the pre-registration anchor). The experiment tools and code arrive via the Spec's MethodPlan, science-domain-* skills, or user-supplied artifacts; the kernel carries no domain code.

• sci-adk execute — runs the frozen experiment (docker / python etc.), capturing provenance.
• sci-adk append-evidence — appends ONE typed, immutable Evidence entry per result, including null and negative results. Each entry carries bears_on[] (which hypotheses it speaks to, and the direction: supports / contradicts / inconclusive) STRICTLY per the Spec MethodPlan's pre-registered mapping.
• provenance.code_ref names the code that produced the result. The reproduction bundle ships the source files it names (under paper/code/) and checks each against its recorded hash; it does not re-run them, because the record holds no command-line arguments or input files. Write the script's path (workspace- or run-relative), followed by sha256=<hex> of that file as it ran (sha256sum <file>); or, for code that is not a workspace file, the commit/ref. A script that built this result's inputs is named the same way, <path> sha256=<hex>, after the first one, and ships too. Only source files ship (program files such as .py, .R, .jl, .m, .sh, .do, .ipynb, recognised by extension); a data file named with its hash (data/x.csv sha256=<hex>) is listed with that hash in paper/reproduce.py and never shipped. Descriptions and notes such as "no git commit" go in the finding. A code_ref that does not resolve to a file is kept as a pointer, and a file whose sha256 no longer matches is not shipped as the recorded code.
• Name every output table a paper may ship by its checksum, in provenance.data_ref. For every output table the analysis writes that a paper may offer its readers (a filtered data set, a results table), write <path> sha256=<hex> (sha256sum <file> on the file as written; workspace- or run-relative path) in the data_ref of the Evidence item that produced or used it, one pair per file, separated by ; . At publish time a table ships with the paper as a checked file only when an Evidence item holds its checksum; a table no item names can only be declared derived, which carries no integrity claim. Name a third-party input file the same way, and each copy the study made in order to read it (decrypted, decompressed, converted) with its own checksum.
• Record the numbers a paper will state as values, not only prose. Every count, constant and quoted value a paper may state goes into the finding JSON as a named number: {"summary": "<what was done>", "<name>": <value>, ...} (counts after each step, sample sizes, coefficients and their standard errors, fractions). The paper session binds each number it states to a recorded field (runs/<id>/numbers.json, checked by verify); a number that exists only in prose has no field, and the paper comes back here for it. A value quoted from another study is recorded the same way, when it is read, in an observation whose finding names the study each value comes from.

Anti-HARKing rule: a result that bears on a hypothesis the Spec did NOT pre-map for it is a finding to report to the orchestrator (possibly an amendment), not a bearing you add yourself. "The script ran without error" is not Evidence of a hypothesis — the typed entry (result + provenance + bearing) is. Never hand-edit runs/<id>/ Evidence files; the verb is the only way to write the record.

Sub-stage 3b — Derive Claims (expert-statistician)

Read the Evidence record (including the recorded bears_on[]) — Evidence is immutable (E1); never modify it. If it looks wrong or incomplete, that is a finding to report, not an edit, and not a reason to estimate.

• sci-adk derive-claim — applies the per-hypothesis frozen DecisionRule to the Evidence and records the Claim status (SUPPORTED / CONTESTED / REFUTED / pending) + confidence + derivation basis. The ENGINE renders the status; you do not decide by judgment that a hypothesis is supported. "The result looks convincing" is not a verdict.
• Make CONTESTED / REFUTED outcomes explicit — they are results, not failures. Claims are non-monotone: a status can move as new Evidence arrives.

Science guards in the experiment stage (G1/G2/G3)

prereg already SET the G3 target and DECLARED the G1/G2/G4/G5 fields at freeze (see science-workflow-prereg); the experiment stage REALIZES the target and HONORS the verdict gate. For what each guard MEANS and WHY, load Skill("science-foundation-rigor") §"The science guards" — here is only the procedure.

3a — produce the G3 negative control (expert-experimentalist). Realize the falsifiability target the frozen MethodPlan pre-registered: run a deliberately broken variant of the method, then sci-adk append-evidence it as a NEGATIVE_CONTROL Evidence item — outcome == not_supported, covering the declared discriminating_cases, with REAL execution provenance (it was actually RUN, not asserted), and bears_on=[] so it NEVER enters the DecisionEngine (a mutant's refutation must not contaminate the hypothesis verdict). It lives in the append-only log (digest-covered; verify re-derives). Without it, a binding SUPPORTS cannot pass G3 under strict_science.

3b — honor the strict verdict gate (expert-statistician). sci-adk derive-claim (and sci-adk run) default to strict_science: a binding formal + threshold SUPPORTS is HARD-halted if G1 (a known result framed as finding), G2 (no discriminating_cases), or G3 (no qualifying NEGATIVE_CONTROL) fails. A halt is the referee working — resolve it by recording the missing thing (add the negative control via append-evidence, or sci-adk amend-spec to declare discriminating_cases / reclassify epistemic_kind / assert novelty) then re-derive. Never route around it with --no-strict-science on a real research run — that opt-out exists for low-level / build-harness callers, not to silence a genuinely weak claim. sci-adk verify --strict-science re-checks G1/G2/G3 read-only: a deleted negative control or stripped discriminating_cases makes a recorded strict SUPPORTED DIVERGE (tamper-evidence).

Proof / qualitative checkpoints

When the engine raises a checkpoint the DecisionRule cannot resolve numerically (a proof obligation, a qualitative judgment the Spec delegates to a human-authored verdict), author the verdict as verdicts/<hyp>.json: a verdict tied to the recorded Evidence and the frozen rule, with an explicit basis. The verdict then flows back through derive-claim / verify like any other recorded belief.

The resolve loop

The engine surfaces checkpoints and halts. Run sci-adk resolve to clear them, then re-derive / re-check. Loop until no open checkpoints remain. As a self-check before returning, run sci-adk verify (read-only) and confirm every Claim reproduces from the record. sci-adk status <run> is a cheap snapshot of what is still open.

Frozen-Spec boundary

Both workers receive a [FROZEN SPEC REFERENCE] block (spec_id, spec_digest). The verbs check the digest against the on-disk Spec; a mismatch raises SpecDigestMismatch and the call fails. Apply the MethodPlan and DecisionRule EXACTLY as frozen — if the rule or method itself must change, that is a manager-prereg amendment, not a worker's call.

Advanced (10+ minutes)

Evidence schema + invariants E1–E4 and Claim schema + invariants C1–C6: design/abstractions.md. The validity halt (an empirical Claim cannot reach SUPPORTED on synthetic / digitized Evidence alone — referent typing) is enforced by sci-adk verify and pre-checked by the evaluator-validity guard. The digitized Evidence kind (proposed → verified gate, extractor ≠ verifier, never auto-promote to measured) is part of the same record-fidelity family.

Works Well With

• science-foundation-rigor — the Evidence/Claim discipline this builds on.
• science-workflow-prereg — produced the frozen Spec + DecisionRule + bears_on[] map.
• science-workflow-publish — narrates the derived Claims (nothing over-stated).
• expert-experimentalist — the worker that runs the MethodPlan and appends Evidence.
• expert-statistician — the worker that derives Claims and authors verdicts.