gpd-compare-experiment

Imported from MichaelsEngineering/sentinel-multiscale-field-dynamics (.agents/skills/gpd-compare-experiment/SKILL.md). Install upstream with npx skills add MichaelsEngineering/sentinel-multiscale-field-dynamics --skill gpd-compare-experiment. Copyright stays with the author.

<codex_runtime_notes> Codex shell compatibility:

• When shell steps call the GPD CLI, use /home/qol/.gpd/venv/bin/python -m gpd.runtime_cli --runtime codex --config-dir ./.codex --install-scope local instead of the ambient gpd on PATH.
• If you intentionally need the repo environment, keep the runtime pin: GPD_ACTIVE_RUNTIME=codex uv run gpd .... </codex_runtime_notes>

<codex_questioning>

• Ask each user-facing question exactly once.
• Present options once.
• Do not restate the prompt or add meta narration. </codex_questioning>

Why a dedicated command: Theory-experiment comparison is not just plotting two curves on the same axes. It requires rigorous treatment of units, uncertainties, systematic effects, and statistical significance. A "good agreement" by eye may be a 3-sigma discrepancy when uncertainties are properly accounted for. Conversely, a visually poor fit may be statistically acceptable when systematic uncertainties are included.

The principle: Agreement between theory and experiment must be quantified. "Looks about right" is not physics. The comparison must state: (1) what was predicted, (2) what was measured, (3) what the uncertainties are on both sides, (4) whether the agreement is statistically significant, and (5) if not, what the discrepancy tells us.

For contract-backed work, the comparison must also state which decisive output or contract target is being tested and emit an explicit verdict ledger keyed by subject_id / reference_id, not just a prose comparison.

Interpretation:

• If a prediction name: compare that specific theoretical prediction with data
• If a dataset path: compare theoretical model against that dataset
• If a phase number: compare all predictions from that phase with available data
• If empty: prompt for comparison target

Load theoretical predictions:

cat .gpd/research-map/ARCHITECTURE.md 2>/dev/null | grep -A 20 "Predictions"
find artifacts/ results/ data/ figures/ simulations/ paper/ -maxdepth 4 \
  \( -name "*.json" -o -name "*.csv" -o -name "*.dat" -o -name "*.h5" \) 2>/dev/null | \
  grep -i "result\|predict\|spectrum\|observable" | head -20

Treat .gpd/** as internal provenance only. Discover predictions and reusable comparison inputs from stable workspace directories such as artifacts/, results/, data/, figures/, simulations/, or paper/.

<execution_context>

Called from $gpd-compare-experiment command. Produces COMPARISON.md with quantified agreement metrics.

Agreement between theory and experiment must be quantified. "Looks about right" is not physics. The comparison must state: (1) what decisive output or contract target was predicted, (2) what was measured, (3) what the uncertainties are on both sides, (4) whether the agreement is statistically significant, and (5) if not, what the discrepancy tells us.

<required_reading> Read these files using the read_file tool:

• ./.codex/get-physics-done/templates/paper/experimental-comparison.md -- Template for systematic theory-experiment comparison (data source metadata, unit conversion checklist, pull calculation, discrepancy classification, root cause hierarchy) </required_reading>

0. Load Project Context

Load project state and conventions to ensure correct unit systems and sign conventions:

• Run:

INIT=$(/home/qol/.gpd/venv/bin/python -m gpd.runtime_cli --runtime codex --config-dir ./.codex --install-scope local init progress --include state)
if [ $? -ne 0 ]; then
  echo "ERROR: gpd initialization failed: $INIT"
  # STOP — display the error to the user and do not proceed.
fi

• Parse JSON for: commit_docs, state_exists, project_exists, current_phase
• If state_exists is true: Read .gpd/state.json to extract convention_lock for unit system, metric signature, and Fourier conventions. Extract active approximations and their validity ranges from state. Load intermediate_results from state for any previously computed quantities.
• If state_exists is false (standalone usage): Proceed with explicit convention declarations required from user via ask_user (unit system, sign conventions, normalization)

Convention context is critical for theory-experiment comparison: unit mismatches and convention mismatches are the two most common sources of discrepancy.

Convention verification (if project exists):

CONV_CHECK=$(/home/qol/.gpd/venv/bin/python -m gpd.runtime_cli --runtime codex --config-dir ./.codex --install-scope local --raw convention check 2>/dev/null)
if [ $? -ne 0 ]; then
  echo "WARNING: Convention verification failed — unit mismatches between theory and experiment are the #1 source of false discrepancies"
  echo "$CONV_CHECK"
fi

1. Identify What to Compare

If the project is contract-backed, first resolve the comparison target against the approved contract:

• subject_id
• subject_kind (claim, deliverable, acceptance_test, or artifact)
• subject_role (decisive, supporting, supplemental)
• reference_id for the benchmark / prior-work / data anchor
• the pass condition or tolerance that makes this comparison decisive

Do not write a generic comparison report without this mapping when a decisive comparison target exists.

1a. Theoretical predictions

For each prediction, establish:

• Quantity predicted: Name, symbol, definition
• Predicted value or functional form: Expression or numerical table
• Theoretical uncertainties: From truncation, approximation, numerical precision
• Conventions: Units, normalization, sign conventions
• Regime of validity: Where the prediction is expected to hold

1b. Experimental data

For each measurement, establish:

• Quantity measured: Name, definition, what detector/apparatus
• Measured values: Central values with error bars
• Statistical uncertainties: From measurement statistics
• Systematic uncertainties: From calibration, background, modeling
• Conventions: Units, normalization, binning, acceptance corrections
• Conditions: Temperature, pressure, energy, other experimental parameters

Ask the user once using a single compact prompt block if data source is ambiguous:

1. Data source -- Published paper | Our own measurements | Database (PDG, NIST, etc.) | Simulation (lattice, MD, etc.)
2. Data format -- CSV/table | Digitized from plot | API/database query | File path
3. Uncertainty treatment -- Statistical only | Statistical + systematic (separate) | Combined | Correlated uncertainties

2. Unit Conversion and Convention Matching

This step catches the most common theory-experiment comparison errors.

2a. Unit alignment

Ensure theory and experiment use the same units:

# Common unit conversion pitfalls in physics:
# - eV vs Kelvin: 1 eV = 11604.5 K
# - eV vs cm^{-1}: 1 eV = 8065.54 cm^{-1}
# - Gaussian vs SI electromagnetism: factor of 4*pi*epsilon_0
# - Natural units to SI: restore factors of hbar, c, k_B
# - Energy vs frequency: E = hbar * omega (not h * nu !)
# - Cross-section: barn (1e-24 cm^2) vs fm^2 (1e-26 cm^2) vs GeV^{-2}
# - Decay width vs lifetime: Gamma = hbar / tau

Document every conversion applied.

2b. Convention matching

Check that theory and experiment use the same conventions:

2c. Acceptance and efficiency corrections

Experimental data often includes detector acceptance and efficiency:

• Is the data already corrected? (Unfolded to particle level)
• If not: apply theory-side smearing/cuts to match detector level
• Document what corrections are applied on which side

3. Perform the Comparison

3a. Point-by-point comparison

import numpy as np

# For each data point:
for i in range(len(data)):
    x_exp = data[i]['x']
    y_exp = data[i]['y']
    dy_exp = data[i]['uncertainty']  # total: stat + syst combined
    y_theory = theory_prediction(x_exp)
    dy_theory = theory_uncertainty(x_exp)

    # Pull: (theory - experiment) / combined_uncertainty
    dy_combined = np.sqrt(dy_exp**2 + dy_theory**2)
    pull = (y_theory - y_exp) / dy_combined

    print(f"x={x_exp:.4f}  exp={y_exp:.6f}+/-{dy_exp:.6f}  "
          f"theory={y_theory:.6f}+/-{dy_theory:.6f}  pull={pull:.2f}")

3b. Global fit quality

# Chi-squared test
chi2 = 0
for i in range(len(data)):
    dy_combined = np.sqrt(data[i]['uncertainty']**2 + theory_uncertainty(data[i]['x'])**2)
    chi2 += ((theory_prediction(data[i]['x']) - data[i]['y']) / dy_combined)**2

ndof = len(data) - n_free_params
chi2_reduced = chi2 / ndof
p_value = 1 - scipy.stats.chi2.cdf(chi2, ndof)

print(f"chi2/ndof = {chi2:.1f}/{ndof} = {chi2_reduced:.2f}")
print(f"p-value = {p_value:.4f}")

3c. Statistical interpretation

3d. Residual analysis

# Check for systematic patterns in residuals:
# - Trending residuals suggest missing physics (wrong functional form)
# - Oscillating residuals suggest discretization artifacts
# - Constant offset suggests normalization error
# - Growing residuals suggest wrong scaling / missing terms

4. Discrepancy Analysis

If the comparison shows significant disagreement:

4a. Classify the discrepancy

4b. Root cause investigation

1. Check units again -- Most theory-experiment discrepancies are unit errors
2. Check conventions -- Second most common: different normalization or phase convention
3. Check data processing -- Binning, acceptance, background subtraction
4. Check approximations -- Is the theoretical prediction valid in this regime?
5. Check for missing physics -- Higher-order corrections, finite-size effects, interactions neglected

4c. Quantify the tension

# Number of sigma tension:
tension_sigma = abs(theory_central - exp_central) / np.sqrt(theory_unc**2 + exp_unc**2)

# Interpretation:
# < 1 sigma: consistent
# 1-2 sigma: mild tension, not significant
# 2-3 sigma: interesting tension, warrants investigation
# 3-5 sigma: significant discrepancy
# > 5 sigma: discovery-level discrepancy

5. Generate Comparison Report

Write COMPARISON.md:

---
date: { YYYY-MM-DD }
theory_source: { derivation/computation path }
data_source: { experiment/measurement reference }
overall_agreement: good | tension | discrepancy
chi2_ndof: { value }
p_value: { value }
max_tension_sigma: { value }
comparison_verdicts:
  - subject_id: claim-id
    subject_kind: claim|deliverable|acceptance_test|artifact
    subject_role: decisive|supporting|supplemental
    reference_id: ref-id
    comparison_kind: benchmark|prior_work|experiment|cross_method|baseline
    metric: chi2_ndof | relative_error | pull
    threshold: "<= 2 sigma"
    verdict: pass | tension | fail | inconclusive
    recommended_action: { what to do next }
---

# Theory-Experiment Comparison

## Quantities Compared

| Quantity | Theory          | Experiment      | Units   |
| -------- | --------------- | --------------- | ------- |
| {name}   | {value +/- unc} | {value +/- unc} | {units} |

## Unit Conversions Applied

{Document all conversions}

## Convention Matching

{Document all convention alignments}

## Point-by-Point Comparison

| x   | y_theory | dy_theory | y_exp   | dy_exp | Pull   | Status       |
| --- | -------- | --------- | ------- | ------ | ------ | ------------ |
| {x} | {value}  | {unc}     | {value} | {unc}  | {pull} | {OK/TENSION} |

## Global Fit Quality

- chi2 / ndof = {value}
- p-value = {value}
- Assessment: {good agreement / tension / discrepancy}

## Residual Analysis

{Systematic patterns, trends, outliers}

## Discrepancy Analysis (if applicable)

- **Type:** {classification}
- **Magnitude:** {N sigma}
- **Likely cause:** {root cause analysis}
- **Resolution path:** {what to investigate}

## Figures

- `{path}`: Theory vs experiment overlay
- `{path}`: Pull distribution
- `{path}`: Residuals vs parameter

## Conclusions

{Summary of agreement/disagreement and its significance}

The comparison_verdicts block is the authoritative machine-readable ledger. The tables and prose explain it; they do not replace it.

Save to:

• Phase-scoped (if running within a phase context and phase_dir is set): ${phase_dir}/COMPARISON-{slug}.md
• Standalone (no phase context):

mkdir -p .gpd/analysis

Write to .gpd/analysis/comparison-{slug}.md

6. Generate Comparison Figures

Create scripts for standard comparison plots:

1. Theory vs experiment overlay -- data points with error bars, theory curve with uncertainty band
2. Pull distribution -- histogram of (theory-exp)/sigma, should be standard normal
3. Residual plot -- (theory-exp) vs parameter, looking for systematic trends
4. Ratio plot -- theory/experiment vs parameter (for normalizations)

Write final comparison figures, tables, and helper scripts to stable workspace roots. Default to artifacts/comparisons/{slug}/ unless the project already has a clearer durable home such as figures/, data/, or scripts/. Do not place final comparison figures, tables, or scripts under .gpd/.

7. Present Results and Route

If good agreement:

## Theory-Experiment Comparison: Good Agreement

chi2/ndof = {value}, p-value = {value}
Maximum tension: {N} sigma at {parameter value}

Theory predictions are consistent with experimental data.
Ready for: `$gpd-write-paper` (Results section)

If discrepancy:

## Theory-Experiment Comparison: {N}-sigma Discrepancy Found

chi2/ndof = {value}, p-value = {value}

### Discrepancy Details
{Classification and magnitude}

### Suggested Investigation
- `$gpd-debug` -- investigate the discrepancy
- `$gpd-limiting-cases` -- check if the prediction is valid in this regime
- Check experimental systematic uncertainties
- Compute next-order theoretical corrections

8. Commit Comparison Report

PRE_CHECK=$(/home/qol/.gpd/venv/bin/python -m gpd.runtime_cli --runtime codex --config-dir ./.codex --install-scope local pre-commit-check --files "${COMPARISON_OUTPUT_PATH}" 2>&1) || true
echo "$PRE_CHECK"

/home/qol/.gpd/venv/bin/python -m gpd.runtime_cli --runtime codex --config-dir ./.codex --install-scope local commit \
  "docs: theory-experiment comparison for {slug}" \
  --files "${COMPARISON_OUTPUT_PATH}"

Where ${COMPARISON_OUTPUT_PATH} is the path chosen in step 5 (phase-scoped or standalone).

<success_criteria>

• Theoretical predictions identified with uncertainties
• Experimental data loaded with uncertainties (stat + syst)
• Unit conversions documented and applied
• Convention matching verified
• Point-by-point comparison performed with pulls
• Global chi-squared computed with p-value
• Residual analysis performed for systematic patterns
• Discrepancies classified and quantified (in sigma)
• Root cause analysis for any significant discrepancy
• Comparison report generated
• Contract-backed comparisons include comparison_verdicts keyed by subject ID / reference ID
• Comparison figures generated (or scripts provided)
• Comparison report committed (if commit_docs enabled)
• Results routed appropriately (paper writing or debugging) </success_criteria>

</execution_context>

Pre-flight check:

CONTEXT=$(/home/qol/.gpd/venv/bin/python -m gpd.runtime_cli --runtime codex --config-dir ./.codex --install-scope local --raw validate command-context compare-experiment "$ARGUMENTS")
if [ $? -ne 0 ]; then
  echo "$CONTEXT"
  exit 1
fi

Follow the compare-experiment workflow: @./.codex/get-physics-done/workflows/compare-experiment.md

<success_criteria>

• Theoretical predictions identified with uncertainties
• Experimental data loaded with uncertainties (stat + syst)
• Unit conversions documented and applied
• Convention matching verified
• Point-by-point comparison performed with pulls
• Global chi-squared computed with p-value
• Residual analysis performed for systematic patterns
• Discrepancies classified and quantified (in sigma)
• Root cause analysis for any significant discrepancy
• Comparison report generated
• Comparison figures generated (or scripts provided)
• Results routed appropriately (paper writing or debugging) </success_criteria>