rietx

Imported from yue-here/rietx (docs/skill/rietx/SKILL.md). Install upstream with npx skills add yue-here/rietx --skill rietx. Copyright stays with the author (MIT).

Refining powder diffraction data with rietx

Audience: an agent driving rietx on real data. Not a tutorial and not an API reference. A protocol: what to do, in what order, what to check before believing a number, and where the package will tell you your answer is wrong even though it looks right.

Rwp is not the objective function of your job. A refinement can converge, report an excellent Rwp, and return displacement parameters biased by 100 %, phase fractions wrong by 5 wt % and a cell that is right for the wrong reason. Every rule below exists because one of those happened and was measured. Sections 1-4 are ordinary Rietveld discipline that would apply in any code; sections 5-10 are specific to running it with no human at the plot.

Load these when the task calls for them

This file is the judgement core: what holds for every fit. Lookup tables and the rules one task shape needs live beside it, one file each. The user manual holds the object model this protocol drives, one page per topic under https://rietx.org/using/.

A name in front of you is its own index. A Diagnostic code, a field or a verb has its row in one of these files, and grep -rn NAME references/ finds it wherever it lives. A code's row says what it means, what to do and what you must not do.

1. Before you refine: what the method can and cannot do

Rietveld refinement fits a structural model you already believe to a whole powder pattern. It is a local, gradient-based optimisation of a strongly non-convex, strongly correlated problem. It is not structure solution, not phase identification, and not a search.

Preconditions, all of which must hold before fit() is meaningful:

Never subtract a background before refining. Subtraction invalidates the counting-statistics weights and can make intensities negative. Hold an estimated background additively (BackgroundFixedPlusChebyshev) or co-refine it under a smoothness penalty (BackgroundPSpline); rx.auto_background(data, two_theta_limits=…) sizes one to the range you fit. A measured blank (empty can, blank capillary, matrix-only scan) is BackgroundFixedPlusChebyshev.from_pattern(blank), its scale freed against a low-order polynomial and judged by its own esd (§8.29).

2. The turn-on order, and why it is not negotiable

Free parameters in groups, cumulatively, in a stable order (McCusker, Von Dreele, Cox, Louër & Scardi, 1999, J. Appl. Cryst. 32, 36), each group running to convergence before the next is freed. The reason is not tradition: the correlations between groups are severe, and a simultaneous release from a poor starting point walks into a local minimum that a staged release avoids. Toby (2024, J. Appl. Cryst. 57, 175, the "recipe problem"): once parameters have refined to unphysical values, adding more parameters no longer lets the fit recover.

The plans encode this. Use them; do not hand-roll a free set unless you have a reason you can state. The staged discipline is what is not negotiable; the preset sequence is a default, because the right next group depends on the data and the current values — and ref.suggest(data) answers that at the current state, one analytic-Jacobian evaluation ranking every held parameter by predicted Δχ², with no fit and no mutation.

plan="mccusker_default"      # scale+bkg → zero → cell → W → U,V,X,Y      (profile only)
plan="mccusker_structural"   # …then coordinates → displacement → PO → extinction → roughness
plan="lab_bragg_brentano"    # …with sample displacement, Kα2 ratio, FCJ axial
plan="lab_calibrate"         # instrument calibration on a standard, certified cell HELD
plan="lab_sample_refine"     # sample against a frozen calibrated instrument
plan="profile_only"          # Le Bail
plan="pawley_default"        # Pawley

rx.PLAN_INFO carries a title, description, modes and when-to-use for each.

Three ordering rules. None is in the guidelines; each is this package's own measured finding.

1. Widths last among the profile terms, W before U,V,X,Y. W is the constant term; freeing the tanθ and 1/cosθ terms first lets them absorb a constant offset and then fight it.
2. Intensity-scaling corrections go last, after the structure has settled. Preferred orientation, extinction and surface roughness rescale intensities in a Q-dependent way, and so do the scale, the occupancies and the displacement parameters. Freeing a correction early lets it eat structure that belongs to the structure.
3. Anisotropic strain is freed inside the sample-broadening stage, not after. A Stephens block locks lor_strain — its isotropic direction is that column — so deferring it leaves the isotropic width unrefined right up to the moment fifteen correlated coefficients turn on at once.

Structure-free first when you can. Le Bail (mode="lebail") extracts intensities from the data instead of computing them, so it converges the cell, zero and profile with no structural assumption. Do that first, then switch to Rietveld with the converged cell and profile: it is the single most reliable way to avoid a structural minimum that is really a profile error. Its cell is the weaker half of that: where reflections are dense, arbitrary intensities can index one pattern more than one way, so check a Le Bail cell against even a rough structural model before quoting it (Peterson 2005). The variable is reflections per FWHM, so on a resolved pattern skip the check. Two rules about it the API does not tell you, both measured on third-party lab data.

4. Iterate the whole plan to a fixed point; one fit() is not enough, and keep the best pass, not the last. The extracted per-hkl intensities are frozen inside each least-squares run, so intensities and profile converge only by alternating, and the alternation is not a descent on one objective, so a later pass can come back worse.
5. Seed the background before the first pass, always. auto_background starts every coefficient at 0.0, so the first lebail_update runs before the background has ever been fitted, is handed the whole pedestal, and gives it to the Bragg reflections. Seed the constant term from a low percentile of y_obs.

Both measurements, and multi-phase Le Bail's one surviving caveat: references/judging.md.

3. The degeneracies. Memorise these

Almost every wrong-but-good-looking Rietveld result is one of these. They are not bugs; they are the geometry of the problem.

6. Do not free the second member of a group without checking the first is pinned by something outside the fit. lab_calibrate exists for this: refining a certified standard with its cell held fixed is what decorrelates zero from displacement from cell, because the cell is supplied rather than fitted.
7. A correlation of 0.98+ means you refined one parameter and reported two. HIGH_CORRELATION fires for you. The right response is almost never "widen the bounds"; it is to fix one, or to extend the data range until the signatures separate.
8. Where chemistry says two quantities are one quantity, constrain them rather than refining both. ref.tie_equal([paths]) makes an equality group, ref.tie(path, source, scale=, offset=) the general affine form (occ₁ = 1 − occ₀ on a mixed site is scale=-1, offset=1), ref.untie releases them. Check the premise first, in the free refinement and never with Rwp: tie only values that lie within their own esds of each other. Why, and what a tie bought on fluorapatite: references/judging.md.

4. Judging a fit — and what Rwp is actually for

Rwp compares your model to the data you have, weighted by counting statistics. It is dominated by the strongest peaks and by the background level: a useful relative number between two fits of the same data over the same channels, and a nearly useless absolute one. Measured, 18 refinements of one identical PbSO₄ dataset returned Rwp 8.2–20.0 % (Hill, 1992, J. Appl. Cryst. 25, 589).

Judge a fit in this order. print(result) renders steps 9 and 17 (per-stage status, every diagnostic, provenance, agreement indices last), and ref.summary(deliverable=…) adds the rows that need the compiled model. The measured evidence behind each rule is references/judging.md.

9. Status and guards outrank every statistic. result.status, then result.diagnostics. statistics.max_shift_over_esd is the measured quantity behind "converged" (McCusker §7 converges at ≤ 0.1). A converged solve satisfies it a fortiori, so read it where a stage stopped on STAGE_MAX_ITER: its magnitude says how far the solve was still moving, in esd units.

10. Read the shape of the difference curve region by region, not its size. report.regions carries per-region local Rwp and χ² share, and cumulative_chi2_breakpoints locates where the model starts failing.

11. Read the unmatched peaks. report.unmatched with kind="unmatched_obs" is an impurity or a missing phase; "unmatched_calc" is a phase you modelled that is not there, or an absence error. Read unmatched_calc in Rietveld mode only. Le Bail and Pawley extraction takes away most of the residual it looks for, so there it fires on noise near a tick. result.tick_hkl pairs with result.ticks by index, so an unmatched peak is nameable — which reflection of which phase — and not only locatable.

12. Ask whether the refined values are physically possible — negative Biso, occupancies above 1, a cell that moved 0.5 %, an ADP tensor that is not an ellipsoid — and ask it of the structure too. result.geometry is a GeometryTable of bonds, contacts and angles, which McCusker §11 ranks with the profile fit and above every R value, so read it before step 16. Nothing scores it: a Si–O at 1.75 Å or a 60° O–M–O is yours to recognise. The number of rows naming an atom is its coordination number, and a None esd means no covariance behind the row or fixed by symmetry, never zero.

13. Quote no esd without its inflation. statistics.esd_inflation is the Bérar-Lelann factor for serial correlation, and it has an expected value of ≈1.51 even for perfectly white residuals, so it is an upper bound on the damage rather than a measurement of it. report.identifiability carries the trio to pass on with any esd — raw χ²_red, the inflation, Durbin-Watson — plus the δR line. Scaling variances by GoF² alone is "highly questionable" (Schwarzenbach, 1989).

14. Ask whether the converged answer is the only one, and settle it by a swap. report.identifiability.exchanges and .soft_modes outrank the statistics, and the verdict that licenses is ambiguous, not converged.

    They are about what "converged" means: converged is a statement about the free set, while an exchangeable=True row says a held parameter's signature is reproducible inside the fitted span and that a fitted partner stands many σ from its null.

    The swap resolves it and is a measurement: fit each member of the pair alone, the other held at its null, and compare χ² — two warm fits, seconds, and rx.report.compare_rivals(ref, data, finding) runs exactly that. R² cannot stand in for it. Read the outcome on RIVAL_DECISIVE_MIN_CHI2_RATIO (= 1.10, rietx.report), the losing rival's χ² over the winner's: at or above it the data has chosen, and you quote the winner without caveat, since hedging a won swap is a measured failure rather than caution; below it the pair is genuinely unresolved and the resolution is protocol or a declared ambiguity. No sentence converts a tie into an answer. The licence also travels as result.statistics.identifiability_clause.

    What you must not do is free the held parameter alongside its partner and refit: both free lands on §3's degenerate ridge and reports the unconstrained combination at a better Rwp — the most common misreading of the clause.

15. Read what the background is doing before you read Rwp, because it decides how to read Rwp. In report.background, worst_absorption (with worst_absorption_path) is how much of a structural parameter the background column span can reproduce, and off_region_chi2_reduced with off_region_durbin_watson is whether the residual between the peak regions is systematic. Layer 0's regions are peak clusters, so that second failure lands in no report.regions entry and step 10 cannot see it.

16. Only then Rwp and GoF, and never alone — as a pair with background.rwp_background_subtracted. The raw number is flattered by whatever the background carries. The subtracted one is what separates two fits of the same data.

17. Read the structure R factors last, and never in isolation. result.phase_agreement carries r_bragg (R_B) and r_f (R_F) per phase. A powder pattern does not measure individual reflection intensities, so I(obs) is the pattern partitioned in proportion to I(calc): a wrong model receives the intensity it predicted and both flatter it (Toby 2006: R_Bragg "has no statistical validity"). Watch R_B fall as you improve a model; never read it as evidence a correction helped. Absent in Le Bail and Pawley, where the intensities are the fit. Do not compare a trace phase's R_B with the major phase's: neither is weighted, and a minor phase's windows sit under the major phase's peaks.

Adding a parameter: its t-ratio first, then ΔBIC at N/f². A powder residual is serially correlated, so at raw N both ΔBIC and Hamilton's test bless any χ² gain. rx.report.compare_freed(ref, trial) returns the pair.

Comparing against another code means adopting its protocol: its refined set, held parameters and excluded regions, then a matching channel count, before any Rwp comparison (references/judging.md).

4b. Declare the deliverable — "good enough" is a question about purpose

Much real work is non-ideal by construction — nanoparticle broadening, intensity error from unknown pore contents. No bar moves for such data: the gates auto-scale to information content, and "good enough" is a different question answered exactly, not a relaxed standard. What changes is which report rows decide your deliverable. Declare it, then read its rows: the report is purpose-neutral and will not infer yours. ref.summary(deliverable=…) prints them for one fit, and a chain's are on SeriesResult.summary(deliverable="series").

The QPA row outranks every statistic beside it: an over-flexible background wins on every agreement index while biasing displacement parameters to 0.958 and 0.000 Ų against a truth of 0.5, and worst_absorption (0.46 against 0.08) is the only row separating the two fits; the plot does not.

resolution_limited is a stopping point, not a failure: the edit directions are indistinguishable on merged peaks, not the model wrong — a legitimate end state for phase-ID work; for structure-grade work, collect better data.

The capability floor. Verify before acting (rx.report.predict_then_verify, or a history branch), treat a capped confidence as an unresolved question rather than a low-priority instruction, and never execute a vetoed action. There is no ceiling: the report supplies evidence, judgement stays with the reader.

Every deliverable's worked measurement, and the round robins' two QPA rules: references/judging.md.

6. Abstention is a result. Do not convert it into a number

The package's hardest rule is never return a confident wrong singleton. Several places will decline to answer. When they do, that is the answer.

18. Propagate an abstention; do not paper over it. report.abstained_reason set means the global maturity gate refused Layer 1 — branch on abstained_kind first, and do not read attribution. INDEX_ABSTAINED means the candidates are there so you can see what was considered, not so you can pick one.
19. Never take candidates[0] because it is ranked first. The ranking orders the hypotheses, the gate judges them, and the two are different questions. IndexingResult.best_or_none() returning None is the most likely outcome of a first indexing run and is not a failure: read each candidate's confidence_caveats and act on the refuting ones first.
20. A failed gate is not a cause. region.gates_passed is False means the coefficients are present for transparency only; read region.gate_failures, whose codes are closed and typed.
21. Two collinear templates are not one answer. A trend reported non-separable, a PAWLEY_OVERLAP_UNRESOLVED group, an INDEX_GEOMETRIC_AMBIGUITY, an EXTINCTION_GROUPS_NOT_SEPARABLE: the information is absent from the measurement, not buried in noise. Extend the range, report both, or carry the whole list forward — a group's sum is the datum, and no counting time separates space groups differing only by elements that produce no absences.
22. A held or unquotable value is not a measurement. PHASE_UNCONSTRAINED, STEPHENS_STRAIN_NOT_POSITIVE, BOUND_HIT and HARMONIC_HELD each say a number in the result did not come from the data. Their esds do not make them measurements, and a good Rwp does not cover them: a parameter that does not move y_calc does not move Rwp either.

Every signal, its meaning and its correct response: references/abstention.md.

10. A worked default

A lab pattern, a CIF and no other information. Adapt, do not skip the checks — the right order depends on the data and the starting values (Toby, 2024).

import rietx as rx

data       = rx.read_pattern("sample.xy")
structure  = rx.Structure.from_cif("phase.cif")
instrument = rx.Instrument.bragg_brentano(radiation="CuKa",
                                          monochromator_two_theta=26.6)
instrument.background = rx.auto_background(data)

ref = rx.Refinement(structure, instrument, history="session.jsonl")

# 1. structure-free first: cell + profile without any structural assumption
ref.fit(data, mode="lebail", plan="profile_only")

# 2. Rietveld from the converged cell/profile
result = ref.fit(data, plan="lab_bragg_brentano")

# 3. the termination view: per-stage status, diagnostics, agreement indices last
print(result)

# 4. guards outrank statistics, and each one carries what to do about it
for d in result.diagnostics:
    print(d.level, d.code, d.where, d.message, "->", d.suggestion)

# 5. numbers, not pixels
report = ref.report(plan="lab_bragg_brentano")
if report.abstained_reason:
    print("Layer 1 abstained:", report.abstained_reason, report.abstained_kind)
else:
    for r in report.attribution:
        if r.gates_passed:
            print(r.two_theta_lo, r.two_theta_hi,
                  [(c.kind, c.value, c.stderr, c.share) for c in r.coefficients])

# 6. impurities / missing phases
print([u for u in report.unmatched if u.kind == "unmatched_obs"])

# 7. the whole judgement in one call, for a declared purpose
print(ref.summary(deliverable="structure"))

The three stop conditions. Stop refining when

24. every diagnostic is understood and either resolved or reported as a caveat;
25. Layer 1 attributes no remaining region above the significance gate; and
26. adding the next parameter group fails a ΔBIC test or trips a guard.

Do not stop merely because Rwp stopped falling, and do not continue merely because it is still falling. These are the structure-grade conditions; §4b maps the earlier stopping points a declared phase-ID or QPA deliverable is entitled to.

What to report. The refined values with their (inflated) esds, the diagnostics you could not resolve named as systematics, the protocol you actually ran (plan, held parameters, excluded ranges, channel count), and the package version, backend and solver from result.provenance. A number without its protocol is not a measurement.

The API

There is one integration surface: the Python API. Dump a typed answer with model_dump(mode="json"); a failure raises, with no envelope or error code.

Do not quote a signature from memory. rx.capabilities() says what this build supports, rx.help_for(path) says what a parameter is, and inspect.signature(obj) or help(obj) gives any call's arguments. The names are references/api.md, every one of them checked against the installed package by test.

See also

• The manual, Part 2 (theory): https://rietx.org/manual.html — every equation with its source, and the bibliography each author-year citation below resolves in
• The repository, https://github.com/yue-here/rietx: README.md, docs/DESIGN.md (why the FitReport is shaped this way), tests/data/README.md (provenance of every bundled dataset); none ships in the wheel