er-text-normalization

Imported from HBX814/AmazonMLChallenge (.claude/skills/er-text-normalization/SKILL.md). Install upstream with npx skills add HBX814/AmazonMLChallenge --skill er-text-normalization. Copyright stays with the author.

Text Normalization (names + addresses)

Overview

Every noise operator of the data generator (native script, accents, suffix swaps, DBA aliases, domain forms, NULL tokens, state code/name/native, department/region) should become string equality after normalization. Core principle: NFKC → transliterate (learned dictionary first, rules fallback) → fold accents → clean with country-keyed maps that fall back to generic rules, applied to DISTINCT strings in a process pool. The native-token dictionary learned from TRAIN links is the largest single gain for India.

When to use

• Playbook step 2 (before blocking / features), or when rebuilding the normalized parquet caches.
• India links with native-script names score low; a France (test-only) or unknown-country row behaves oddly; a new noise pattern shows up in error analysis.
• Normalization is slow or runs out of memory.

Not for: reading TSVs safely (REQUIRED SUB-SKILL: er-data-loading — literal "NULL"/"None" must stay strings), choosing blocking passes (See also: er-blocking-candidates), pairwise similarity features (See also: er-pair-features).

Quick reference

Order of operations (never reorder)

1. unicodedata.normalize("NFKC") + quote/dash unification (fullwidth and math letters become ASCII).
2. transliterate: each Indic run (9 scripts) looked up in the learned dictionary per token, else rules (ISCII-parallel table, schwa deletion for northern scripts, ph→f). Latin accents are untouched here.
3. fold: NFKD, drop combining marks, lowercase (ASCII fast path).
4. Clean: DBA/aka/fka split (keep the part AFTER the marker; 1,007/1,007 measured), [[junk]], " | tail", phone numbers, domain → glued token, @handle, dotted abbreviations (L.L.C.→llc), M/s, &→and, possessive, leetspeak (5ervices), lnc→inc; tokens [0-9a-z]+.
5. Country-keyed maps (legal forms, street types, states), then the generic union.

Why this order: NFD+strip before transliteration deletes Indic vowel signs (लिमिटेड → लमटड); crude folding gives mean Jaro-Winkler 0.425 on native-script true links.

Representations (columns of add_normalized_columns)

Measured 2026-09-25 on mini-train true links (42,094 links) vs random same-country S1 unless stated.

house_nums and city_key numbers were measured by this skill's author (mini train, same protocol); the rest come from reference-normalization-research.md.

Countries: keyed maps + generic fallback

country_key(country) → us | india | france | generic. Any other label, None or "" gets generic = union of the three maps minus ambiguous short forms (dr, ste, ch, mg, ...); bare 2-letter state codes are mapped only inside a known country (IN = Indiana vs India, FL = Florida vs floor). France (15% of test S1, absent from train) has its own legal forms (SARL, SAS, SASU, EURL, SCI, EI, Cie, Ets, & Fils/Frères), street types (R./Rue, AV., BD, Imp., Allée), N°/bis/ter, department→region and city→department→region maps. To support a new country, add a key to the maps; never filter rows by country or one-hot it.

The native-token dictionary (a learned artifact)

The generator spells each English token the same way per script, so 162,174 distinct (S1 name, native name) train pairs contain only 1,347 native tokens: a token dictionary inverts it almost exactly.

End to end, name_core exact equality on these links goes from 1.4% to 95.3%. Coverage on the unlabeled test India names (867,245 rows): 96.4% of token occurrences, 88.7% of types. The 171 missing types are new shop words (stores, traders, jewellers, bakery, motors, ...), so keep the rules fallback and soft token matching in the matcher (unseen token vs the true S1: phonetic-key JW 0.864, vs a random S1 0.616). Hard snapping to a large vocabulary is wrong 58% of the time; enable_snapping(counts, min_count=500, thr=0.7) is 62.7% right / 3.1% wrong and stays off by default.

Regenerate it inside the pipeline (prepare stage) from train links; the committed native_token_dict.example.tsv is for tests only, and nobody hand-edits it:

import polars as pl
from ber import normalize_frame as NF        # flat layout: import normalize_frame as NF

def normalize_split(tr, te, work, val_s1=None):
    # TRAIN links only. When scoring a validation fold, hold its S1 out of the dictionary too.
    NF.learn_dict_from_train(tr["s1"], pl.concat([tr["s2"], tr["s3"]]), tr["gt"],
                             f"{work}/native_token_dict.tsv", exclude_s1=val_s1)
    out = {}
    for name, split in (("train", tr), ("test", te)):
        pool = NF.add_normalized_columns(pl.concat([split["s2"], split["s3"]]), extras=True)
        s1 = NF.add_normalized_columns(split["s1"], extras=True, city_counts=NF.city_counts(pool))
        out[name] = (s1, pool)                # write each to work/*.parquet right away
    return out

if __name__ == "__main__":                    # spawn workers re-import __main__ on Windows
    ...

The workers receive the dictionary loaded in the parent (initializer), so load or learn it before calling add_normalized_columns. Full train: 53-64 s, under 1 GB (measured). Record in the Documentation that the dictionary is derived from the provided train links only.

Running it at scale

• Distinct values, not rows: unique() per (string, country), map, join back (maintain_order="left"). On S2/S3 the saving is modest: distinct names/addresses are 94.3% / 95.8% of 200k real India S2 rows, 96.5% / 87.0% on mini train S2.
• Speed (i5-1235U laptop, CPU 77-91% busy with other agents, so treat as upper bounds): mini files 300-600 µs/row in-process; 200k real India S2 rows: 71 s with the pool (357 µs/row wall) (single-process wall time was not measured; the pool only pays off above ~20k distinct strings, see _min_parallel). Scalar costs: name_variants 50-80 µs, address_all 150-240 µs.
• Pool: n_jobs=-1 → min(8, cpu−2) spawn workers, capped at free RAM / 150 MB; frames under 20k distinct values run in-process (spawn start-up costs more than it saves). One pool serves both passes; at most 2×workers chunks are in flight.
• Full data (≈24M rows) is about 2 CPU-hours: run per country partition, write parquet after each, or rent a box (See also: er-compute-and-aws).

Verify

cd C:/Users/harsh/Downloads/AmazonMLChallenge/.claude/skills/er-text-normalization
set PYTHONDONTWRITEBYTECODE=1 & set PYTHONIOENCODING=utf-8
..\..\..\.venv\Scripts\python -m pytest -q -p no:cacheprovider test_normalize.py test_normalize_frame.py

Expected: 38 + 15 passed (the two mini-data tests skip when ER_MINI_DIR has no mini train). Exit gate for playbook step 2: gate on full train reports dictionary name_core TSR ≳ 99 on held-out native-script links (measured on 3,852 held-out mini native-script links: core TSR 99.7, JW 0.996, exact core match 95.3% with the dictionary vs TSR 63.5 without it).

Common mistakes

See also: er-error-analysis (adding a new noise rule: add a failing test in test_normalize.py first), reference-normalization-research.md (full measurements, per-script numbers, all maps).