Name: 03-cleaning-preprocessing
Author: kelvinlim

Prompt file imported from kelvinlim/localexpert (.github/prompts/03-cleaning-preprocessing.prompt.md). Copyright stays with the author.

When to use this: After EDA/missingness diagnostics, before statistical testing.

You are performing a cleaning-preprocessing analysis (Phase 3) in the open Jupyter notebook. Work incrementally: add and run one cell at a time and inspect each output before continuing. Load the dataset the user names (ask which file if unclear). Follow the procedure below and satisfy every check, then write a short markdown summary as the final cell.

Objective

Transform messy raw data into an analysis-ready dataset, choosing each mitigation from the Phase 2 diagnostics rather than by default, and documenting every decision.

Procedure

Handle missingness using the mechanism identified in Phase 2:
- MCAR/MAR and minimal (<5% on the variable): listwise deletion (df.dropna()) is acceptable.
- Widespread missingness: multiple imputation to preserve statistical power via sklearn.experimental.enable_iterative_imputer then sklearn.impute.IterativeImputer (the scikit-learn analogue of MICE).
Mitigate outliers: detect with the IQR rule (values outside Q1 - 1.5*IQR .. Q3 + 1.5*IQR) or z-scores (scipy.stats.zscore, |z| > 3). Decide a priori whether to remove, winsorize (scipy.stats.mstats.winsorize), or transform — do not silently drop.
Transform variables that violate normality: log, square-root, or Box-Cox (scipy.stats.boxcox, requires strictly positive values). Re-check skewness after.
Scale features when combining diverse streams (e.g. millisecond reaction times with microvolt skin conductance): z-score standardisation (StandardScaler) or min-max (MinMaxScaler).

Checks

Every missing value is either imputed or its row removed — no silent NaNs remain in the analysis columns.
The outlier policy (remove/winsorize/transform) is stated before it is applied.
Post-transformation skewness is reported so the effect is visible.
Row count before vs. after cleaning is reported.

Output

A cleaned DataFrame plus a markdown changelog: what was imputed, what was dropped/winsorized/transformed, and the resulting row count and skewness.

**When to use this:** After EDA/missingness diagnostics, before statistical testing. You are performing a **cleaning-preprocessing** analysis (Phase 3) in the open Jupyter notebook. Work incrementally: add and run one cell at a time and inspect each output before continuing. Load the dataset the user names (ask which file if unclear). Follow the procedure below and satisfy every check, then write a short markdown summary as the final cell. ## Objective Transform messy raw data into an analysis-ready dataset, choosing each mitigation from the Phase 2 diagnostics rather than by default, and documenting every decision. ## Procedure 1. **Handle missingness** using the mechanism identified in Phase 2: - MCAR/MAR and minimal (<5% on the variable): listwise deletion (`df.dropna()`) is acceptable. - Widespread missingness: multiple imputation to preserve statistical power via `sklearn.experimental.enable_iterative_imputer` then `sklearn.impute.IterativeImputer` (the scikit-learn analogue of MICE). 2. **Mitigate outliers**: detect with the IQR rule (values outside `Q1 - 1.5*IQR` .. `Q3 + 1.5*IQR`) or z-scores (`scipy.stats.zscore`, |z| > 3). Decide *a priori* whether to remove, winsorize (`scipy.stats.mstats.winsorize`), or transform — do not silently drop. 3. **Transform** variables that violate normality: log, square-root, or Box-Cox (`scipy.stats.boxcox`, requires strictly positive values). Re-check skewness after. 4. **Scale** features when combining diverse streams (e.g. millisecond reaction times with microvolt skin conductance): z-score standardisation (`StandardScaler`) or min-max (`MinMaxScaler`). ## Checks - Every missing value is either imputed or its row removed — no silent NaNs remain in the analysis columns. - The outlier policy (remove/winsorize/transform) is stated before it is applied. - Post-transformation skewness is reported so the effect is visible. - Row count before vs. after cleaning is reported. ## Output A cleaned DataFrame plus a markdown changelog: what was imputed, what was dropped/winsorized/transformed, and the resulting row count and skewness.

{ "prompt": "**When to use this:** After EDA/missingness diagnostics, before statistical testing.\n\nYou are performing a **cleaning-preprocessing** analysis (Phase 3) in the open Jupyter notebook. Work incrementally: add and run one cell at a time and inspect each output before continuing. Load the dataset the user names (ask which file if unclear). Follow the procedure below and satisfy every check, then write a short markdown summary as the final cell.\n\n## Objective\nTransform messy raw data into an analysis-ready dataset, choosing each mitigation\nfrom the Phase 2 diagnostics rather than by default, and documenting every decision.\n\n## Procedure\n1. **Handle missingness** using the mechanism identified in Phase 2:\n - MCAR/MAR and minimal (<5% on the variable): listwise deletion (`df.dropna()`)\n is acceptable.\n - Widespread missingness: multiple imputation to preserve statistical power via\n `sklearn.experimental.enable_iterative_imputer` then\n `sklearn.impute.IterativeImputer` (the scikit-learn analogue of MICE).\n2. **Mitigate outliers**: detect with the IQR rule (values outside\n `Q1 - 1.5*IQR` .. `Q3 + 1.5*IQR`) or z-scores (`scipy.stats.zscore`, |z| > 3).\n Decide *a priori* whether to remove, winsorize (`scipy.stats.mstats.winsorize`),\n or transform — do not silently drop.\n3. **Transform** variables that violate normality: log, square-root, or Box-Cox\n (`scipy.stats.boxcox`, requires strictly positive values). Re-check skewness after.\n4. **Scale** features when combining diverse streams (e.g. millisecond reaction times\n with microvolt skin conductance): z-score standardisation (`StandardScaler`) or\n min-max (`MinMaxScaler`).\n\n## Checks\n- Every missing value is either imputed or its row removed — no silent NaNs remain\n in the analysis columns.\n- The outlier policy (remove/winsorize/transform) is stated before it is applied.\n- Post-transformation skewness is reported so the effect is visible.\n- Row count before vs. after cleaning is reported.\n\n## Output\nA cleaned DataFrame plus a markdown changelog: what was imputed, what was\ndropped/winsorized/transformed, and the resulting row count and skewness.", "variables": [], "notes": "Turn raw data into an analysis-ready state without introducing bias.", "metadata": { "source": { "provider": "github-prompts", "repository": "https://github.com/kelvinlim/localexpert", "path": ".github/prompts/03-cleaning-preprocessing.prompt.md", "ref": "ed5b72f51fa9923c9e13a693fc31ae6b87eb6119", "url": "https://github.com/kelvinlim/localexpert/blob/ed5b72f51fa9923c9e13a693fc31ae6b87eb6119/.github/prompts/03-cleaning-preprocessing.prompt.md", "key": "kelvinlim/localexpert/.github/prompts/03-cleaning-preprocessing.prompt.md" } } }

03-cleaning-preprocessing

About

Objective

Procedure

Checks

Output

Use it

Manifest

Reviews