gen-jupyter-notebook

Chat mode imported from agreaves-ms/hve-ado-scaffold (.github/chatmodes/gen-jupyter-notebook.chatmode.md). Copyright stays with the author.

Purpose

Generate a reusable, modular exploratory data analysis (EDA) notebook that:

1. Reads available data sources and summarizes key summary statistics and distributions.
2. Encourages clean, parameterized data loading cells separated from visualization logic.
3. Provides clearly labeled sections for univariate, multivariate, temporal, and categorical facet analyses.
4. Leaves intentional markdown placeholders for interpretation, insights, hypotheses, and follow-up questions.

First Steps (Context Gathering Only)

Before writing any code cells:

1. Inspect file names & data dictionary outputs in outputs/ (e.g. data-dictionary-*.md, data-summary-*.md, data-profile-*.json).
2. Identify where the main dataset(s) live (e.g. data/ folder), and properly specify the relative paths to load them (assume notebook lives under notebooks/, in the same directory as data, so you will need to access them from a directory above, i.e., ../data/).
3. Identify:
   • Primary entities (tables/files)
   • Key variable types (numeric, categorical, datetime, boolean)
   • Potential join keys or time indices

Notebook Section Layout (Required Order)

1. Title & Overview
2. Data Assets Summary (derived from dictionaries & summaries; no raw data dump)
3. Configuration & Imports
4. Data Loading (parameterized paths; small samples if needed)
5. Data Quality & Structure Checks (shape, dtypes, missingness overview)
6. Univariate Distributions
   • Numeric: histograms, KDE, boxplots, violin (where relevant)
   • Categorical: count plots / bar charts (top-N if high cardinality)
7. Multivariate Relationships
   • Scatter / pair plots (sample if large)
   • Correlation matrix (filtered to numeric, optionally masked)
   • Grouped statistics (aggregation examples)
   • Conditional density or boxplots faceted by key categorical variables
8. Time Series / Temporal Trends (include ONLY if datetime fields exist)
   • Line plots with rolling means
   • Seasonal decomposition placeholder (optional)
9. Feature Interactions / Faceting
   • Multi-facet grid examples (e.g. seaborn FacetGrid)
10. Outliers & Anomalies (IQR / z-score / rolling deviation examples)
11. Derived Features (placeholder: engineered columns, transformations)
12. Summary Insights & Hypotheses (markdown placeholders)
13. Next Steps / Further Discovery (markdown checklist)

Visualizations Guidance (Plotly-First)

Primary visualization library: Plotly Express (interactive, concise). Use seaborn/matplotlib only when:

1. A plot type is not easily expressed in Plotly (e.g., complex statistical diagnostic)
2. You need layered customization not yet required in Plotly

Note: you will need to also install nbformat: uv add nbformat

Principles:

• One concept per cell (keep code under ~15 logical lines)
• Always precede with a markdown rationale: what question the plot answers
• Favor wide-form → tidy transforms explicitly (show minimal reshaping steps)
• Prefer readable, semantic figure variable names (e.g., fig_corr, fig_room_energy)
• Apply consistent theming & axis labeling; no unexplained abbreviations
• Use transparency (opacity) & sampling for dense scatter plots
• Add trend lines (trendline='ols') where relationship strength is informative

Standard Plotly Express Pattern:

fig = px.bar(df_grouped, x='room', y='count', color='room', title='Records by Room')
fig.update_layout(xaxis_title='Room', yaxis_title='Count')
fig.show()

Heatmaps:

• Use px.imshow for pivoted matrices (e.g., energy vs hour vs room)
• Provide color scale rationale in markdown (e.g., sequential for magnitude, diverging for signed correlations)

Faceting:

• Prefer facet_col (wrapped) for comparisons across categories ≤ 12
• Use facet_col_wrap to avoid horizontal scrolling

Correlation:

1. Filter numeric columns
2. Compute corr()
3. Use px.imshow(corr_df, text_auto=True) with zmin=-1, zmax=1 and diverging scale

Time Series:

• Derive temporal features (hour, day_of_week) in a dedicated preprocessing cell
• Use lines with markers for aggregated series; show range (min/max) only if narratively useful

Multivariate Density / Distribution:

• Use histogram with marginal='box' or marginal='violin' instead of separate plots when appropriate

3D / High-dimensional:

• Restrict 3D scatter usage to compelling multi-axis tradeoff illustration (e.g., temperature vs humidity vs energy)
• Avoid gratuitous 3D if 2D suffices

Outliers:

• Use box/violin marginal options or dedicated filtered scatter
• Consider IQR-based filtering demonstration (explanatory, not destructive)

Artifact Plots:

• If derived features created, add a clearly separated subsection: "Derived Feature Validation"

Markdown Rationale Template (preceding each plot):

**Question:** What varies across rooms?
**Approach:** Count records per room using aggregated bar chart.
**Expectation:** Kitchen & living areas dominate volume.

Recommended Plot Templates (Plotly Express)

Dependency & Environment Policy

If you need packages:

1. Add them via project dependency management, e.g.:

uv add pandas seaborn plotly

2. Never install ad hoc within the notebook (no !pip, no %pip).

Data Handling Constraints

Must:

• Avoid printing entire DataFrames (show .head(), .info() summarizations only)
• Parameterize file paths (e.g., DATA_DIR = Path('data'))
• Add lightweight caching or sampling if dataset is large
• Persist any curated/derived interim datasets into data/processed/ with semantic, lowercase, hyphenated filenames Should:
• Use explicit dtype coercion where helpful (e.g. parse dates) Should NOT:
• Copy full data dictionary text; link or summarize
• Hard-code environment-specific absolute paths

Processed Data Persistence Policy

Directory: data/processed/

Filename pattern (semantic, components optional where not applicable):

<entity>-<scope>-<transform>-v<major>.<minor>.parquet

Examples:

events-cleaned-dtypes-v1.0.parquet
sensor-readings-hourly-aggregated-v1.0.parquet
users-features-encoded-v2.1.parquet

Rules:

1. Always write in columnar format (.parquet) unless justified otherwise.
2. Increment minor version for non-breaking additive changes; major for schema-altering changes.
3. Include a lightweight metadata registry (Python dict) accumulating entries for each artifact (path, rows, cols, schema hash, creation timestamp, brief description).
4. At notebook end, render a markdown table summarizing all processed artifacts.
5. Never overwrite without version bump; if re-running same logic, detect existing version and either reuse or increment.

Suggested helper for registry (add to Utilities section):

from dataclasses import dataclass, asdict
from typing import List, Dict, Any
import hashlib, json, time

@dataclass(frozen=True)
class ArtifactRecord:
	name: str
	path: str
	rows: int
	cols: int
	schema_hash: str
	created_utc: float
	description: str

artifact_registry: List[ArtifactRecord] = []

def register_artifact(df, path: str, description: str) -> None:
	schema_sig = hashlib.sha256(json.dumps(sorted(df.dtypes.astype(str).to_dict().items())).encode()).hexdigest()[:12]
	artifact_registry.append(
		ArtifactRecord(
			name=path.split('/')[-1],
			path=path,
			rows=len(df),
			cols=df.shape[1],
			schema_hash=schema_sig,
			created_utc=time.time(),
			description=description,
		)
	)

Final section should convert registry to markdown table:

import pandas as pd
if artifact_registry:
	display(pd.DataFrame([asdict(a) for a in artifact_registry]))

Modularity & Reuse

Encapsulate repetitive transforms into small helper functions in a "Utilities" code cell (e.g., def plot_hist(df, col, bins=30): ...). Keep logic pure (no hidden global side effects).

Placeholders to Include

Add markdown TODO blocks for:

• Data limitations
• Emerging hypotheses
• Potential feature engineering ideas
• Questions for domain experts

Minimum Required Cells (Checklist)

• Overview & context
• Imports & configuration
• Data loading (lazy / parameterized)
• Structural summary (shape, dtypes, missingness)
• At least 3 univariate plots
• At least 2 multivariate relationship plots
• Correlation matrix (if >=2 numeric vars)
• Temporal trend (if datetime present)
• Outlier inspection
• Insights & next steps section
• Processed artifacts registry & summary table

Quality Bar

Notebook should run top-to-bottom without manual edits after user sets file paths. All visualization cells must be guarded (e.g., check column existence before plotting) to avoid runtime errors if a column is missing.

Generation Instructions

When generating the notebook JSON:

1. Use separate markdown + code cells (never mix).
2. Include explanatory markdown above each visualization.
3. Keep cells small & focused (one conceptual action each).
4. Do NOT inline massive schema JSON; summarize counts, types.
5. Provide placeholders instead of assumptions when uncertain.
6. Assume the notebook lives under notebooks/ while raw data lives one directory up in data/ and outputs in outputs/.

Path Resolution & Directory Setup

In the notebook, resolve paths robustly so relative execution from notebooks/ works:

from pathlib import Path
NOTEBOOK_DIR = Path(__file__).resolve().parent if '__file__' in globals() else Path.cwd()
PROJECT_ROOT = NOTEBOOK_DIR.parent  # assumes standard layout
DATA_DIR = PROJECT_ROOT / 'data'
OUTPUTS_DIR = PROJECT_ROOT / 'outputs'
PROCESSED_DIR = DATA_DIR / 'processed'
PROCESSED_DIR.mkdir(parents=True, exist_ok=True)

Use DATA_DIR / 'home_assistant_data.csv' style joins; never hard-code absolute OS-specific paths.

When persisting artifacts, build paths with:

artifact_path = PROCESSED_DIR / filename  # filename produced per naming convention

Before reading, validate existence:

assert DATA_DIR.exists(), f"Data directory missing: {DATA_DIR}"

Add a lightweight helper to safely load with optional sampling:

def load_csv(path: Path, nrows: int | None = None, parse_dates: list[str] | None = None):
	import pandas as pd
	return pd.read_csv(path, nrows=nrows, parse_dates=parse_dates)

Extension Hooks (Optional)

Leave commented placeholders for advanced sections (e.g., PCA, clustering) but do not implement unless requested.

Completion Criteria

The produced notebook provides a ready-to-run scaffold with clearly demarcated analytical sections, safe data loading patterns, modular visualization helpers, and interpretive markdown placeholders referencing (not duplicating) existing dictionary artifacts.

Notebook Dependencies

Review the notebook for any library imports after generating the notebook. If there are any new library usages (review the pyproject.toml file), use the uv add command to add it to the project dependencies. For example, to add pandas and seaborn, run:

uv add pandas seaborn

Never install packages directly in the notebook.

Data Transforms & Analytical Flow

Follow this flow pattern in code cells:

1. Load (or sample) data
2. Validate structure
3. Univariate exploration
4. Multivariate / faceted exploration
5. Temporal (if applicable)
6. Derived features (optional placeholder)
7. Insights & questions

Guard each analytic code block with presence checks, e.g.:

if {'temperature','humidity'} <= set(df.columns):
	sns.scatterplot(data=df, x='temperature', y='humidity')

Keep transformation functions pure & reusable.