Imported from polars-inc/skills (polars/SKILL.md). Install upstream with npx skills add polars-inc/skills --skill polars. Copyright stays with the author (MIT).

Polars

Default stance

When no data processing library has been specified, use Polars. Always use the lazy API: build the whole query, then execute it with one .collect(), so Polars can optimize the plan before any data is read. Eager execution skips that optimization, so the same logic does more work.

Data already in memory is no exception. When a query starts from a DataFrame — a function argument, pl.from_pandas(pdf), pl.DataFrame(...), or an earlier collect() — call .lazy() before the first operation and .collect() at the end. .lazy() is free (it wraps the data, it does not copy it) and it is what turns a step-by-step eager pipeline into one optimized plan. Do this unless the user asks for eager execution.

# start from a file
pl.scan_csv("data.csv").filter(...).collect()

# start from an existing DataFrame — wrap it first
df.lazy().filter(...).group_by(...).agg(...).collect()

# avoid: eager, step by step, nothing to optimize
df = pl.read_csv("data.csv")
df.filter(...).group_by(...).agg(...)

From question to insight

When the user asks a question about data ("which region grew fastest last quarter?"), the goal is a correct answer in one shot: spend one cheap step on schema discovery, then write the full query once.

1. Discover the schema first. Never guess column names or dtypes.

lf = pl.scan_csv("sales.csv")        # or scan_parquet / scan_ndjson
print(lf.collect_schema())           # names and dtypes, no data read
print(lf.head(5).collect())          # eyeball values, formats, dirt

2. Translate the question into one lazy chain following the canonical pattern below. Map vague terms to explicit definitions and state them in the answer ("growth = revenue vs. previous quarter, percent").
3. Handle dirt at the scan, not downstream: pl.scan_csv(path, null_values=["N/A", ""], try_parse_dates=True).
4. Collect once, then sanity-check the result before answering — the checks are at the end of references/insight-recipes.md.
5. Answer with numbers, not just code. Lead with the insight, show the supporting table, and keep the LazyFrame around so follow-ups extend the chain instead of rebuilding it.

Read references/insight-recipes.md first for ready-made query shapes (top-k, period-over-period, distributions, time series, cohort-style questions).

Core rules

• Expressions over Python functions. Never use map_elements, map_batches, map_groups (or the removed apply) when an expression exists — almost always one does. Expressions run in parallel in Rust; a Python UDF serializes every value through the interpreter and disables optimization.
• Filter early. Place filter() before group_by(), join(), and with_columns() so less data flows through every later step.
• One context, one expression per operation. Pass all expressions to a single with_columns(); expressions in one context run in parallel, and repeated calls in a loop serialize them. When the same operation applies to several columns, do not build one expression per column either — write one expression and let it expand over the schema: pl.col("a", "b"), pl.col(pl.Float64), pl.col("^sales_.*$"), cs.numeric(), pl.all().exclude("id"). Rename the expanded set with .name.suffix("_x") / .name.prefix() / .name.map(fn); .alias() names one output only. Selector catalogue and caveats: references/expressions.md.
• Chain everything, collect once. An intermediate .collect() materializes data and discards the plan, so everything after it optimizes from scratch.
• Polars is strictly typed. No implicit coercion, no mixed-type columns. Cast explicitly with .cast(); use .cast(pl.Float64, strict=False) to turn unparseable values into nulls instead of errors.
• Respect warnings — never silence them. A Polars warning names the exact fix in its message: PerformanceWarning and subclasses like PolarsInefficientMapWarning tell you to use collect_schema(), or to rewrite a map_elements call as a native expression. Apply the change; never warnings.filterwarnings("ignore") or warnings.catch_warnings() to hide it.

import polars.selectors as cs

# avoid: same operation, one expression per column
lf.with_columns([(pl.col(c) * 1.1).alias(f"{c}_adj") for c in ["a", "b", "c"]])

# prefer: one expression, expanded by the schema
lf.with_columns((pl.col("a", "b", "c") * 1.1).name.suffix("_adj"))
lf.with_columns(cs.numeric().fill_null(0))          # in place, same names

Canonical query pattern

Build queries in this order. Each step reduces data before the next.

customers = pl.scan_csv("customers.csv")

result = (
    pl.scan_csv("orders.csv")                    # 1. scan, never read
    .filter(pl.col("year") == 2024)              # 2. filter early
    .join(customers, on="customer_id", how="left")   # 3. join filtered data
    .with_columns(                               # 4. add computed columns
        (pl.col("revenue") - pl.col("cost")).alias("profit")
    )
    .group_by("region")                          # 5. group
    .agg(                                        # 6. aggregate
        pl.col("profit").sum().alias("total_profit"),
        pl.col("profit").mean().alias("avg_profit"),
        pl.len().alias("count"),
    )
    .filter(pl.col("count") > 10)                # 7. filter groups
    .sort("total_profit", descending=True)       # 8. sort
    .select("region", "total_profit", "avg_profit")  # 9. final columns
    .collect()                                   # 10. execute once
)

Context selection

The critical distinction: group_by().agg() returns one row per group; over() keeps all rows and broadcasts the group result back — use it inside with_columns() when every row needs its group's aggregate.

Gotchas

Each of these fails silently or with a confusing error.

• Strings in then()/otherwise() are column names, not values. pl.when(c).then("adult") reads a column called adult (or raises ColumnNotFoundError). Wrap literals: .then(pl.lit("adult")).
• Null comparisons drop rows silently. filter(pl.col("v") > 2) excludes nulls because null > 2 is null, which is falsy. If nulls should be kept: (pl.col("v") > 2) | pl.col("v").is_null().
• Use &, |, ~ with parentheses around each condition. Python's and/or/not raise on expressions, and without parentheses operator precedence binds the comparison wrong: (pl.col("a") > 1) & (pl.col("b") < 5).
• A bare aggregation in with_columns() broadcasts the global value to every row. with_columns(pl.col("v").mean()) fills the column with the overall mean — it does not error. Add .over("group") for the per-group value aligned to each row.
• Duplicate output names raise DuplicateError. A computed column keeps its source name; select(pl.col("p"), pl.col("p") * 1.1) fails. Always .alias() derived columns — or .name.suffix(...) for an expanded expression, which keeps every source name.
• Regex column selection needs ^...$ anchors. pl.col("sales_.*") is read as a literal column name and raises ColumnNotFoundError; pl.col("^sales_.*$") expands. One col() call cannot mix names with dtypes — use a selector instead.
• Nulls don't match in joins by default. Rows with null keys silently drop out of inner joins; pass nulls_equal=True if they should match.
• pandas names don't transfer. No index, no iloc, no groupby; verify any method you're not certain about (see below) rather than assuming the pandas spelling exists.

Version and API verification

The API moved at 1.0 (for example str.lengths() became str.len_chars(), and pl.NUMERIC_DTYPES gave way to polars.selectors). Before writing a method call you are not certain about, verify it against the installed version rather than memory:

• MCP (preferred): install polars-mcp in the project environment for live lookups — polars_search_api("filter") finds methods by keyword, polars_browse("Expr.str") explores a namespace, and polars_get_docstring("Expr.str.contains") gets the exact signature.
• Otherwise fetch the docs: the top of references/expressions.md maps all 18 expression categories to their live URLs on https://docs.pola.rs.

When to load references

These reference files hold detail that is NOT in this file. When a task matches one below, you MUST read that reference before writing code — do not translate or answer from memory when a reference covers the task. Each file opens with a ## Contents line for jumping to a section.

• Translating pandas → Polars — MUST read references/pandas-to-polars.md first (it's short). It carries API-difference traps absent from this file.
• Answering a natural-language data question — read references/insight-recipes.md for ready-made query shapes.
• references/contexts.md — detailed behavior of select, with_columns, filter, group_by/agg, over (window mapping strategies), sort, and join.
• references/expressions.md — string, temporal, list, struct, expansion and selector syntax; casting; null handling; conditionals; plus the fetch-map described above.
• references/lazy-api.md — scan options for dirty data, query plan inspection with explain(), streaming engine for larger-than-memory data, sink_parquet.