policast

Imported from open-lakehouse/policast (AGENTS.md). Install upstream with npx skills add open-lakehouse/policast. Copyright stays with the author.

AGENTS.md — policast-cel

What this project is

policast-cel is a portable data governance compiler and enforcement layer for the open lakehouse. It compiles Cedar authorization policies into CEL (Common Expression Language) expressions, then enforces those expressions at query time across multiple engines — Apache DataFusion (Rust) and Apache Spark (Scala/JVM).

The core thesis: author governance rules once in Cedar, compile to a portable manifest of CEL expressions, and push enforcement into query engine optimizer rules and physical plan wrappers. No policy logic leaks into application SQL.

Why Cedar

Cedar is an open-source policy language (Rust-native, CNCF Sandbox track) purpose-built for authorization. It was chosen for this project because of properties that matter specifically in a multi-catalog lakehouse:

• Default deny, forbid overrides permit, order independence — composable safety guarantees across teams and catalogs.
• Bounded evaluation, no side effects — sub-millisecond decisions on the catalog hot path.
• SMT-based policy analysis — provable invariants about who can access what, not just unit tests.
• Reads like SQL — governance reviewers can audit policies without learning a programming language.
• Schema validation at authoring time — typos and type errors caught before deployment.

The full rationale, including a comparison with OPA/Rego and OpenFGA, and a portable Lakehouse Cedar schema covering Polaris, Lakekeeper, and Unity Catalog, is in research/cedar-open-lakehouse.md.

Architecture

Cedar Policy (.cedar)
        │
        ▼
   policast-core (Rust)
   ├── Cedar Parser (cedar-policy crate, EST)
   ├── Cedar EST → CEL Emitter
   └── Policy Manifest (JSON)
        │
        ├──────────────────────┐
        ▼                      ▼
   policast-datafusion     policast-spark
   (Rust)                  (Scala/JVM)
   ├── GovernedTable       ├── PolicastPlugin
   ├── CEL→Expr compiler   ├── Catalyst Rules
   ├── Row filters         └── CelEvaluator (cel-java)
   ├── Column masks
   └── Delta Lake (delta-rs, feature-gated)

Crate / module layout

Module-specific agent guides

Use these focused guides when working inside a specific module:

• crates/policast-core/AGENTS.md
• crates/policast-datafusion/AGENTS.md
• crates/policast-uc/AGENTS.md
• crates/policast-playground/AGENTS.md
• policast-spark/AGENTS.md

Key dependencies

• policast-core: cedar-policy (4.x), serde, serde_json, clap, thiserror; optional redis (redis feature) and prost (proto feature — checked-in policast.v1 codegen)
• policast-datafusion: datafusion (53), cel-interpreter (0.10), cel-parser (0.10), optional deltalake (0.32)
• policast-uc: reqwest (client/UC REST), axum/tower (sidecar), optional redis (redis feature — materialized protobuf PolicyStore) and deltalake/datafusion (uc-bootstrap — Delta snapshot + owner-aware editor)
• policast-playground: axum, datafusion, deltalake (with s3), policast-datafusion (GovernedTable), policast-uc (UC REST + compile)
• policast-spark: Spark 4.2, cel-java 0.12, Gson

Governance model

Three policy types are supported, all authored in Cedar and compiled to CEL:

Cedar policy annotations (@id, @filter_type, @target_table) drive the manifest structure. The PolicyManifest JSON is the handoff format between the compiler and the engines.

How the pieces connect

1. Author governance in .cedar files under examples/policies/.
2. Compile with cargo run -p policast-core -- --output manifest.json *.cedar.
3. Enforce in DataFusion — GovernedTable::new(inner_table, manifest, table_name, identity) wraps any TableProvider and injects filters/masks into the physical plan.
4. Enforce in Spark — PolicastPlugin + PolicastExtensions + Spark conf properties; Catalyst optimizer rule PolicastRowFilterRule rewrites logical plans.

Research and reference material

Catalog integration targets

The project is designed to produce governance that works across three catalog systems. Current status:

Conventions

• Branching: work on feature branches (feat/, fix/, chore/, refactor/, test/), never commit directly to main.
• Testing: every code change must be accompanied by tests; target ~80% coverage.
• Policy files: Cedar policies live in examples/policies/; compiled manifests are JSON.
• Feature gates: Delta Lake support is behind the delta Cargo feature in policast-datafusion.
• CI: lint + rust + spark run on every PR and are the required checks for main; the full docker smoke is advisory and opt-in (push to main, weekly, workflow_dispatch, or a run-smoke PR label). See docs/ci.md for the matrix, artifact reuse, and branch-protection setup.

Current state

Both completed plan files (.cursor/plans/) document the work done so far:

1. Cedar-to-CEL compiler POC — Rust core, parser, CEL emitter, manifest, DataFusion and Spark integrations, healthcare demo. All tasks completed.
2. DataFusion governance + Delta-rs — real CEL→Expr compilation (replacing string pattern-matching POC), column masks in the physical plan, Delta Lake feature gate. All tasks completed.

Next likely directions

Based on the research in research/cedar-open-lakehouse.md and the current codebase:

• Catalog PDP integration — implement Cedar as an embedded PDP for Lakekeeper (Authorizer trait) or as a sidecar PDP for Polaris (replacing OPA).
• Portable Lakehouse schema — adopt the Lakehouse Cedar namespace from the research doc as the canonical schema, mapping actions across Polaris/Lakekeeper/Unity Catalog.
• ABAC and context-aware policies — extend the QueryIdentity / request context to carry MFA, network, purpose attributes for richer policy evaluation.
• Compile-to-grants — build a Cedar→SQL GRANT compiler for Unity Catalog integration.
• SMT-based policy analysis — integrate cedar-policy-symcc to prove invariants about the policy set before deployment.
• Batch authorization — leverage is_authorized_batch and targeted partial evaluation for list operations.