bruin

Imported from bruin-data/bruin (AGENTS.md). Install upstream with npx skills add bruin-data/bruin. Copyright stays with the author.

AGENTS.md/CLAUDE.md - AI Agent Contribution Guide

This document gives AI agents the project-specific operating guidance needed to work safely in Bruin.

Before You Finish: Checks

You MUST run these commands before completing any task that modifies application code:

1. Format the code: Run make format in the project root. Check git diff afterward — if there are formatting changes, stage and include them in your work.
2. Run the tests: Run make test in the project root. If any tests fail, fix the issues before finishing.

Do not consider an application-code task complete until both checks pass. Use the /format-fix and /test commands if needed.

For changes that only touch non-application files, such as documentation, GitHub Actions workflows, agent instructions, repository metadata, or other configuration that does not affect the Bruin binary/runtime behavior, do not run the full make format / make test suite by default. Instead, run the smallest relevant validation available for the changed files, such as YAML syntax checks, Markdown checks, workflow review, or git diff inspection, and clearly report what was and was not run.

Table of Contents

1. Project Overview
2. Architecture & Core Concepts
3. Development Environment
4. Build System
5. CLI Source of Truth
6. Codebase Organization
7. Testing Strategy
8. Contributing Guidelines
9. Common Development Tasks

Project Overview

Bruin is a CLI-first data framework for ingestion, SQL/Python/R transformations, data quality, materialization, lineage, and pipeline execution across many data platforms.

Most behavior is configured in version-controlled text files: pipeline.yml, asset files, templates, docs, and connection/config files. When changing behavior, prefer existing package patterns and keep docs/tests aligned with user-visible changes.

Core Features

• Data Ingestion: Using ingestr and Python scripts
• Transformations: SQL, Python, and R on multiple platforms
• Data Quality: Built-in quality checks and validations
• Materializations: Table/view materializations and incremental tables
• Python Isolation: Isolated Python environments via uv
• Templating: Jinja templating for reusable code
• Lineage: Dependency visualization and tracking
• Multi-platform Execution: Runs locally, on EC2, or GitHub Actions
• Secrets Management: Environment variable injection
• VS Code Extension: Enhanced developer experience

Architecture & Core Concepts

Assets

Anything that carries value derived from data:

• Tables/views in databases
• Files in S3/GCS
• Machine learning models
• Documents (Excel, Google Sheets, Notion, etc.)

Assets consist of:

• Definition: Metadata enabling Bruin to understand the asset
• Content: The actual query/logic that creates the asset

Pipelines

Groups of assets executed together in dependency order. Structure:

my-pipeline/
├─ pipeline.yml
└─ assets/
   ├─ asset1.sql
   └─ asset2.py

Pipeline Runs

Execution instances containing one or more asset instances with specific configuration and timing.

Development Environment

Prerequisites

• Go: Use the version declared in go.mod
• Python: For Python asset development and formatting
• CGO: Required for DuckDB support
• Git: For version control and repository detection

Dependencies

The project uses extensive Go dependencies including:

• CLI framework: urfave/cli
• Database drivers: BigQuery, Snowflake, PostgreSQL, MySQL, DuckDB, etc.
• Cloud SDKs: AWS, GCP
• Templating: Jinja via Gonja
• Testing: Testify

Build System

The Makefile provides comprehensive build and development targets:

Core Targets

Build Targets

make build          # Build with DuckDB support (CGO_ENABLED=1)
make build-no-duckdb # Build without DuckDB (CGO_ENABLED=0)

Development Targets

make deps           # Install dependencies and tools
make clean          # Remove build artifacts
make format         # Format Go/Python and run fast changed-package lint
make lint           # Run fast lint on changed packages in every Go module
make lint-full      # Run all Go linters across the primary Go modules
make tools-update   # Update development tools

Testing Targets

make test                      # Run fast unit tests
make test-full                 # Run unit tests with race detection
make test-unit                 # Run unit tests specifically
make integration-test          # Full integration tests with ingestr
make integration-test-light    # Light integration tests without ingestr
make integration-test-cloud    # Cloud-specific integration tests

Development Utilities

make lint-python                     # Format and lint Python code
make refresh-integration-expectations # Update integration test expectations

Build Configuration

• Build metadata: The Makefile injects build metadata via linker flags
• Telemetry: Controlled via TELEMETRY_KEY and TELEMETRY_OPTOUT environment variables
• Tags: Uses no_duckdb_arrow for standard builds, bruin_no_duckdb for no-DuckDB builds

CLI Source of Truth

Do not treat this guide as a current command inventory. If a task depends on commands, flags, hidden subcommands, or runtime help text, verify against the source of truth:

1. Check main.go for top-level command registration.
2. Check cmd/*.go and cmd/mcp/*.go for command definitions, flags, and action wiring.
3. After make build, use ./bin/bruin --help and ./bin/bruin <command> --help to confirm runtime behavior.
4. Check docs/commands/ when changing user-facing command behavior, and update docs when behavior changes.

When adding or changing a command, update the command implementation, tests, and user-facing docs together.

Codebase Organization

Package Structure (pkg/)

The codebase is organized into focused packages:

Core Packages

• pipeline/: Pipeline parsing, execution, and management
• config/: Configuration file handling (.bruin.yml)
• connection/: Database connection management
• executor/: Asset execution engine
• lineage/: Dependency tracking and visualization
• query/: Query execution and management

Data Platform Packages

Each supported platform has its own package:

• Database platforms: bigquery/, snowflake/, postgres/, mysql/, duckdb/, clickhouse/, athena/, mssql/, databricks/, oracle/, sqlite/, trino/, synapse/, hana/, spanner/
• Cloud storage: s3/, gcs/
• Ingestion sources: 50+ packages for different data sources (e.g., shopify/, hubspot/, salesforce/, stripe/, etc.)

Utility Packages

• jinja/: Template processing
• python/: Python asset execution
• lint/: Code linting and validation
• diff/: Data comparison functionality
• path/: File system utilities
• git/: Git repository operations
• telemetry/: Usage analytics
• secrets/: Secret management
• logger/: Logging utilities

Command Implementation (cmd/)

Each CLI command is implemented in its own file:

• Command structure definition
• Flag parsing and validation
• Business logic delegation to appropriate packages
• Error handling and output formatting

Testing Strategy

Test Types

Unit Tests

• Location: Throughout pkg/ packages with *_test.go files
• Execution: make test-unit
• Coverage: Fast local tests by default; make test-full adds race detection
• Scope: Excludes cloud integration tests

Use narrow test loops while developing, then run the required full checks before finishing:

# Target one package or test while iterating
go test -tags="no_duckdb_arrow" ./pkg/foo -run TestName
go test -tags="no_duckdb_arrow" ./cmd/... ./pkg/...

# Required final unit-test command for application-code changes
make test

Integration Tests

• Light Integration: make integration-test-light (excludes ingestr)
• Full Integration: make integration-test (includes ingestr)
• Cloud Integration: make integration-test-cloud (cloud platforms)

Use make integration-test-light for changes that affect parsing, command workflows, local pipeline execution, or integration-test fixtures. Use full integration tests when touching ingestr behavior. Cloud integration tests require local cloud credentials/config and may skip tests when matching connections are absent.

Test Data

• Location: integration-tests/test-pipelines/
• Coverage: Parse tests, lineage tests, execution tests
• Expectations: JSON files with expected outputs
• Refresh: make refresh-integration-expectations updates expectations

Test Patterns

• Mock databases with existing SQL mock helpers
• Mock PostgreSQL with existing pgx mock patterns
• Use existing concurrency helpers for parallel work
• Use the existing file system abstraction patterns in packages that already use them

Contributing Guidelines

Code Style & Formatting

Go Code

Tools automatically installed and run via make format:

• gci: Import organization
• gofumpt: Stricter Go formatting
• golangci-lint: Fast changed-package linting; make lint-full runs the comprehensive suite
• govet: Enabled through golangci-lint

Python Code

Tools run via make lint-python:

• ruff format: Code formatting
• ruff check --fix: Linting with auto-fixes

Secrets, Credentials, and Generated Files

• Do not commit local credentials, tokens, keys, or personal environment files.
• Treat .bruin.yml, .bruin.cloud.yml, cloud integration configs, and local connection files as sensitive unless they are clearly committed examples.
• If a task requires cloud integration config, use local untracked files and document what was needed.
• make refresh-integration-expectations rewrites JSON expectations. Inspect the diff carefully and include only intentional expectation changes.
• Do not commit build outputs, virtual environments, local caches, logs, or generated binaries unless the repository already tracks that exact artifact and the change is intentional.

Development Workflow

1. Setup: make deps to install tools and dependencies
2. Development: Edit code with VS Code extension for enhanced experience
3. Formatting: make format before committing
4. Testing: make test for unit tests, integration tests as appropriate
5. Building: make build to verify compilation

Adding New Data Platforms

1. Create package: pkg/newplatform/
2. Implement interfaces: Connection, query execution, schema introspection
3. Add CLI command: Register in main command list
4. Add tests: Unit and integration tests
5. Update documentation: Add to supported platforms list

Adding New CLI Commands

1. Create command file: cmd/newcommand.go
2. Implement command structure: Using cli.Command pattern
3. Add business logic: In appropriate pkg/ package
4. Register command: In main.go commands slice
5. Add tests: Command and business logic tests

Common Development Tasks

Running Locally

# Basic build and run
make build
./bin/bruin --help

# Development mode with debug
make build
./bin/bruin --debug [command]

Adding New Asset Types

1. Define asset type in pkg/pipeline/asset.go
2. Implement execution logic in pkg/executor/
3. Add parsing logic if needed
4. Update lineage detection if applicable
5. Add tests and integration tests

Debugging Integration Tests

# Run specific test pipeline
(cd integration-tests && ../bin/bruin run test-pipelines/your-test)

# Refresh expectations after changes from the repository root
make refresh-integration-expectations

Working with Templates

# Test template rendering
./bin/bruin render path/to/template.sql

# Test complete pipeline parsing
./bin/bruin internal parse-pipeline path/to/pipeline

# Regenerate the docs pages that mirror a template README
make sync-template-docs

Most templates have a docs page under docs/getting-started/templates-docs/ that is written by hand and diverges from the template's own README. For the templates listed in SYNCED in scripts/sync_template_docs.py, the docs page is instead generated from the README: edit templates/<name>/README.md, run make sync-template-docs, and commit both. Those pages open with a <!-- Generated from … --> comment; make test fails if one has drifted.

Database Connection Testing

# List connections
./bin/bruin connections list

# Test connection
./bin/bruin connections test --name connection-name

# Add new connection
./bin/bruin connections add

This guide provides the foundational knowledge needed to contribute effectively to the Bruin project. For specific implementation details, refer to the extensive documentation in the docs/ directory and examine existing patterns in the codebase.