graphframes

Imported from graphframes/graphframes (AGENTS.md). Install upstream with npx skills add graphframes/graphframes. Copyright stays with the author.

AGENTS.md

This file provides guidance to AI agents when working with code in this repository.

🚨 Legacy Codebase Guidelines 🚨

CRITICAL: This is a mature, mission-critical codebase (10+ years old in some areas). Follow these rules strictly:

Regression Prevention

• NEVER introduce breaking changes without explicit approval
• NEVER modify existing public APIs unless specifically requested
• NEVER change existing test assertions or remove tests without approval
• NEVER modify CI/build configuration without explicit request
• ALWAYS run the complete test suite before committing changes
• ALWAYS test against all supported Spark versions

Code Change Philosophy

• Prefer addition over modification: Add new functionality alongside existing code
• Preserve existing behavior: If unsure, err on the side of maintaining status quo
• No "cowboy-style" refactoring: Avoid large-scale structural changes
• Conservative by default: Only make changes that directly address the specific request

When in Doubt

• Ask before changing: If a change seems risky, request explicit approval
• Start small: Make minimal changes to achieve the goal
• Maintain backward compatibility: Existing user code must continue to work
• Document reasoning: Explain why changes are necessary in commit messages

Build and Development Commands

Core Build System

• Build tool: sbt (Scala Build Tool) version 1.11.0
• Build project: build/sbt compile
• Run all tests: build/sbt test
• Run single test suite: build/sbt "testOnly *PregelSuite"
• Run single test: build/sbt "testOnly *PregelSuite -- -z 'test name pattern'"
• Check code formatting: build/sbt scalafmtCheckAll
• Apply code formatting: build/sbt scalafmtAll
• Check code style: build/sbt "scalafixAll --check"
• Generate documentation: build/sbt doc
• Run with coverage: build/sbt "coverage test coverageReport"

Multi-Spark Version Testing

The project supports multiple Spark versions. Use -Dspark.version=X.Y.Z to specify:

• build/sbt -Dspark.version=3.5.7 test
• build/sbt -Dspark.version=4.0.1 test
• build/sbt -Dspark.version=4.1.0 test

Pre-PR Checklist Commands

⚠️ MANDATORY: These commands MUST pass before any PR. Failures indicate potential regressions.

Before raising a pull request, run these commands to ensure code quality and compatibility:

1. Format code: build/sbt scalafmtAll
2. Check formatting: build/sbt scalafmtCheckAll
3. Check style/linting: build/sbt "scalafixAll --check"
4. Build documentation: build/sbt doc
5. Run tests with coverage: build/sbt "coverage test coverageReport"
6. Test against multiple Spark versions:
   • build/sbt -Dspark.version=3.5.7 test
   • build/sbt -Dspark.version=4.0.1 test
   • build/sbt -Dspark.version=4.1.0 test

Alternative using pre-commit: pre-commit run all-files (handles Python formatting: black, isort, flake8)

Python Components

• Python package location: /python/ directory
• Python tests: Located in /python/tests/

Architecture Overview

Core Structure

GraphFrames is a graph processing library built on Apache Spark DataFrames with three main components:

1. Core GraphFrame API (core/src/main/scala/org/graphframes/)

   • GraphFrame.scala: Main graph abstraction using DataFrames for vertices/edges
   • Provides graph algorithms, motif finding, and subgraph operations

2. Algorithm Library (core/src/main/scala/org/graphframes/lib/)

   • Standard graph algorithms: PageRank, ConnectedComponents, ShortestPaths, etc.
   • Pregel.scala: Implements Pregel-style bulk synchronous parallel processing
   • AggregateMessages.scala: Lower-level message-passing API

3. GraphX Compatibility Layer (graphx/src/main/scala/)

   • Provides GraphX-compatible APIs for migration
   • Bridges between DataFrame and RDD-based graph operations

Key Design Patterns

Spark Version Compatibility: Uses SparkShims pattern for version-specific implementations:

• core/src/main/scala-spark-3/: Spark 3.x specific code
• core/src/main/scala-spark-4/: Spark 4.x specific code
• Common interface in main scala directory

DataFrame-Centric Design: Unlike GraphX's RDD approach, GraphFrames uses DataFrames throughout:

• Vertices and edges are DataFrames with required schema (id column for vertices, src/dst for edges)
• Leverages Spark SQL optimizations and Catalyst query planner
• Allows mixing graph operations with relational queries

Algorithm Framework: Most algorithms follow this pattern:

• Extend base traits like Logging, WithLocalCheckpoints, WithIntermediateStorageLevel
• Use builder pattern for configuration (e.g., graph.pregel.withVertexColumn(...).sendMsgToDst(...).run())
• Support checkpointing and persistence for iterative algorithms

Performance Considerations

Pregel Optimizations: The Pregel implementation includes automatic optimizations:

• Detects when destination vertex state is unneeded and skips expensive joins
• Conditional partitioning (source-only vs source+destination)
• Automatic caching of intermediate results when beneficial

Memory Management:

• Uses configurable storage levels for persistence
• Automatic cleanup of intermediate DataFrames between iterations
• Checkpoint support for long-running iterative algorithms

Testing Structure

Test Organization

• Base test class: SparkFunSuite provides SparkContext setup
• Algorithm tests: Each algorithm has corresponding *Suite.scala in lib/ subdirectory
• Integration tests: GraphFrameSuite.scala for core API functionality
• Pattern matching: PatternSuite.scala for motif finding features

Running Specific Tests

• Algorithm-specific: build/sbt "testOnly *PageRankSuite"
• Core functionality: build/sbt "testOnly *GraphFrameSuite"
• Pregel framework: build/sbt "testOnly *PregelSuite"
• All lib tests: build/sbt "testOnly org.graphframes.lib.*"

Multi-Version Testing

Tests run against multiple Spark versions in CI. The build system automatically:

• Selects appropriate Scala versions based on Spark version
• Uses version-specific SparkShims implementations
• Validates compatibility across Spark 3.5+ and 4.0+

Multi-Language Support

Scala (Primary)

• Main implementation in core/src/main/scala/
• Uses Scala 2.12/2.13 depending on Spark version
• Follows Spark's coding conventions and patterns

Python

• Python bindings in python/graphframes/
• Wraps Scala API using Spark's Python gateway
• Separate test suite in python/tests/

Java

• Java examples in core/src/main/java/
• Uses Scala API through Java interop
• Follows JavaBean conventions where applicable

Development Workflow

Code Quality

• Formatting: scalafmt for Scala, black/isort/flake8 for Python
• Style checking: scalafix for Scala linting
• Pre-commit hooks: Available via pre-commit run all-files

Contribution Requirements

⚠️ Legacy Codebase Rules:

• NO breaking changes to existing APIs without explicit approval
• NO modification of existing test behavior or CI configuration
• NO large-scale refactoring or architectural changes
• ALL changes must maintain backward compatibility

Standard Requirements:

• All new algorithms must include comprehensive test coverage
• Performance-critical code should include benchmarks (benchmarks/ directory)
• Multi-version compatibility must be maintained across all supported Spark versions
• Documentation updates required for new features
• Zero test failures across all Spark versions before submitting PR

Release Process

The project uses automated releases through GitHub Actions:

• Scala artifacts published to Maven Central
• Python packages published to PyPI
• Documentation deployed to graphframes.io