archive

Imported from jixiexiaoge/enhancedopenpilot (docs/archive/AGENTS.md). Install upstream with npx skills add jixiexiaoge/enhancedopenpilot --skill archive. Copyright stays with the author.

EnhancedOpenPilot (EP) Hardware Architecture Guide

Born from dragonpilot | RK3588 with direct socketd CAN | External TC275 safety module | sunnypilot/FrogPilot feature porting

EnhancedOpenPilot represents a fundamental architectural evolution from traditional comma openpilot, specifically designed around RK3588 hardware with direct CAN communication and external safety isolation. This guide covers the unique hardware architecture and optimization strategies for EP development.

EP Hardware Architecture Overview

Core Architectural Innovations:

• Direct socketd CAN communication - No panda intermediary, 15-20ms latency reduction
• External TC275 safety module - Safety isolated at ~/tc275_freertos, ASIL-D capable
• RK3588 NPU acceleration - 6 TOPS across 3 cores, currently using 1 core
• Dragonpilot heritage - Foundation codebase with sunnypilot/FrogPilot feature integration
• Multi-camera fusion pipeline - Stereo depth + side cameras for 360° awareness

Hardware Specifications:

• SoC: RK3588 (4x A76 @ 2.4GHz + 4x A55 @ 1.8GHz)
• Memory: 8GB LPDDR4X with 25.6 GB/s bandwidth
• Acceleration: 3x NPU cores (6 TOPS), Mali-G610 GPU, RGA 2D accelerator
• CAN: Dual native CAN controllers with direct kernel socket access
• Safety: External Infineon TC275 microcontroller with FreeRTOS

Technology Stack

Core Technologies

• Python 3.11-3.12: Primary development language
• C++17: Performance-critical components and safety code
• SCons: Build system with cross-platform support
• Cap'n Proto: High-performance messaging framework
• Qt5: User interface framework
• ONNX: Neural network model format

Key Dependencies

• Computer Vision: OpenCV, libyuv for image processing
• Machine Learning: ONNX Runtime, numpy for model inference
• Controls: acados for MPC optimization, casadi for symbolic computation
• Hardware: libusb, spidev for device communication
• Networking: zmq, aiohttp for inter-process communication

Architecture

Directory Structure

EnhancedOpenPilot/
├── cereal/           # Messaging definitions and generated code
├── common/           # Shared utilities and libraries
├── selfdrive/        # Core driving functionality
│   ├── modeld/      # Neural network perception
│   ├── pathd/       # Path planning and lateral control
│   ├── controls/    # Longitudinal and lateral control
│   ├── locationd/   # Localization and mapping
│   ├── sensord/     # Sensor data processing
│   └── ui/          # User interface
├── system/          # System services and hardware abstraction
│   ├── manager/     # Process orchestration
│   ├── loggerd/     # Data logging
│   ├── camerad/     # Camera management
│   └── hardware/    # Hardware-specific code
├── opendbc/         # CAN database and car interfaces
├── tools/           # Development and debugging tools
└── third_party/     # External dependencies

Key Components

EP-Specific Architecture

• socketd: Direct SocketCAN daemon replacing pandad
• tc275_freertos: External safety module isolation
• modeld: RK3588 NPU-accelerated perception
• Multi-camera fusion: Stereo + side camera integration

Perception Pipeline

• modeld: Forward-facing camera neural network processing (RK3588 NPU)
• stereod: Stereo depth perception (EP2+, NPU core 1)
• leftd/rightd: Side camera perception (EP2 Pro+, NPU core 2)
• trackd: Multi-object tracking and fusion (360° awareness)

Control Systems

• pathd: Path planning with Smart Offset Control (SOC)
• controls: Longitudinal and lateral MPC controllers
• DEC: Dynamic Experimental Controller for adaptive ACC
• RED: Road Edge Detection for safety

System Services

• manager: Process lifecycle management
• loggerd: Data recording and upload
• camerad: Camera capture and preprocessing
• sensord: IMU and sensor fusion

Build System

Build Commands

# Full build
scons -u -j$(nproc)

# Minimal build (no tests/tools)
scons -u --minimal

# Build with specific options
scons -u --asan          # Address sanitizer
scons -u --ubsan         # Undefined behavior sanitizer
scons -u --coverage      # Test coverage
scons -u --clazy         # Static analysis

Platform Support

• aarch64: Native RK3588 builds with direct CAN support
• x86_64: Development builds (with simulated CAN)
• RK3588 variants: LubanCat, generic RK3588/RK3588S boards

EP Hardware Architecture

• Direct CAN: socketd replaces pandad for <1ms latency
• External safety: TC275 module at ~/tc275_freertos
• NPU utilization: 3-core RK3588 acceleration
• Multi-camera: Native MIPI CSI-2 support

RK3588 Hardware-Specific Optimizations

• NPU acceleration: 6 TOPS across 3 cores (currently using 1 core)
• Direct CAN socketd: Native SocketCAN integration, <1ms latency
• DMA-BUF: Zero-copy GPU→NPU data transfer
• RGA accelerator: Hardware image preprocessing
• External safety module: TC275-based isolation at ~/tc275_freertos
• Memory optimization: DDR frequency scaling (2.112GHz target)

Testing Strategy

Test Categories

# Run all tests
pytest

# Skip slow tests
pytest -m "not slow"

# Device-specific tests
pytest -m tici

# Parallel testing
pytest -n auto

Test Types

• Unit tests: Component-level testing
• Integration tests: Service interaction testing
• Process replay: Route-based regression testing
• Hardware-in-the-loop: Real hardware validation
• Performance tests: Latency and throughput validation

Continuous Integration

• Automated testing on every commit
• Multi-platform build verification
• Safety-critical code analysis
• Route replay validation

Development Workflow

Setup

# 1. Clone repository
git clone https://github.com/commaai/openpilot.git
cd openpilot

# 2. Run setup script
tools/op.sh setup

# 3. Activate Python environment
source .venv/bin/activate

# 4. Build
scons -u -j$(nproc)

Development Tools

• cabana: CAN message analysis and plotting
• replay: Route replay and debugging
• plotjuggler: Log visualization
• joystick: Manual vehicle control
• sim: Simulation environment

Code Quality

• ruff: Python linting and formatting
• mypy: Type checking
• codespell: Spell checking
• pre-commit: Git hooks for quality gates

Safety Considerations

EP Safety Architecture

1. External safety module: TC275-based isolation at ~/tc275_freertos
2. Direct CAN monitoring: Hardware-level message validation
3. Independent watchdog: Separate from main system
4. Fail-safe behavior: Graceful degradation on communication loss

Safety Requirements

1. Driver override: Brake pedal or cancel button immediately retakes control
2. Trajectory limits: Actuator constraints prevent unsafe maneuvers
3. ISO 26262 compliance: Following automotive safety standards
4. MISRA C:2012: Strict coding guidelines for safety-critical code

TC275 Safety Module Responsibilities

• Monitor CAN message timing and validity
• Implement independent safety checks
• Provide emergency stop capability
• Report safety status to main system
• Maintain independent watchdog timer

Validation Process

• Software-in-the-loop testing
• Hardware-in-the-loop validation with TC275
• In-vehicle testing before releases
• External safety module verification
• Continuous monitoring and data collection

Product Tiers

EP Hardware-Specific Enhancements

Direct CAN Architecture (socketd)

• Latency reduction: 15-20ms improvement vs panda
• Hardware integration: Native RK3588 CAN controllers
• Real-time performance: <1ms message round-trip
• Safety isolation: External TC275 module validation

RK3588 NPU Acceleration

• 3-core utilization: Currently 1 core, 2 cores available
• Model optimization: INT8 quantization, batch processing
• Memory bandwidth: 25.6 GB/s DDR optimization
• Pipeline parallelism: Overlap preprocessing/inference

Key Enhancements

Enhanced Controllers

• DEC: Dynamic Experimental Controller with Kalman filtering
• RED: Road Edge Detection for lane change safety
• SOC: Smart Offset Control for lateral positioning

Multi-Camera Fusion (RK3588-Optimized)

• Stereo depth estimation: IMX415 dual camera setup
• Side camera integration: 360° situational awareness (EP2 Pro+)
• NPU distribution: YOLOv8 + PP-LiteSeg across cores
• Zero-copy pipeline: DMA-BUF camera data sharing

Dragonpilot Feature Porting

• sunnypilot integration: DEC, SOC, RED controllers
• FrogPilot features: Traffic light, stop sign detection
• Direct CAN control: No panda firmware dependency
• External safety validation: TC275 module verification

Hardware Acceleration (RK3588-Specific)

• 3-core NPU utilization: 6 TOPS distributed processing
• DMA-BUF zero-copy: GPU→NPU direct transfer
• Memory optimization: DDR frequency scaling, cache coherency
• RGA preprocessing: Hardware image format conversion
• CPU affinity: A55 for real-time, A76 for compute

Contributing Guidelines

Pull Request Requirements

• Clear stated purpose
• Every line contributes to the goal
• Verification and testing documentation
• Performance benchmarks for optimizations
• CI test passage

What Gets Merged

• Simple, well-tested bug fixes
• Car model/brand ports
• Code cleanup and refactoring
• Documentation improvements

What Doesn't Get Merged

• Style changes without functional improvement
• Large PRs (>500 lines)
• UI design changes
• New features without clear justification
• Code with negative expected value

Deployment

EP Hardware Requirements

• RK3588-based device: LubanCat or generic RK3588 board
• External safety module: TC275 at ~/tc275_freertos
• Direct CAN interface: Native SocketCAN support
• Multi-camera setup: MIPI CSI-2 camera support

Installation

# EP installation with RK3588 optimizations
bash <(curl -fsSL <enhanced-openpilot-url>)

# Verify hardware detection
python3 -c "from system.hardware import HARDWARE; print(HARDWARE.get_device_type())"

# Check CAN interfaces
python3 -c "from system.hardware import HARDWARE; print(HARDWARE.get_can_interfaces())"

Branches

• release3: Stable release (openpilot.comma.ai)
• release3-staging: Pre-release testing
• nightly: Development builds
• master: Main development branch

RK3588 Development & Optimization

Hardware Detection and Configuration

# RK3588 hardware detection
from system.hardware import HARDWARE
print(f"Device: {HARDWARE.device_variant}")
print(f"Capabilities: {HARDWARE.hardware_capabilities}")
print(f"CAN interfaces: {HARDWARE.get_can_interfaces()}")

CPU Core Affinity Strategy

# A55 cores (0-3): Real-time tasks (CAN, sensors)
# A76 cores (4-7): Compute tasks (models, planning)

taskset -cp 0-3 $SOCKETD_PID    # CAN daemon on A55
taskset -cp 4-7 $MODELD_PID     # Neural networks on A76

NPU Optimization

# RK3588 NPU core assignment
# Core 0: modeld (YOLOv8 + PP-LiteSeg)
# Core 1: stereod (depth estimation)  
# Core 2: driverd (driver monitoring)

export RKNN_DEVICE_CORE_MASK=0b111  # Use all 3 cores
export RKNN_PERF_TIMEOUT=0           # No timeout for profiling

Memory Optimization

# DDR frequency scaling for bandwidth
sudo echo userspace > /sys/class/devfreq/dmc/governor
sudo echo 2112000000 > /sys/class/devfreq/dmc/userspace/set_freq

# Memory management for 4GB systems
echo 5 > /proc/sys/vm/swappiness
echo 10 > /proc/sys/vm/dirty_ratio

Direct CAN Development

# SocketCAN testing and debugging
from selfdrive.socketd.socketd import candevice, cansend

# Create direct CAN interface
device = candevice(interface='can0', bus=0)
device.send(0x123, b'\x01\x02\x03\x04\x05\x06\x07\x08')

# Monitor CAN traffic
messages = device.recv()
for addr, data, bus in messages:
    print(f"CAN {addr:x}: {data.hex()} on bus {bus}")

Performance Profiling

# RK3588-specific profiling
sudo perf stat -e cache-misses,cache-references -p $PID
sudo perf record -g -C 0-7 ./your_application

# NPU utilization monitoring
cat /sys/class/devfreq/fdab0000.npu/cur_freq
cat /sys/kernel/debug/rknpu/load

External Safety Module Integration

# TC275 safety module communication
# Located at ~/tc275_freertos
cd ~/tc275_freertos
make build  # Build safety firmware
make flash  # Flash to TC275 device

# Monitor safety status via CAN
candump can0 | grep 0x7XX  # Safety module CAN IDs

Security Notes

This is alpha quality software for research purposes only. Users are responsible for:

• Complying with local laws and regulations
• Maintaining driver alertness at all times
• Understanding system limitations
• No warranty is expressed or implied

For detailed safety information, see docs/SAFETY.md.