profile-bee

Imported from zz85/profile-bee (AGENTS.md). Install upstream with npx skills add zz85/profile-bee. Copyright stays with the author.

Profile-Bee Developer Guide

For AI coding assistants. Read this before making changes.

What Is This?

An eBPF-based CPU profiler for Linux, written in Rust. Single binary (probee), no BCC/libbpf dependencies. Walks stacks in-kernel via frame pointers or DWARF unwind tables, outputs flamegraphs (TUI, SVG, HTML, JSON, web server).

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  profile-bee-ebpf (kernel)     profile-bee (userspace)          │
│  ─────────────────────────     ───────────────────────          │
│  perf_event fires              tokio async runtime              │
│    → collect_trace()             ← RingBuf polling              │
│    → FP or DWARF unwinding       → read COUNTS, STACK_TRACES,  │
│    → write to eBPF maps            STACKED_POINTERS maps        │
│    → submit StackInfo via        → symbolize (blazesym/gimli)   │
│      RING_BUF_STACKS            → output flamegraph             │
│                                                                 │
│  profile-bee-common (#![no_std])                                │
│  ────────────────────────────────                               │
│  Shared #[repr(C)] types: StackInfo, FramePointers,             │
│  UnwindEntry, ProcInfo, DwarfUnwindState, constants             │
└─────────────────────────────────────────────────────────────────┘

Crate Layout

Key Files — Read Order

Module Map — Userspace (profile-bee/)

bin/profile-bee.rs          CLI entry point, Opt (clap), main loop, output writers
src/
  lib.rs                    Re-exports; declares modules
  ebpf.rs                   EbpfProfiler: load/attach eBPF, map I/O, DWARF table loading
  dwarf_unwind.rs           DwarfUnwindManager: .eh_frame → UnwindEntry tables → eBPF shards
  kernel_layout.rs          preempt_count layout detection (BTF reader + kallsyms) for exec-context
  trace_handler.rs          TraceHandler: address → symbol resolution (blazesym), caching
  types.rs                  StackFrameInfo, FrameCount, StackInfoExt trait
  cache.rs                  ProcessCache, AddrCache, PointerStackFramesCache (LRU-style)
  html.rs                   HTML flamegraph output (Stack tree builder)
  spawn.rs                  SpawnProcess: child process lifecycle for --cmd
  probe_spec.rs             ProbeSpec enum: parse uprobe specs (exact/glob/regex/source/demangled)
  probe_resolver.rs         ProbeResolver: spec → ResolvedProbe via ELF .symtab/.dynsym scanning
  legacy/                   Legacy symbol resolution code (addr2line-based, being replaced by blazesym)

Module Map — eBPF (profile-bee-ebpf/)

src/main.rs                 Entry points: #[perf_event], #[kprobe], #[uprobe], etc.
src/lib.rs                  All implementation + map definitions:
                              collect_trace()              — main sampling path (FP + optional DWARF)
                              collect_trace_stackid_only() — tracepoint path (bpf_get_stackid only)
                              collect_trace_raw_syscall()  — raw_tracepoint sys_enter
                              collect_trace_raw_tp_with_task_regs() — raw TP with bpf_task_pt_regs
                              dwarf_unwind_one_frame()     — single DWARF frame step
                              dwarf_copy_stack_regs()      — legacy inline path (21 frames max)
                              dwarf_try_tail_call()        — tail-call path entry (165 frames max)
                              dwarf_unwind_step_impl()     — tail-call target (5 frames per call)
                              dwarf_finalize_stack()       — write completed stack to maps
                              handle_process_exit()        — PID exit detection
                              (register access is arch-neutral via RawRegs +
                              reg_ip/reg_sp/reg_fp, gated on bpf_target_arch;
                              build.rs emits that cfg — x86_64 + aarch64)

Data Flow — One Sample

1. Kernel perf_event fires → profile_cpu() [main.rs]
2. → collect_trace() [lib.rs]
   a. Read target_pid, skip_idle filters
   b. Get kernel stack via bpf_get_stackid → stack_traces map
   c. Get user stack:
      - FP mode: walk frame pointer chain via get_frame()
      - DWARF mode: dwarf_try_tail_call() → dwarf_unwind_step_impl() (×33 tail calls)
   d. Build StackInfo { tgid, pid, cpu, kstack_id, ustack_id, ... }
   e. Increment counts map (kernel-side aggregation)
   f. Submit StackInfo to RING_BUF_STACKS
3. Userspace tokio task polls RingBuf
4. → TraceHandler.format_stack_trace()
   a. Read kernel frames from stack_traces map
   b. Read user frames from stacked_pointers map (FP/DWARF) or stack_traces map
   c. Symbolize via blazesym (with AddrCache)
   d. Produce Vec<StackFrameInfo> + count
5. → Output: collapse format → inferno (SVG), or HTML, JSON, TUI, web server

CLI Structure (Opt in bin/profile-bee.rs)

Key flags — all optional, sensible defaults:

eBPF Programs

All in profile-bee-ebpf/src/main.rs, implementations in lib.rs:

Why tail-call only works for perf_event: eBPF tail calls require matching program types. Only profile_cpu and dwarf_unwind_step are both perf_event, so kprobe/uprobe/tracepoint programs fall back to the 21-frame inline DWARF path.

Shared eBPF Maps

DWARF Unwinding

Two paths:

Tail-call path (primary, up to 165 frames, perf_event only):

collect_trace → dwarf_try_tail_call → PROG_ARRAY.tail_call(0)
  → dwarf_unwind_step_impl (5 frames) → tail_call(0) → ... (up to 33×)
  → dwarf_finalize_stack → writes to STORAGE + COUNTS + RING_BUF

Legacy inline path (fallback, up to 21 frames):

collect_trace → dwarf_copy_stack_regs (flat loop of 21)

Unwind table generation (userspace, DwarfUnwindManager):

1. Read /proc/[pid]/maps → find executable mappings
2. Parse .eh_frame from each ELF (via gimli)
3. Classify CFA rules into compact UnwindEntry (12 bytes each)
4. Load into sharded eBPF Array maps (shard_0..shard_7)
5. Background thread polls for dlopen-loaded libraries (~1s)

Build Commands

# Build eBPF (requires nightly Rust + bpf-linker)
cargo xtask build-ebpf --release

# Build userspace (uses fresh eBPF build if available, otherwise prebuilt in ebpf-bin/)
cargo build --release

# Build both and run with sudo
cargo xtask run -- --tui

# Build without TUI
cargo build --release --no-default-features

The eBPF binary is embedded at compile time via build.rs. A prebuilt copy at profile-bee/ebpf-bin/profile-bee.bpf.o is used when no fresh build exists (enables cargo install on stable Rust).

Testing

# Unit/integration tests (no sudo needed)
cargo test --workspace

# E2E tests (requires sudo — eBPF needs CAP_SYS_ADMIN)
bash tests/build_fixtures.sh          # compile C/Rust test binaries
sudo tests/run_e2e.sh --verbose       # ~14 tests: FP, DWARF, deep stacks, shared libs, PIE, Rust

# Run specific e2e tests
sudo tests/run_e2e.sh --filter dwarf

E2E test structure: tests/run_e2e.sh runs probee --collapse against fixture binaries in tests/fixtures/bin/, then validates output against expected patterns in tests/output/*.collapse. Each test checks that expected function names appear in the stack traces.

CI: GitHub Actions (.github/workflows/rust.yml) runs fmt, clippy, build+test, and e2e tests. The eBPF crate is built separately with cargo +nightly build -Z build-std=core --target bpfel-unknown-none.

How To: Common Tasks

Add a new CLI flag

1. Add field to Opt struct in bin/profile-bee.rs (uses clap derive)
2. Wire it through in the main function body (same file)

Add a new output format

1. Add the writer function (see html.rs for pattern)
2. Add CLI flag to Opt in bin/profile-bee.rs
3. Call writer from the output section of main()

Add a new eBPF program type

1. Define entry point in profile-bee-ebpf/src/main.rs with appropriate macro
2. Implement (or call shared impl) in profile-bee-ebpf/src/lib.rs
3. Load and attach from setup_ebpf_profiler() in profile-bee/src/ebpf.rs
4. Rebuild eBPF: cargo xtask build-ebpf --release

Add a new shared type between eBPF and userspace

1. Define in profile-bee-common/src/lib.rs
2. Must be #[repr(C)], Copy, Clone with explicit padding
3. No String, Vec, HashMap — it's #![no_std]

Add a new uprobe spec syntax

1. Extend ProbeSpec enum in probe_spec.rs
2. Update ProbeSpec::parse() in the same file
3. Handle the new variant in ProbeResolver methods in probe_resolver.rs

Modify DWARF unwinding

• Userspace table generation: DwarfUnwindManager::scan_and_update() in dwarf_unwind.rs
• eBPF-side unwinding: dwarf_unwind_one_frame() in profile-bee-ebpf/src/lib.rs
• Shared data structures: UnwindEntry, ProcInfo, DwarfUnwindState in profile-bee-common/src/lib.rs
• After eBPF changes: cargo xtask build-ebpf --release && cargo build --release

Critical Pitfalls

BPF Verifier

• All loops MUST be bounded (for _ in 0..CONST). No while, no dynamic bounds.
• Map accesses must be bounds-checked. The verifier tracks these statically.
• shard_lookup() uses ArrayOfMaps::get_value() (from the Brskt/aya hashmapofmaps-new branch) which fuses the outer and inner bpf_map_lookup_elem into a single call without intermediate struct indirection. Using the two-step approach (get() then Array::get()) causes verifier state explosion (~10s vs ~400ms). The fused get_value() is the required pattern.
• The eBPF crate uses extreme optimization (opt-level=3, lto, codegen-units=1) even in debug — required for verifier to accept the code.
• Adding code to collect_trace can push it over the verifier instruction limit. The DWARF inline path (21 iterations × 8 mappings × 16 binary search steps) is near the edge.

Two Separate Workspaces

• profile-bee-ebpf/ is NOT in the root workspace. cargo build at root only builds userspace.
• Always run cargo xtask build-ebpf before cargo build when changing eBPF code.
• The eBPF crate requires nightly Rust and bpf-linker: cargo install bpf-linker.

#![no_std] in Common Crate

• profile-bee-common is #![no_std]. No String, Vec, HashMap, println!.
• All structs must be #[repr(C)], Copy, Clone with deterministic layout.
• Explicit padding fields (_pad, _pad2) ensure layout matches across eBPF and userspace.

Binary Names

• probee and pbee are the same binary (both [[bin]] entries in profile-bee/Cargo.toml, same source bin/profile-bee.rs).

--pid vs --cmd

• --pid with stream_mode 2 (default) can miss samples for already-running processes.
• --cmd (or -- <command>) handles the process lifecycle and works reliably.
• E2E tests use --cmd.

Frame Pointers Everywhere

• .cargo/config.toml sets force-frame-pointers=yes globally. All workspace crates are built with frame pointers for self-profiling support.

Proc-Maps Library

• The codebase uses procfs for all /proc/pid/maps parsing.

Known Limitations (DWARF)

Key Dependencies

Docs Reference