trajectory-slicing-code

Imported from nguyenddat/trajectory-slicing-code (AGENTS.md). Install upstream with npx skills add nguyenddat/trajectory-slicing-code. Copyright stays with the author.

AGENTS.md

Project Overview

This repository contains research code for trajectory slicing and error-localization experiments on LLM multi-agent system traces.

The codebase is organized so that:

• src/data/ owns dataset-specific schemas and loaders.
• src/methods/ owns reusable research methods.
• src/metrics/ owns reusable evaluation metrics.
• src/experiments/ owns experiment-specific orchestration and analysis.
• src/core/ owns shared project configuration and registries.
• src/llm_package/ owns LLM-related prompts and supporting logic.
• src/utils/ contains small reusable utilities that do not belong to a more specific module.

Do not move reusable method implementations into experiment folders.

Python Runtime

Before using the Python runtime, run:

conda activate trajectory-slicing

This applies before running Python scripts, modules, tests, notebooks, or any Python-based tooling.

Engineering Rule

Follow YAGNI strictly: implement only what the current task requires.

Do not introduce speculative abstractions, generalized frameworks, new configuration layers, helper modules, registries, pipelines, or compatibility code unless the current task actually needs them.

Prefer the smallest change that cleanly solves the requested task.

Source Layout

src/core/

Contains shared project-level configuration and registries.

Examples:

• settings
• paths
• model configuration
• dataset or method registries when they are genuinely shared

Keep this package small. Do not place experiment-specific configuration here.

src/data/

Contains dataset-specific code.

Each dataset should have its own package, for example:

src/data/
├── aegis/
├── agent_error_bench/
├── agentrx/
├── clawbench_v2_trace/
├── mast/
├── telbench/
├── trace_elephant/
├── trail/
└── who_and_when/

A dataset package may contain:

• schema definitions
• loading logic
• dataset-specific normalization
• conversion into the project's common internal representation

src/data/load.py is the preferred common entry point for loading datasets when applicable.

Dataset-specific parsing logic must stay inside the corresponding dataset package.

Do not add dataset-specific conditions throughout methods or experiments if they can be handled inside the dataset loader.

src/methods/

Contains reusable research methods.

Current high-level groups include:

src/methods/
├── baselines/
├── slicing/
└── techniques/

baselines/

Contains baseline approaches used in experiments.

A baseline should be reusable across experiments and datasets whenever possible.

slicing/

Contains trajectory slicing or segmentation methods.

Examples include:

• fixed-step slicing
• fixed-token slicing
• soft/content-aware slicing
• TextTiling-style methods

A slicing method should focus on producing boundaries, slices, or segments from an input trajectory.

techniques/

Contains reusable lower-level techniques that support methods.

Do not place one-off experiment logic here.

src/metrics/

Contains reusable metrics.

Examples:

• performance metrics
• cost metrics

Metrics must not depend on a specific experiment directory.

Prefer metric functions that accept normalized predictions/results and return plain values or structured metric outputs.

src/experiments/

Experiments are organized by research question or scientific claim, not by implementation name.

Use the naming convention:

eXXX_<experiment_goal>

Examples:

e001_baseline_performance_drop
e002_baseline_with_hard_slicing

Each experiment should answer one concrete research question.

Examples:

• Does baseline performance drop as trajectory length increases?
• Does hard slicing recover baseline performance?
• Does method A improve over the baseline?
• How sensitive is a method to a specific parameter?

An experiment directory should contain only experiment-specific orchestration, configuration, and analysis.

Recommended structure:

src/experiments/eXXX_name/
├── config.py
├── run.py
└── analyze.py

Create only the files that the current experiment actually needs.

config.py

Contains configuration specific to the experiment, such as:

• datasets
• baselines
• methods
• parameter values
• output identifiers

Do not implement reusable algorithms in config.py.

run.py

Runs the expensive or state-producing part of the experiment.

Typical flow:

load data
-> apply method
-> call model if needed
-> collect predictions
-> save raw experiment results

Do not mix plotting or final scientific interpretation into run.py unless the task explicitly requires it.

analyze.py

Reads saved experiment outputs and computes:

• metrics
• aggregations
• comparisons
• tables
• figures

Analysis should be rerunnable without repeating expensive LLM inference whenever possible.

Experiment Design

Treat one experiment as one scientific question.

Do not organize experiments only by method name.

Prefer:

e001_baseline_performance_drop
e002_baseline_with_hard_slicing

over:

baseline/
hard_slicing/
texttiling/

Reusable implementations belong under src/methods/; experiments import and compose them.

Example:

src/methods/slicing/fixed_step.py
        ↑
        |
src/experiments/e002_baseline_with_hard_slicing/run.py

Do not duplicate a method implementation inside an experiment directory.

Data and Output Separation

Keep code, input data, and generated results separate.

Conceptually:

src/        -> source code
data/       -> raw or prepared input data
outputs/    -> generated experiment results

Do not store generated result files inside src/.

When an experiment writes results, prefer a structure such as:

outputs/experiments/<experiment_id>/
├── predictions/
├── metrics/
├── tables/
└── figures/

Create only the subdirectories required by the current task.

Dependency Direction

Prefer the following dependency direction:

experiments
    ↓
methods
    ↓
data / llm_package / shared utilities

experiments
    ↓
metrics

Lower-level reusable modules must not import experiment modules.

Avoid circular dependencies.

Implementation Guidelines

• Reuse existing project abstractions before creating new ones.
• Inspect nearby modules before introducing a new pattern.
• Keep dataset-specific logic in src/data/<dataset>/.
• Keep reusable algorithms in src/methods/.
• Keep reusable evaluation logic in src/metrics/.
• Keep experiment-only composition in src/experiments/.
• Avoid hidden global state.
• Prefer explicit configuration over hard-coded absolute paths.
• Use project settings for shared paths when they already exist.
• Preserve existing public interfaces unless the task requires changing them.
• Do not refactor unrelated code while completing a focused task.
• Do not add dependencies unless they are necessary for the current task.

Validation

After making changes:

1. Activate the environment with conda activate trajectory-slicing.
2. Run the narrowest relevant validation first.
3. Prefer testing the modified module or experiment instead of running unrelated workloads.
4. If the change affects an experiment, verify its imports and configuration before starting expensive inference.
5. Do not run costly LLM/API experiments unless they are required to validate the task.

When a full experiment is expensive, validate structure and local logic separately whenever possible.

Working Style for Agents

Before editing:

1. Read the relevant existing files.
2. Identify the smallest set of files that must change.
3. Reuse existing naming and code conventions.
4. Avoid speculative restructuring.

While editing:

1. Make focused changes.
2. Keep reusable code outside experiment folders.
3. Do not create unused abstractions.
4. Do not modify unrelated files.

After editing:

1. Run targeted validation.
2. Report which files changed.
3. Report what was validated.
4. Explicitly mention anything that was not run, especially expensive experiments or API calls.