Imported from Winston-Gu/IMPACT (
AGENTS.md). Install upstream withnpx skills add Winston-Gu/IMPACT. Copyright stays with the author.
AGENTS.md
This file gives coding agents project-specific context for IMPACT. The public user-facing instructions live in README.md; keep this file focused on implementation guidance, project boundaries, and release hygiene.
Project Overview
IMPACT studies internal-model predictive control for forceful robotic manipulation. The code trains an image-conditioned diffusion policy from demonstrations, then deploys the policy with robot controllers that can compensate for forceful or out-of-distribution interaction conditions such as lifting heavier objects.
The current Python package name is impact. Do not reintroduce deprecated package or project names in imports, configs, docs, ROS node names, Slurm scripts, or examples.
Repository Layout
impact/: Python training and inference package.impact/train.py: Hydra entrypoint for policy training.impact/config/: Hydra configs for dataset, logging, dataloaders, normalizers, workspace, policy, and optimizer.impact/dataset/: exported demonstration dataset loaders and replay-buffer utilities.impact/model/: diffusion, vision encoder, and tensor utility modules.impact/policy/: diffusion image policy implementation.impact/workspace/: training loop, checkpointing, logging, validation, and runner orchestration.impact/deploy/: ROS-facing policy loading and deployment helpers.
config/: runtime configs outside Hydra, mainly deployment and post-processing YAML files.scripts/: dataset checks, post-processing, viewers, benchmark helpers, Joy-Con pairing helpers, and Slurm helper.ros_ws/src/: ROS 2 Humble workspace packages.system_bringup: real Franka bringup, RealSense launch, teleoperation, dataset recording, gripper bridge, pose-error broadcasting.sim_bringup: MuJoCo data collection, simulation deployment, benchmarking, video recording, fake gripper, scripted pick-place episodes.predictive_controller: C++ controllers, including Cartesian impedance and predictive controller variants.mujoco_sim: MuJoCo hardware interface and rendering integration.joycon_teleop: Joy-Con input library and teleoperation nodes.cameras: RealSense camera node.
ros_ws/vendor.repos: external Franka ROS dependencies imported intoros_ws/vendor/.assets/: public README assets only.
Development Rules
- Prefer existing module boundaries. Training code belongs under
impact/; robot/simulation runtime code belongs under the relevant ROS package. - Keep examples and defaults repo-relative where possible. User-specific values must be launch/config parameters or documented README settings.
- Do not add datasets, checkpoints, logs, ROS build products, vendored dependency checkouts, virtual environments, or generated caches to git.
- Do not use private absolute paths. Acceptable user-set paths include documented parameters such as
dataset.dataset_path,checkpoint_path,output_root,MUJOCO_DIR, andPROJECT_ROOT. - Keep Python package imports rooted at
impact. - Keep Hydra
_target_values synchronized with actual import paths. - Preserve robot safety. Any change to controllers, deployment loops, action scaling, gripper commands, reset logic, or collision behavior should be treated as high risk and tested in simulation or fake hardware first.
- Be conservative with controller gains, frame names, topic remaps, and action interpretation. Small changes can cause unsafe robot motion.
- Do not change licensing metadata casually. The top-level repo license is Apache-2.0; package-level exceptions are declared in ROS package manifests and source comments.
Common Workflows
Python environment:
uv sync
source .venv/bin/activate
Training entrypoint:
HYDRA_FULL_ERROR=1 uv run python -m impact.train \
experiment_name=DemoSim \
training.device=cuda:0 \
dataset.dataset_path=datasets/exported/DemoSim/DemoSim_2.pkl
Dataset post-processing:
uv run python scripts/postprocess/align_timesteps.py --dataset-name DemoSim --config MJ_side_front
uv run python scripts/postprocess/export_trajectories.py --dataset-name DemoSim --config MJ_side_front --stream
ROS workspace build on Ubuntu 22.04 with ROS 2 Humble:
source /opt/ros/humble/setup.zsh
cd ros_ws
mkdir -p vendor
vcs import . < vendor.repos
rosdep install --from-paths src vendor --ignore-src -r -y
colcon build --symlink-install
Verification Checklist
Run the checks that match the files you changed:
uv lock --check
uv run python -m compileall impact scripts
uv run python -c "import impact; import impact.train; print(impact.__name__)"
git diff --check
For ROS changes, additionally run colcon build --symlink-install from ros_ws in a ROS Humble environment. For controller or deployment changes, validate in simulation before hardware.
Before release or PR handoff, scan for stale project names, private absolute paths, secrets, placeholder comments, and generated artifacts. Keep the scan pattern outside committed docs so the docs do not match their own release check.
find . -name '.DS_Store' -o -name '__pycache__' -o -name '*.pyc' -o -name '.venv'
TODO-style comments should not describe release blockers. Generated files found by the find command should be removed unless they are intentionally ignored local artifacts.
User-specific Configuration
These values are expected to vary by user and should remain configurable:
dataset.dataset_pathfor training.checkpoint_pathfor deployment and benchmarking.output_rootfor dataset recorders.MUJOCO_DIRandLD_LIBRARY_PATH.- CUDA device in
training.deviceandconfig/deploy/*.yaml. - Franka robot IP and namespace in
ros_ws/src/system_bringup/config/franka.config.yaml. - RealSense serial numbers in
ros_ws/src/system_bringup/config/realsense_cameras.yaml. - Slurm
PROJECT_ROOTandCONDA_ENV_NAMEforscripts/train.slurm. - Benchmark checkpoint arrays and
resume_rootinscripts/run_mass_sweep.sh.
