Agentic-Tool-Optimization

Imported from WillNigri/Agentic-Tool-Optimization (AGENTS.md). Install upstream with npx skills add WillNigri/Agentic-Tool-Optimization. Copyright stays with the author.

AGENTS.md — Using ATO from a Coding Agent

This file is for AI coding agents (Claude Code, Codex, Cursor agent mode, Aider, OpenClaw, Hermes, and similar). If you're a human, see README.md instead.

ATO is your local war room for humans and LLMs: decide together, call tools, and verify every outcome. You and the human share the same cockpit — same data, same operations, same audit trail. This doc tells you (the agent) how to use it via CLI and MCP.

1. What ATO is, in one paragraph

ATO sits on the developer's machine. It traces every dispatch (Claude / Codex / Gemini / OpenClaw / Hermes / Ollama / 16+ API providers), joins config changes against trace stats to surface regressions, captures real per-million-token cost, supports cross-runtime replay, and persists everything in a local SQLite database. You operate it through a CLI, an MCP server, or by reading the SQLite database directly. No cloud round-trip is required for any core operation.

2. The three actor surfaces

You can drive ATO three ways, in order of recommendation:

Default to CLI. It is the surface ATO is most optimized for agent use. MCP is for harness integration. Direct SQLite is for power reads.

3. Local file layout — where data lives

ATO writes everything to ~/.ato/. Read these directly if you need to:

4. Setup — how to make ATO usable in this session

If the developer hasn't already set ATO up:

# Check if ATO is installed
ato --version

# If not, the human runs:
brew tap WillNigri/ato && brew install --cask ato      # macOS
# or downloads from agentictool.ai

First-run PATH setup. After installing ATO from a desktop bundle (DMG, AppImage, NSIS, .deb), the CLI is shipped inside the bundle but may not be on PATH yet. The human runs:

ato setup-path
# Symlinks the CLI binary to /usr/local/bin/ato (or ~/.local/bin/ato as fallback).
# Outputs JSON describing what it did. Idempotent — safe to run repeatedly.

If your which ato already returns a path, this is a no-op. If ato isn't on PATH after setup, you're a coding agent reading this who can't use ATO yet — surface this to the human:

"ATO's CLI isn't on PATH. Run ato setup-path from a terminal where the desktop bundle is reachable (or use /Applications/ATO.app/Contents/Resources/binaries/ato-<your-arch> setup-path on macOS to bootstrap)."

Once installed, you can use the CLI immediately. No sign-in required for local operations.

For MCP, the human needs to add ATO's MCP server to their harness config once:

{
  "mcpServers": {
    "ato": {
      "command": "npx",
      "args": ["tsx", "services/mcp-server/src/index.ts"]
    }
  }
}

After that, every MCP-aware coding agent the human runs has ATO's tools available.

5. CLI reference

All commands output JSON to stdout by default. Add --human for human-readable formatting. Add --quiet to suppress non-essential output. Add --db /custom/path.db to point at a non-default SQLite path (rarely needed).

Status legend: [v2.1] = shipped today. [v2.3+] = on the roadmap; not yet available. Use ato --version to check what's live on this machine. If a command isn't there yet, fall back to the equivalent MCP tool or read SQLite directly.

Observation

# Recent dispatches (default: last 20)
ato dispatches recent [--limit N] [--runtime claude|codex|...] [--status success|error]
# [v2.3+]

# Active dispatches (currently running)
ato runs live
# [v2.3+]

# Specific run by ID
ato runs get <run-id>
# [v2.3+]

# Configuration changes for an agent (the ledger)
ato config-changes list --agent <slug> [--since 7d]
# [v2.3+]

# Regressions detected (joins config changes × trace stats, local-mode)
ato regressions list [--days 30] [--window-hours 168] [--min-samples 20]
# [v2.3.2] — runs entirely over local SQLite, no cloud sign-in needed.
# Returns "local-no-schema" source string when the desktop hasn't applied
# the v2.3.2 migration yet — the agent should surface that to the human.

# Failing examples are included inline in `regressions list` output
# under failing_trace_ids (up to 10 per change). Drill them with
# `ato runs get <id>`.

# Cost recommendations (when historical multi-runtime data justifies a swap)
ato cost recommendations [--days 30] [--min-runs 10]
# [v2.3.2] — also local-mode. Same schema dependency as regressions.
#
# v2.3.6 caveat: cost-recommendations surface only when the relevant
# dispatches were API-billed (cost_usd_estimated IS NOT NULL).
# Dispatches via runtime-CLI subscriptions (claude --print, codex exec,
# gemini -p) deliberately persist NULL cost — they don't bill per
# token, so an "API-equivalent" dollar value would have been misleading.
# As a result, users who exclusively use subscriptions will see empty
# recommendations; that's correct (there's no cost to optimize). The
# feature regains traction once a direct-API dispatch path lands and
# subscription vs API rows can be distinguished by billing_source.

# File attribution for a specific dispatch
ato files-touched <run-id>
# [v2.3+]

# Replay history for a trace
ato replays for-trace <trace-id>
# [v2.3+]

Operations

# Dispatch a prompt to a runtime
ato dispatch <runtime> "<prompt>" [--model <model>] [--agent <slug>]
# [v2.3+]

# Start a replay of a past trace on a different runtime
ato replay start <trace-id> --runtime <target> [--model <model>]
# [v2.3+]

# Get the result of a replay (poll until done)
ato replay get <job-id> [--wait]
# [v2.3+]

# Kill a stuck dispatch
ato kill <run-id>
# [v2.3+]

# Bulk kill (kill all matching pattern)
ato kill --all [--runtime <runtime>] [--older-than 5m]
# [v2.3+]

# Compare two traces
ato compare <trace-id-a> <trace-id-b>
# [v2.3+]

Authoring (write things back)

# Draft a skill from a successful replay
ato skills draft --from-replay <job-id> [--out ~/.claude/skills/my-skill/SKILL.md]
# [v2.3+]

# Create a new agent
ato agents create --slug <slug> --runtime <runtime> --system-prompt "<prompt>"
# [v2.3+]

# Update an agent's model
ato agents update <slug> --model <model>
# [v2.3+]

# Create an ops recipe (trigger → action workflow)
ato recipes create --name <name> --trigger <event> --action <action>
# [v2.3+]

Events (long-lived subscriptions)

# Watch ATO events. Blocks until ^C or --until fires.
ato events watch [--type regression|dispatch_failed|replay_done|cost_threshold]
# [v2.3+]
# Emits one JSON event per line to stdout. Pipe to your handler:
# ato events watch --type regression | while read line; do ... done

Reading the database directly

For zero-overhead reads, just query SQLite:

# Most recent dispatches
sqlite3 -json ~/.ato/local.db "
  SELECT id, runtime, status, duration_ms, cost_usd_estimated, created_at
  FROM execution_logs
  ORDER BY created_at DESC
  LIMIT 20
"

# Configuration changes since yesterday
sqlite3 -json ~/.ato/local.db "
  SELECT agent_slug, field, old_value, new_value, changed_at
  FROM agent_config_changes
  WHERE changed_at > datetime('now', '-1 day')
  ORDER BY changed_at DESC
"

The SQLite schema is documented at the top of apps/desktop/src-tauri/src/lib.rs.

6. MCP server reference

Connect over stdio. Tools below are available today (v2.1.x) or planned. Use tools/list to see what's actually exposed on the running server.

Currently exposed (v2.1.x)

v2.3.4 expansion (shipped — Phase 3)

Mirrors the CLI reference. Each CLI subcommand has an equivalent MCP tool, implemented as a thin shell-out so the CLI is the canonical implementation:

• Observation: get_recent_dispatches, get_active_runs, get_run, get_regressions, get_failing_examples, get_config_changes, get_cost_recommendations, get_file_attribution, get_replay_history
• Operations: start_dispatch, start_replay, get_replay_job, kill_run, compare_traces
• Authoring: draft_skill_from_replay, create_agent, update_agent

The MCP server needs the ato CLI to be reachable. Set ATO_CLI_PATH env var to override the lookup; otherwise it tries ato on PATH (most common after ato setup-path), then falls back to the macOS bundle sidecar path.

Planned for Phase 4+ (not yet shipped)

• create_ops_recipe — programmable trigger→action workflows
• Events: subscribe_to_events, acknowledge_event — long-lived event subscription protocol

7. Common recipes — patterns to copy

These are example workflows you can run on the human's behalf when given the goal. They show how to compose the primitives.

Recipe: "Skillify a regression"

The human shipped a config change; quality dropped on N examples. The replay shows the previous runtime would have handled some of them correctly. Make the fix structural so it can't recur.

# 1. Find the most recent regression
REG=$(ato regressions list --severity regression --days 7 | jq '.[0]')
CHANGE_ID=$(echo $REG | jq -r '.change_id')
AGENT=$(echo $REG | jq -r '.agent_slug')

# 2. Get the failing examples
ato regressions failing-examples $CHANGE_ID > failing.json

# 3. Replay the first failing example on the runtime that was working before
PROMPT=$(jq -r '.[0].prompt' failing.json)
OLD_RUNTIME=$(echo $REG | jq -r '.old_value')
JOB_ID=$(ato replay start $(jq -r '.[0].trace_id' failing.json) --runtime $OLD_RUNTIME)

# 4. Wait for replay
RESULT=$(ato replay get $JOB_ID --wait)

# 5. If replay succeeded, draft a skill that routes this prompt pattern
#    back to the old runtime
if [ $(echo $RESULT | jq -r '.status') = "done" ]; then
  ato skills draft --from-replay $JOB_ID --out ~/.claude/skills/route-${AGENT}/SKILL.md
fi

# 6. Tell the human in the activity feed
echo "{\"type\":\"agent_post\", \"text\":\"Drafted skill from regression of @${AGENT}. Review at ~/.claude/skills/route-${AGENT}/SKILL.md.\"}" | ato events post

Recipe: "Watch this dev session for regressions"

The human is about to make a series of edits. Stand by and notify them if anything goes wrong.

# Subscribe to regression events, exit after 60 minutes
timeout 3600 ato events watch --type regression | while read event; do
  AGENT=$(echo $event | jq -r '.agent_slug')
  DELTA=$(echo $event | jq -r '.eval_delta_pp')
  echo "{\"type\":\"agent_post\",\"text\":\"⚠️ Regression on @${AGENT}: eval score dropped ${DELTA}pp. Want me to investigate?\"}" | ato events post
done

Recipe: "Check if there's a cheaper way before this dispatch"

Before running an expensive prompt, see whether ATO's cost recommendations suggest a swap.

ato cost recommendations --agent $AGENT_SLUG | jq '.[] | select(.savings_per_month_usd > 10)'

8. Safety — what's destructive

Every write through the CLI or MCP automatically appends to ~/.ato/agent-logs.jsonl with timestamp, actor, operation, and (when applicable) the diff. The human can grep this any time to see what their agent did.

9. How ATO fits with other tools (so you don't double-instrument)

• Langfuse / Helicone / LangSmith / Phoenix / Braintrust: these are production observability SDKs. They log end-user conversations from a deployed app. ATO is the developer-workflow side. Use both: Langfuse logs your live users; ATO is the cockpit for what you build and replay during development. Don't try to send Langfuse data to ATO or vice versa; they cover different sides of the same agent.
• Cursor / Continue / Cody: these are authoring tools (write code with AI in your editor). ATO is operations (operate the agents). Coexist.
• Aider: single-runtime CLI. ATO sits above it for multi-runtime workflows. They coexist.
• Hermes Agent's skill_manage: Hermes creates skills autonomously; ATO is the verification layer (run tests against historical traces, surface regressions). Complementary.

10. Asking for help

If a command isn't documented here, try:

ato --help
ato <command> --help

If the CLI doesn't have what you need, fall back to MCP (tools/list) or read SQLite directly. If neither has it, the operation probably doesn't exist yet — tell the human, and don't fabricate output.

If you (the agent) are uncertain whether an operation is safe in the human's context, ask before acting. ATO is built for human + agent co-piloting; the human is one message away.

Spec version: 2026-05-11. Reflects ATO v2.2.1 (current) + the v2.3 platform-expansion plan. Commands marked [v2.3+] are documented for the agent's planning purposes but not yet shipped.