About

Imported from shipshitdev/skills (skills/agent-architecture-audit/SKILL.md). Install upstream with npx skills add shipshitdev/skills --skill agent-architecture-audit. Copyright stays with the author.

Agent Architecture Audit

Name: agent-architecture-audit
Author: shipshitdev

Diagnose failures in agent systems by inspecting each layer between the raw model call and the user-visible output.

Contract

Inputs:

Agent or LLM application repository, failing run, trace, log, or symptom
Optional model/provider list, tool definitions, memory files, and UI output

Outputs:

Severity-ranked findings with evidence references
Layer-by-layer failure diagnosis
Ordered fix plan that prefers code gates over prompt-only changes

Creates/Modifies:

None in audit mode
Follow-up fixes only when explicitly requested

External Side Effects:

None by default
External logs, observability tools, or production systems only when already authorized

Confirmation Required:

Before changing prompts, memory, tool contracts, persistence, production config, or user data

Delegates To:

debug for ordinary software defects
evaluation or advanced-evaluation for benchmark design
security-audit for prompt injection, secrets, auth, or privileged tool risk

When to Use

An agent works in a model playground or direct API call but fails in the app.
A wrapper, orchestration layer, memory system, or tool router was added and quality regressed.
The model skips required tools, claims tool use that did not happen, or misreads tool output.
Old conversation facts, stale memories, or compressed summaries leak into new tasks.
Logs show a correct answer but the CLI, API, UI, streaming layer, or renderer shows a different result.
Hidden retry, fallback, repair, or summarization loops may be mutating output.

Do not use for general code review — only when the failure is likely in the agent stack.

Layer Model

Audit the stack from source instructions to delivered output:

Layer	What Can Fail
System prompt	Conflicts, bloat, stale rules, unclear priorities
Session history	Old turns dominate or contradict the current task
Long-term memory	Cross-project leakage, stale preferences, agent-written facts
Distillation	Compaction turns guesses into pseudo-facts
Active recall	Redundant summaries waste context or revive old context
Tool selection	Required tools are optional in code, not enforced
Tool execution	Missing calls, failed calls, unvalidated arguments
Tool interpretation	Output is ignored, overtrusted, or read backward
Answer shaping	Format, schema, markdown, or JSON is changed after reasoning
Transport/rendering	Streaming, API, CLI, or UI mutates valid content
Hidden repair loops	A second model pass silently rewrites the answer
Persistence	Cached artifacts or expired state are reused as live evidence

Audit Process

1. Scope the System

Identify:

user entrypoints and output surfaces
model providers, model tiers, and retry/fallback behavior
prompt assembly path and instruction sources
memory, retrieval, compaction, and persistence paths
tool schemas, tool routing, and execution validation
known symptoms and the first release or commit where they appeared

2. Collect Evidence

Search for:

prompt templates and instruction injection points
tool schemas, routers, validators, and execution logs
memory admission, retrieval, and compaction code
provider calls outside the primary agent loop
fallback, retry, repair, or output rewriting passes
response serialization, markdown rendering, JSON parsing, and stream handling

Prefer file and line evidence. Compare raw model output, post-processed output, transported output, and rendered output when logs are available.

3. Map Failure Mechanisms

For each suspected issue, record:

symptom: what the user sees
layer: where the corruption enters
mechanism: how the layer changes behavior
root cause: code, config, prompt, memory, transport, or process
evidence: file, line, trace id, log row, or reproduced run
confidence: high, medium, or low

4. Prioritize Fixes

Prefer fixes in this order:

Enforce required tool use in code.
Remove or expose hidden repair and fallback agents.
Reduce duplicated context across prompt, memory, history, and summaries.
Tighten memory admission so user corrections outrank agent assertions.
Narrow compaction triggers and preserve uncertainty markers.
Pass through already-valid output instead of rewriting it.
Use typed envelopes for internal protocol boundaries.
Add trace tests that compare raw, processed, transported, and rendered output.

Do not solve tool discipline, memory safety, or transport corruption only by adding stronger prompt wording.

Severity

Severity	Meaning
Critical	The system can confidently perform wrong or unsafe operational behavior
High	Correctness or stability degrades frequently under normal use
Medium	The agent usually works, but the stack is fragile, wasteful, or hard to debug
Low	Maintainability, observability, or cosmetic output issues

Output Format

Lead with findings:

## Findings

| Severity | Layer | Finding | Evidence | Fix |
| --- | --- | --- | --- | --- |
| High | Tool selection | Required retrieval is prompt-only and can be skipped | `src/agent/router.ts:42` | Gate final answer on retrieval result |

## Diagnosis

[Explain which layer corrupts the behavior and why.]

## Fix Plan

1. [Code-first fix]
2. [Validation]
3. [Follow-up hardening]

If no issues are found, say so and list the evidence checked plus remaining blind spots.

Anti-Patterns

Blaming the model before testing wrapper-layer behavior.
Accepting "must use tool" prompt text when no code enforces the requirement.
Treating compressed summaries as ground truth.
Allowing hidden model passes to rewrite final output without traceability.
Reporting green status without comparing internal output to delivered output.

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml

targets:
  - https://api.opensmartroute.ai/api/v1/registry/shipshitdev-skills-agent-architecture-audit/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

shipshitdev-skills-agent-architecture-audit.ocm.jsonjson

{
  "ocm": "1",
  "id": "shipshitdev-skills-agent-architecture-audit",
  "kind": "skill",
  "name": "agent-architecture-audit",
  "description": "Audit LLM and agent applications for wrapper regressions, prompt or memory contamination, tool discipline failures, hidden repair loops, and output rendering corruption. Use before shipping agent features or when an agent works in a direct model call but fails inside the product.",
  "publisher": "shipshitdev",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "general"
    ],
    "tags": [
      "skill-md",
      "agents",
      "llm",
      "architecture",
      "audit",
      "debugging",
      "skills-sh"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Audit LLM and agent applications for wrapper regressions, prompt or memory contamination, tool discipline failures, hidden repair loops, and output rendering corruption. Use before shipping agent features or when an agent works in a direct model call but fails inside the product."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "skills.sh",
      "repository": "https://github.com/shipshitdev/skills",
      "path": "skills/agent-architecture-audit/SKILL.md",
      "ref": "HEAD",
      "url": "https://github.com/shipshitdev/skills/blob/HEAD/skills/agent-architecture-audit/SKILL.md",
      "key": "shipshitdev/skills/skills/agent-architecture-audit/SKILL.md"
    }
  },
  "instructions": "# Agent Architecture Audit\n\nDiagnose failures in agent systems by inspecting each layer between the raw model call and the user-visible output.\n\n## Contract\n\nInputs:\n\n- Agent or LLM application repository, failing run, trace, log, or symptom\n- Optional model/provider list, tool definitions, memory files, and UI output\n\nOutputs:\n\n- Severity-ranked findings with evidence references\n- Layer-by-layer failure diagnosis\n- Ordered fix plan that prefers code gates over prompt-only changes\n\nCreates/Modifies:\n\n- None in audit mode\n- Follow-up fixes only when explicitly requested\n\nExternal Side Effects:\n",
  "cost": {
    "context_tokens": 1471
  }
}

Fetch it by URL: GET /api/v1/registry/shipshitdev-skills-agent-architecture-audit/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.