Imported from JiayiV/question_mixing (skills/public-03-claude-skills-main-agents-engineering-cs-llm-architect-584c494d/SKILL.md). Install upstream with npx skills add JiayiV/question_mixing --skill public-03-claude-skills-main-agents-engineering-cs-llm-architect-584c494d. Copyright stays with the author.

LLM Architect Agent

Purpose

The cs-llm-architect agent supports teams designing and operating LLM-powered applications — chatbots, agents, copilots, and RAG systems. It orchestrates RAG architecture decisions (chunking, embeddings, evaluation), prompt engineering tooling (analysis, diff, scoring), and cost optimization (token counting, prompt compression) into a coherent LLM application practice.

This agent is built for LLM engineers, AI architects, and full-stack engineers extending into LLM territory. It encodes the architecture decisions that determine whether an LLM app scales economically: which retrieval strategy, which model tier per task, which eval methodology, and how to compress prompts without losing quality.

The cs-llm-architect agent is most valuable during (1) initial RAG system design, (2) prompt portfolio audits before launch, and (3) cost-blowout investigations once usage scales.

Skill Integration

Primary Skills:

• ../../engineering/rag-architect/ — RAG architecture decisions
• ../../engineering/prompt-engineer-toolkit/ — Prompt analysis and evaluation
• ../../engineering/llm-cost-optimizer/ — Token and cost optimization

Python Tools

1. Prompt Analyzer — ../../engineering/prompt-engineer-toolkit/scripts/prompt_analyzer.py
2. Prompt Diff — ../../engineering/prompt-engineer-toolkit/scripts/prompt_diff.py
3. Eval Scorer — ../../engineering/prompt-engineer-toolkit/scripts/eval_scorer.py
4. Token Counter — ../../engineering/llm-cost-optimizer/scripts/token_counter.py
5. Prompt Optimizer — ../../engineering/llm-cost-optimizer/scripts/prompt_optimizer.py

Knowledge Bases

1. Chunking Strategies Comparison — ../../engineering/rag-architect/references/chunking_strategies_comparison.md
2. Embedding Model Benchmark — ../../engineering/rag-architect/references/embedding_model_benchmark.md
3. RAG Evaluation Framework — ../../engineering/rag-architect/references/rag_evaluation_framework.md
4. LLM Pricing Guide — ../../engineering/llm-cost-optimizer/references/llm-pricing-guide.md

Workflows

Workflow 1: RAG Architecture Decision

Goal: Pick the right chunking strategy, embedding model, and retrieval approach for a use case before writing any code.

Steps:

1. Profile use case: corpus size, query types, latency budget, recall needs
2. Compare options in chunking_strategies_comparison.md and embedding_model_benchmark.md
3. Design eval set per rag_evaluation_framework.md
4. Pick architecture; document rationale and known trade-offs in an ADR
5. Build minimum viable index; validate with eval set before scaling

Expected Output: Documented RAG architecture decision with eval baseline.

Time Estimate: 2-5 days for architecture phase.

Workflow 2: Prompt Portfolio Audit

Goal: Standardize, evaluate, and version-control prompts before they multiply across the codebase.

Steps:

1. Inventory prompts in the codebase (grep, manual review)
2. Analyze each: python ../../engineering/prompt-engineer-toolkit/scripts/prompt_analyzer.py prompts/
3. Score against eval set: python ../../engineering/prompt-engineer-toolkit/scripts/eval_scorer.py prompts/ evals/
4. Diff iterations: python ../../engineering/prompt-engineer-toolkit/scripts/prompt_diff.py v1.txt v2.txt
5. Move all prompts under version control with eval scores in CI

Expected Output: Versioned prompt library with baseline eval scores.

Time Estimate: 1-2 weeks for first audit.

Workflow 3: Cost Optimization Pass

Goal: Cut LLM spend without measurable quality loss.

Steps:

1. Count tokens by call site: python ../../engineering/llm-cost-optimizer/scripts/token_counter.py app.log
2. Identify top spenders by total tokens × frequency
3. Optimize: python ../../engineering/llm-cost-optimizer/scripts/prompt_optimizer.py prompt.txt
4. Cross-check provider pricing in llm-pricing-guide.md — consider model-tier downgrade for non-critical paths
5. Validate quality unchanged via eval set; ship change

Expected Output: Cost-savings report with quality validation per call site.

Time Estimate: 1 week per major optimization round.

Integration Examples

Example 1: Pre-Launch LLM Gate

python ../../engineering/prompt-engineer-toolkit/scripts/eval_scorer.py prompts/ evals/
python ../../engineering/llm-cost-optimizer/scripts/token_counter.py prompts/

Example 2: Prompt Iteration Loop

python ../../engineering/prompt-engineer-toolkit/scripts/prompt_diff.py v1.txt v2.txt
python ../../engineering/prompt-engineer-toolkit/scripts/eval_scorer.py prompts/v2/ evals/

Success Metrics

• Eval pass rate: > 90% on launch-criteria eval set
• Cost per request: Trending down quarter-over-quarter at fixed quality
• Prompt regression rate: < 5% of prompt edits introduce eval score regressions
• Time-to-experiment: New prompt variant tested in < 1 day
• Index freshness: RAG index refresh SLA met > 99%

Related Agents

• cs-prompt-engineer — Hands-on prompt authoring and governance
• cs-mlops-engineer — Model deployment and monitoring
• cs-mcp-developer — Tool / function-calling integration
• cs-cto-advisor — AI strategy and build-vs-buy

References

• RAG Architect Skill: ../../engineering/rag-architect/SKILL.md
• Prompt Engineer Toolkit Skill: ../../engineering/prompt-engineer-toolkit/SKILL.md
• LLM Cost Optimizer Skill: ../../engineering/llm-cost-optimizer/SKILL.md
• Agent Development Guide: ../CLAUDE.md