ai-ops

Instruction file imported from carlos17031691/base-ai (.cursor/rules/ai-ops.mdc). Copyright stays with the author.

AI Usage, Token Optimization, Model Selection & Cost Monitoring (AI FinOps)

Apply disciplined engineering to keep AI features fast, accurate and cost-effective:

1. Right-sizing & model routing (Model Tiering)

Never use a frontier reasoning model by default for every task. Route requests by complexity:

• Tier 1 — Fast / Small models (e.g. Flash-Lite, Haiku, Mini, small open-weight models): classification, intent routing, entity extraction, PII scrubbing, formatting, simple summaries and query rewriting.
• Tier 2 — Balanced production models (e.g. Sonnet, GPT-4o, Gemini Pro/Flash): general RAG synthesis, code generation, multi-step tool calling and structured customer-facing responses.
• Tier 3 — Frontier / Deep reasoning models: complex architectural analysis, multi-constraint planning, ambiguous debugging or formal verification.
• Cascade pattern & quota rollover: try a prioritized list of models (e.g. AI_MODELS=modelA:free,modelB:free,modelC). The adapter iterates across candidates upon 429 (rate limit or shared pool exhaustion), transient timeouts, or 5xx errors. For schema/confidence checks, escalate from Tier 1 to Tier 2 only when output validation fails.
• Provider JSON mode compliance: when setting response_format: { type: "json_object" }, always include explicit directives mentioning "json" in the system prompt to satisfy OpenAI/Azure/OpenRouter provider API contracts.

2. Input token optimization

1. Prompt Caching (Prefix Caching):
   • Place static content first and in a deterministic order: System Instructions → Tool Definitions → Few-Shot Examples → Shared Reference Context → Dynamic User Turn.
   • Never inject changing timestamps, random UUIDs or per-request metadata at the top of the system prompt (it invalidates the prefix cache and forfeits 75–90% input token discounts).
2. Context pruning & compression:
   • Pass only the top-ranked chunks from RAG (typically 3–6 concise chunks after reranking, not raw full documents).
   • Project database/API payloads to the minimal fields required before placing them in the prompt.
   • Summarize older conversation turns asynchronously (memory skill) instead of resending unbounded chat histories.
3. Exact & semantic response caching:
   • Cache deterministic requests by hash(model + system_prompt_version + normalized_input).
   • Use Batch APIs (~50% cost reduction) for non-interactive workloads (bulk classification, nightly evaluations, backfill embeddings).

3. Output token optimization (the most expensive & slowest tokens)

• Output tokens typically cost 3x–5x more than input tokens and dominate latency:
  • Always set an explicit, tight max_tokens cap per endpoint/task.
  • Request compact structured schemas (short keys, enum codes, booleans) instead of verbose prose explanations when the consumer is code.
  • Use stop sequences and streaming (SSE) to improve perceived latency (Time To First Token).

4. Token & cost monitoring (LLM Telemetry & Budgets)

Every LLM or embedding call in src/<name> must record structured telemetry (observability skill):

• Required fields per call:
  • trace_id, service, feature (use-case name), tenant_id / user_id
  • provider, model, prompt_version
  • input_tokens, cached_input_tokens, output_tokens, total_tokens
  • cache_hit_ratio (cached_input_tokens / input_tokens)
  • estimated_cost_usd
  • ttft_ms (time to first token), latency_ms, status / fallback_used
• Hard budgets & circuit breakers:
  • Enforce per-request token limits, per-user/tenant rate & spend quotas (daily/monthly USD caps) and alert on sudden token-burn spikes.
  • Define graceful degradation when a budget is reached (switch to a cheaper fallback model or return 429 Quota Exceeded).

5. Coding-agent token hygiene (inside base-ai)

• Keep .ai/rules.md and .ai/memory.md concise and high-signal (they load on every turn). Run base-ai ai stats to inspect the token footprint.
• Use base-ai context search "<query>" to read targeted file/line ranges instead of loading entire repositories into context.