# Agent engineering with OpenSmartRoute How the eight disciplines of building production agents map onto the SDK and the hosted platform - what each one means, where it lives in the product, how it is used, and how it connects to the tools teams already run (Langfuse, Arize Phoenix, TruLens, DeepEval, Ragas, OpenTelemetry). | Discipline | What it is about | Where it lives | |---|---|---| | [Prompt engineering](#1-prompt-engineering) | The wording that steers a model: versioned, deployed by label, visible in every trace | Prompt registry (`/api/v1/prompts`, dashboard *Prompts*), personas, marketplace prompts | | [Harness engineering](#2-harness-engineering) | The loop that runs a target: tools, rounds, retries, fallback, limits | `execute` / `execute_with_fallback`, `AgentHarness`, `ResourceLimits` | | [Context engineering](#3-context-engineering) | What goes in the window and in which order: compaction, retrieval, knowledge, budgets | `compress_request`, `Retriever`, `ContextEngineer`, the knowledge service | | [Memory engineering](#4-memory-engineering) | What persists between requests and how it returns: conversations, facts, tiers | Conversation memory, `/api/v1/memory`, `MemoryRouter` | | [Agent engineering](#5-agent-engineering) | Multi-step work: plan, execute, judge, refine, hand off | `RoutePlan`, `MultiRoundExecutor`, human hand-offs, the autonomy crew | | [Agent integrations](#6-agent-integrations) | Speaking every protocol an agent stack uses | MCP (client, server, gateway), A2A, LangGraph, Microsoft Agent Framework, OpenAI tools, Agent Skills, the SDKs and apps | | [Agent safety and guardrails](#7-agent-safety-and-guardrails) | Keeping prompts, tools, data and spend inside the lines | `InputGuard` / `OutputGuard`, data protection, origin policy, signed manifests, budgets, approvals | | [Agent analytics and evals](#8-agent-analytics-and-evals) | Knowing whether it works: traces, scores, datasets, judges, gates - in your tools too | Trace export to Langfuse / OTLP, datasets in Ragas / DeepEval / Langfuse shape, judge, quality gates, `osr eval` | Every surface below is in the product today; the request examples are real calls against a workspace API key. ## 1. Prompt engineering A prompt is a product artefact, not a string in a service. The **prompt registry** keeps, per workspace, prompts by name with immutable numbered versions, labels that deploy a version (`production`, `staging` - a label names one version, so moving it is the release), `{{variable}}` placeholders with optional defaults (`{{tone|calm}}`), and a preview that renders a version without routing anything. Two kinds of prompt: - **text** - the instruction; it goes in front of the request's own system message; - **chat** - a list of `{role, content}` messages: the system turns become the instruction, the user / assistant turns are prepended to the history as few-shot examples. ```bash # version 1 of a text prompt, deployed as production curl -X POST $OSR/api/v1/prompts -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{ "name": "support-reply", "content": "You answer for {{company}} in a {{tone|calm}} voice. Keep it under three sentences.", "labels": ["production"], "message": "first cut" }' # a request names it; the resolved version comes back in X-OSR-Prompt and in the request's trace curl -X POST $OSR/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{ "model": "auto", "messages": [{"role": "user", "content": "Where is my order?"}], "osr": {"prompt": {"name": "support-reply", "label": "production", "variables": {"company": "Acme"}}} }' ``` `/api/v1/route` takes the same object as `prompt`. The registry is on the dashboard (*Prompts*): versions with their change notes, labels moved by a click, a preview with the variables filled in. `PUT /api/v1/prompts/{name}/{version}/labels` deploys, `DELETE` removes a version or a name, and the workspace export (`GET /api/v1/me/export`) carries every version. The guard and the data-protection pass see the rendered instruction exactly as the model will. **Did the change help?** The usage row of every request that named a prompt keeps the `name@version` tag (`facts.prompt`), and `GET /api/v1/prompts/{name}/stats?days=30` lays the versions side by side: requests, the answered share, tokens and cost, median latency, and the outcomes reported on them - success rate and mean quality from `POST /api/v1/feedback`, the judge's grade, or a score imported from Ragas / DeepEval (section 8). The dashboard shows the same next to each version, so moving `production` from v3 to v4 is a decision with numbers, and moving it back is one click. Around the registry: **personas** (`opensmartroute.adapters.personas`) turn Markdown / agent-mode files into instruction-bearing targets the router can choose; marketplace **prompt listings** publish a prompt with its variables and examples for other workspaces; the SDK's `compress_request` compacts an over-long composed instruction before it costs tokens. ## 2. Harness engineering The harness is what runs a chosen target and keeps it honest. In the SDK, `execute` / `execute_with_fallback` (`opensmartroute.execution`) run a decision or a plan: skills chain or run in parallel, tools are offered and run in bounded rounds (`tool_spec`, `_run_tool_round`), a failed target is replaced by the next admissible one (stopping once a stream has committed), and every step is recorded (`ExecutionStep`) with its usage. External agents join through `AgentHarness` adapters - `CallableHarness`, `HTTPHarness`, `SubprocessHarness` - that return a `HarnessResult` (success, usage, quality, artefacts). `ResourceLimits` / `ResourceLimiter` (`opensmartroute.security.limits`) bound steps, calls, depth, tokens, cost and elapsed time; `TimeoutMiddleware` checks routing time. On the platform, `POST /api/v1/route` with `execute: true` and the OpenAI-compatible `/v1/chat/completions` are the harness: fallback re-plans around the failing target, the spend admission prices the call before it is made, and the usage row keeps what each step cost. ## 3. Context engineering What the model sees, in what order, under what budget. `compress_request` (`opensmartroute.compress`) removes whitespace and repetition and trims old history while protecting fenced code, system messages and tool-call groups, reporting `tokens_saved`; the `context_window` policy rule refuses an input the chosen model cannot hold. `Retriever.narrow` (`opensmartroute.retrieval`, BM25 + dense with reciprocal-rank fusion) narrows the candidate catalogue so routing scales to hundreds of targets. `ContextEngineer` (`opensmartroute.knowledge`) ranks user facts, shared knowledge and live web snippets into section budgets, and `WebGroundingMiddleware` adds fresh web context when the request is about something recent. On the platform the **knowledge service** (`/api/v1/knowledge`: facts, forget, context, suggest, export, train) keeps a graph of what a workspace knows, with decay and an optional embedding model, and the dashboard *Memory* page shows it next to the conversations. ## 4. Memory engineering What persists. **Conversation memory** (`/api/v1/conversations`, `/api/v1/memory`) keeps a conversation's turns and a rolling summary so a client sends only the new message (`osr.conversation_id`), learns facts a person states about themselves (name, role, language, preferences; rule-based, editable, forgettable) and puts the relevant ones in front of each request; subjects are the personal workspace, the organization member or the API key, so colleagues never see each other's memory. In the SDK, `MemoryRouter` / `MemoryTier` / `ImportanceGate` (`opensmartroute.strategies.memory`) give an in-process tiered memory with budgets, decay and learned usefulness, and `RequestMemory` keeps the evidence a feedback signal needs. Retention: conversations are capped per thread, deletion is explicit (`DELETE /api/v1/conversations/{id}`, *Memory* page), knowledge decays on its own, and the request-log retention of the plan applies to the rows. ## 5. Agent engineering Multi-step work. A **plan** (`RoutePlan` / `PlanSlot`) composes personas, skills, tools and a model into stages; `MultiRoundExecutor` (`opensmartroute.strategies.progress`) runs route → execute → judge → refine with target switching, best-answer return and round / cost / deadline bounds. **Human hand-offs** (`/api/v1/handoffs`, the *Human queue* page) queue a request for a person whose answer becomes the learning signal. On the platform the **autonomy crew** (`/api/v1/admin/autonomy`, console *Automation*) runs the deployment's own jobs through the router and grades them. The frameworks adapters make the router a node of someone else's agent: `langgraph_node` / `langgraph_condition`, `maf_router_executor` for Microsoft Agent Framework, `openai_tool_spec` / handler for function calling. ## 6. Agent integrations - **MCP** - the SDK is a client (`StdioMCPClient`, `HttpMCPClient`; tools, resources and prompts become targets) and a server (`MCPServer`; the hosted `/mcp` endpoint with OAuth 2.1); the platform's **MCP gateway** (`/api/v1/workspace/mcp-servers`, dashboard *MCP servers*) connects a workspace's own servers and puts their tools in the routing pool under approval modes and per-key limits. - **A2A** - agent cards become targets; `a2a_handler` runs tasks and polls them. - **Agent Skills** - `SKILL.md` packages load as skills (`load_skills`) and compose through `SkillGraph`. - **LangGraph, Microsoft Agent Framework, OpenAI tools** - see above. - **Dialects** - OpenAI chat / Responses, Anthropic Messages and Gemini `generateContent` on one endpoint. - **Clients** - the TypeScript SDK, the VS Code extension, the desktop and browser apps, the CLI. [docs/COMPATIBILITY.md](https://opensmartroute.ai/docs/COMPATIBILITY.md) lists every connection with how it was verified. ## 7. Agent safety and guardrails - **Input** - `InputGuard` / `GuardMiddleware` (`opensmartroute.security`) screen size, injection gadgets and steering, with a trainable `GadgetDetector`; `Redactor` masks personal data. - **Output** - `OutputGuard` / `StreamScreen` flag, redact or block leaked secrets, personal data and injection in answers, streamed ones included. On the platform the **`output_guard` policy** (`off | flag | redact | block`, per workspace or tenant; `pii: false` keeps addresses a support answer must quote) switches it on for every executed answer: a streamed answer is held until screened under `redact` / `block`, the finding rides on the answer, in `X-OSR-Output-Guard` and on the usage row (`facts.output_guard`) - kinds and counts, never the values. - **Data** - the `data_protection` policy (`off | detect | mask | block`, per workspace or tenant) decides what happens to personal and health data before any model sees it; `X-OSR-Data-Protection` says what acted. - **Tools** - `OriginPolicy` enforces where a tool argument may come from; signed MCP manifests (`sign_manifest` / `verify_manifest`); gateway approval modes (`auto`, `read_only`, `manual`). - **Spend and shape** - budgets, quotas, key scopes and limits (`tokens_per_minute`, `allowed_ips`, spend caps), tenant constraints, `ResourceLimits`. - **Operations** - four-eyes approvals for sensitive operator actions, an IP blocklist and maintenance mode (console *Access control*), quality gates that alert when routing regresses. - `osr safety` runs the red-team suite against a catalogue. [docs/SECURITY.md](https://opensmartroute.ai/docs/SECURITY.md) is the full account. ## 8. Agent analytics and evals **Traces.** Every request carries a trace: the decision's signals, candidates, policy rejections and strategy scores, the execution steps, usage and outcome (`GET /api/v1/trace/{request_id}`, dashboard *Activity* and *Events*). Spans and events flow to the SDK sinks - memory, metrics, log, file and OpenTelemetry ([docs/OBSERVABILITY.md](https://opensmartroute.ai/docs/OBSERVABILITY.md)) - and, on the platform, to Application Insights. **Trace export to your observability tool.** A workspace's request log can be shipped continuously to the tools teams already use, as traces rather than lines (`/api/v1/workspace/log-export`, dashboard *Governance › Data*): | Destination (`kind`) | Receives | Credential | |---|---|---| | `langfuse` | Langfuse ingestion batches: a trace per request, a generation with model, token counts and cost, your reported outcomes and the judge's grade as scores; the prompt and the answer when payloads are kept | `public key:secret key` | | `otlp` | OTLP/HTTP spans with OpenInference attributes (`openinference.span.kind` CHAIN for the decision and LLM for the generation, `llm.model_name`, `llm.token_count.*`, `input.value` / `output.value`, scores) - read natively by Arize Phoenix, Arize AX and TruLens, stored by any OpenTelemetry collector (Grafana Tempo, Honeycomb, Datadog) | a bearer token, or `header=value;header=value` (Arize: `space_id=...;api_key=...`) | | `webhook`, `azure_blob` | the rows as JSON lines, for a SIEM or a data lake | bearer token / container SAS | Trace ids derive from the request id, so a client that forwards `X-Request-Id` into its own tracing can join the two. The cursor and schedule are the same as for the row exports; *Export now* ships the next batch. **Scores.** Outcomes reported with `POST /api/v1/feedback` (success, quality, preference) are the learning signal and the scores exported above; the **answer judge** (`/api/v1/evaluations/judge`) grades a sampled share of executed answers on correctness, completeness, relevance and safety and writes the grade as an outcome. **Datasets.** Kept datasets (`/api/v1/evaluations/datasets`: from logged requests, from a window of kept prompts, or explicit rows) run against the router (`.../run`), replay against a candidate policy, and gate regressions (**quality gates**, re-run on a schedule and after every policy change; **shadow routing** compares a candidate policy on live traffic without serving it). A dataset downloads in the shape another framework reads: | `?format=` | Shape | For | |---|---|---| | `jsonl` | the rows `osr eval` reads | `osr eval` on the file, `--min-accuracy` as a CI gate | | `ragas` | `user_input`, `reference`, `retrieved_contexts` | a Ragas `EvaluationDataset` (faithfulness, answer relevancy, context precision) | | `deepeval` | `input`, `expected_output`, `context`, `retrieval_context` | DeepEval goldens (G-Eval, hallucination, answer relevancy) | | `langfuse` | `input`, `expected_output`, `metadata` | Langfuse dataset items and experiments | The routing label (expected target, acceptable targets, task type) travels in each row's metadata so a run in the other tool joins back to the target the row names. Rows carry `context.documents` and `context.reference_answer` when the dataset has them; those become the retrieved contexts and the reference. **And back.** What the other tool computed returns with `POST /api/v1/evaluations/datasets/{id}/results` - a Ragas result table (`result.to_pandas().to_dict("records")`), DeepEval test results (`metrics_data`), Langfuse run items with their `scores`, or plain `{row | request_id | input, scores, passed}` rows from TruLens, Arize or anything else. Rows join by the `row` / `request_id` metadata the download carried (or by prompt text); the scores are kept as an **imported-scores** run on the Evaluation page (per-metric mean / min / max, pass rate, the per-row scores), and every row that came from a logged request receives the mean score as an outcome with source `eval:` - so the faithfulness Ragas measured shows in the trace, in the prompt-version statistics and in the signals the learners weigh, next to the outcomes your own users reported. ```python # Ragas on a downloaded dataset, scores back to the platform from ragas import evaluate, EvaluationDataset rows = [json.loads(l) for l in open("capitals.ragas.jsonl")] result = evaluate(EvaluationDataset.from_list(rows), metrics=[faithfulness, answer_relevancy]) table = result.to_pandas().to_dict("records") # keeps the "osr" column with row / request_id requests.post(f"{OSR}/api/v1/evaluations/datasets/{DATASET}/results", headers=H, json={"tool": "ragas", "format": "ragas", "rows": table, "threshold": 0.7}) ``` **Offline.** `osr eval` (baselines, robustness, ablation, cost / quality frontiers, calibration - ECE, Brier, conformal - and a minimum-accuracy gate for CI), `osr ope` (off-policy evaluation of logged decisions: IPS, SNIPS, doubly robust), and the public routing-overhead measurement. [docs/REFERENCE.md](https://opensmartroute.ai/docs/REFERENCE.md) has every call. ## Putting it together A team adopting all eight at once does this: keep the instruction in the **prompt registry** and reference it by label; let the **harness** run the plan with fallback and limits; let **compaction, retrieval and knowledge** fit the window; turn **conversation memory** on so clients send only the new turn; compose **plans** and hand the rest to people; connect the stack's **MCP servers** and frameworks; set the **data-protection** mode and budgets; and point a **trace export** at Langfuse or Phoenix while **quality gates** hold the line and datasets go to Ragas or DeepEval for the metrics those tools compute best. Every one of those is a setting or a request field, not a rewrite.