Agent engineering with OpenSmartRoute

How the eight disciplines of building production agents map onto the SDK and the hosted platform - what each one means, where it lives in the product, how it is used, and how it connects to the tools teams already run (Langfuse, Arize Phoenix, TruLens, DeepEval, Ragas, OpenTelemetry).

Every surface below is in the product today; the request examples are real calls against a workspace API key.

1. Prompt engineering

A prompt is a product artefact, not a string in a service. The prompt registry keeps, per workspace, prompts by name with immutable numbered versions, labels that deploy a version (production, staging - a label names one version, so moving it is the release), {{variable}} placeholders with optional defaults ({{tone|calm}}), and a preview that renders a version without routing anything.

Two kinds of prompt:

• text - the instruction; it goes in front of the request's own system message;
• chat - a list of {role, content} messages: the system turns become the instruction, the user / assistant turns are prepended to the history as few-shot examples.

# version 1 of a text prompt, deployed as production
curl -X POST $OSR/api/v1/prompts -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{
  "name": "support-reply",
  "content": "You answer for {{company}} in a {{tone|calm}} voice. Keep it under three sentences.",
  "labels": ["production"], "message": "first cut"
}'
# a request names it; the resolved version comes back in X-OSR-Prompt and in the request's trace
curl -X POST $OSR/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{
  "model": "auto", "messages": [{"role": "user", "content": "Where is my order?"}],
  "osr": {"prompt": {"name": "support-reply", "label": "production", "variables": {"company": "Acme"}}}
}'

/api/v1/route takes the same object as prompt. The registry is on the dashboard (Prompts): versions with their change notes, labels moved by a click, a preview with the variables filled in. PUT /api/v1/prompts/{name}/{version}/labels deploys, DELETE removes a version or a name, and the workspace export (GET /api/v1/me/export) carries every version. The guard and the data-protection pass see the rendered instruction exactly as the model will.

Did the change help? The usage row of every request that named a prompt keeps the name@version tag (facts.prompt), and GET /api/v1/prompts/{name}/stats?days=30 lays the versions side by side: requests, the answered share, tokens and cost, median latency, and the outcomes reported on them - success rate and mean quality from POST /api/v1/feedback, the judge's grade, or a score imported from Ragas / DeepEval (section 8). The dashboard shows the same next to each version, so moving production from v3 to v4 is a decision with numbers, and moving it back is one click.

Around the registry: personas (opensmartroute.adapters.personas) turn Markdown / agent-mode files into instruction-bearing targets the router can choose; marketplace prompt listings publish a prompt with its variables and examples for other workspaces; the SDK's compress_request compacts an over-long composed instruction before it costs tokens.

2. Harness engineering

The harness is what runs a chosen target and keeps it honest. In the SDK, execute / execute_with_fallback (opensmartroute.execution) run a decision or a plan: skills chain or run in parallel, tools are offered and run in bounded rounds (tool_spec, _run_tool_round), a failed target is replaced by the next admissible one (stopping once a stream has committed), and every step is recorded (ExecutionStep) with its usage. External agents join through AgentHarness adapters - CallableHarness, HTTPHarness, SubprocessHarness - that return a HarnessResult (success, usage, quality, artefacts). ResourceLimits / ResourceLimiter (opensmartroute.security.limits) bound steps, calls, depth, tokens, cost and elapsed time; TimeoutMiddleware checks routing time. On the platform, POST /api/v1/route with execute: true and the OpenAI-compatible /v1/chat/completions are the harness: fallback re-plans around the failing target, the spend admission prices the call before it is made, and the usage row keeps what each step cost.

3. Context engineering

What the model sees, in what order, under what budget. compress_request (opensmartroute.compress) removes whitespace and repetition and trims old history while protecting fenced code, system messages and tool-call groups, reporting tokens_saved; the context_window policy rule refuses an input the chosen model cannot hold. Retriever.narrow (opensmartroute.retrieval, BM25 + dense with reciprocal-rank fusion) narrows the candidate catalogue so routing scales to hundreds of targets. ContextEngineer (opensmartroute.knowledge) ranks user facts, shared knowledge and live web snippets into section budgets, and WebGroundingMiddleware adds fresh web context when the request is about something recent. On the platform the knowledge service (/api/v1/knowledge: facts, forget, context, suggest, export, train) keeps a graph of what a workspace knows, with decay and an optional embedding model, and the dashboard Memory page shows it next to the conversations.

4. Memory engineering

What persists. Conversation memory (/api/v1/conversations, /api/v1/memory) keeps a conversation's turns and a rolling summary so a client sends only the new message (osr.conversation_id), learns facts a person states about themselves (name, role, language, preferences; rule-based, editable, forgettable) and puts the relevant ones in front of each request; subjects are the personal workspace, the organization member or the API key, so colleagues never see each other's memory. In the SDK, MemoryRouter / MemoryTier / ImportanceGate (opensmartroute.strategies.memory) give an in-process tiered memory with budgets, decay and learned usefulness, and RequestMemory keeps the evidence a feedback signal needs. Retention: conversations are capped per thread, deletion is explicit (DELETE /api/v1/conversations/{id}, Memory page), knowledge decays on its own, and the request-log retention of the plan applies to the rows.

5. Agent engineering

Multi-step work. A plan (RoutePlan / PlanSlot) composes personas, skills, tools and a model into stages; MultiRoundExecutor (opensmartroute.strategies.progress) runs route → execute → judge → refine with target switching, best-answer return and round / cost / deadline bounds. Human hand-offs (/api/v1/handoffs, the Human queue page) queue a request for a person whose answer becomes the learning signal. On the platform the autonomy crew (/api/v1/admin/autonomy, console Automation) runs the deployment's own jobs through the router and grades them. The frameworks adapters make the router a node of someone else's agent: langgraph_node / langgraph_condition, maf_router_executor for Microsoft Agent Framework, openai_tool_spec / handler for function calling.

6. Agent integrations

• MCP - the SDK is a client (StdioMCPClient, HttpMCPClient; tools, resources and prompts become targets) and a server (MCPServer; the hosted /mcp endpoint with OAuth 2.1); the platform's MCP gateway (/api/v1/workspace/mcp-servers, dashboard MCP servers) connects a workspace's own servers and puts their tools in the routing pool under approval modes and per-key limits.
• A2A - agent cards become targets; a2a_handler runs tasks and polls them.
• Agent Skills - SKILL.md packages load as skills (load_skills) and compose through SkillGraph.
• LangGraph, Microsoft Agent Framework, OpenAI tools - see above.
• Dialects - OpenAI chat / Responses, Anthropic Messages and Gemini generateContent on one endpoint.
• Clients - the TypeScript SDK, the VS Code extension, the desktop and browser apps, the CLI.

Compatibility matrix lists every connection with how it was verified.

7. Agent safety and guardrails

• Input - InputGuard / GuardMiddleware (opensmartroute.security) screen size, injection gadgets and steering, with a trainable GadgetDetector; Redactor masks personal data.
• Output - OutputGuard / StreamScreen flag, redact or block leaked secrets, personal data and injection in answers, streamed ones included. On the platform the output_guard policy (off | flag | redact | block, per workspace or tenant; pii: false keeps addresses a support answer must quote) switches it on for every executed answer: a streamed answer is held until screened under redact / block, the finding rides on the answer, in X-OSR-Output-Guard and on the usage row (facts.output_guard) - kinds and counts, never the values.
• Data - the data_protection policy (off | detect | mask | block, per workspace or tenant) decides what happens to personal and health data before any model sees it; X-OSR-Data-Protection says what acted.
• Tools - OriginPolicy enforces where a tool argument may come from; signed MCP manifests (sign_manifest / verify_manifest); gateway approval modes (auto, read_only, manual).
• Spend and shape - budgets, quotas, key scopes and limits (tokens_per_minute, allowed_ips, spend caps), tenant constraints, ResourceLimits.
• Operations - four-eyes approvals for sensitive operator actions, an IP blocklist and maintenance mode (console Access control), quality gates that alert when routing regresses.
• osr safety runs the red-team suite against a catalogue.

Security model is the full account.

8. Agent analytics and evals

Traces. Every request carries a trace: the decision's signals, candidates, policy rejections and strategy scores, the execution steps, usage and outcome (GET /api/v1/trace/{request_id}, dashboard Activity and Events). Spans and events flow to the SDK sinks - memory, metrics, log, file and OpenTelemetry (Tracing and observability) - and, on the platform, to Application Insights.

Trace export to your observability tool. A workspace's request log can be shipped continuously to the tools teams already use, as traces rather than lines (/api/v1/workspace/log-export, dashboard Governance › Data):

Trace ids derive from the request id, so a client that forwards X-Request-Id into its own tracing can join the two. The cursor and schedule are the same as for the row exports; Export now ships the next batch.

Scores. Outcomes reported with POST /api/v1/feedback (success, quality, preference) are the learning signal and the scores exported above; the answer judge (/api/v1/evaluations/judge) grades a sampled share of executed answers on correctness, completeness, relevance and safety and writes the grade as an outcome.

Datasets. Kept datasets (/api/v1/evaluations/datasets: from logged requests, from a window of kept prompts, or explicit rows) run against the router (.../run), replay against a candidate policy, and gate regressions (quality gates, re-run on a schedule and after every policy change; shadow routing compares a candidate policy on live traffic without serving it). A dataset downloads in the shape another framework reads:

The routing label (expected target, acceptable targets, task type) travels in each row's metadata so a run in the other tool joins back to the target the row names. Rows carry context.documents and context.reference_answer when the dataset has them; those become the retrieved contexts and the reference.

And back. What the other tool computed returns with POST /api/v1/evaluations/datasets/{id}/results - a Ragas result table (result.to_pandas().to_dict("records")), DeepEval test results (metrics_data), Langfuse run items with their scores, or plain {row | request_id | input, scores, passed} rows from TruLens, Arize or anything else. Rows join by the row / request_id metadata the download carried (or by prompt text); the scores are kept as an imported-scores run on the Evaluation page (per-metric mean / min / max, pass rate, the per-row scores), and every row that came from a logged request receives the mean score as an outcome with source eval:<tool> - so the faithfulness Ragas measured shows in the trace, in the prompt-version statistics and in the signals the learners weigh, next to the outcomes your own users reported.

# Ragas on a downloaded dataset, scores back to the platform
from ragas import evaluate, EvaluationDataset
rows = [json.loads(l) for l in open("capitals.ragas.jsonl")]
result = evaluate(EvaluationDataset.from_list(rows), metrics=[faithfulness, answer_relevancy])
table = result.to_pandas().to_dict("records")           # keeps the "osr" column with row / request_id
requests.post(f"{OSR}/api/v1/evaluations/datasets/{DATASET}/results", headers=H,
              json={"tool": "ragas", "format": "ragas", "rows": table, "threshold": 0.7})

Offline. osr eval (baselines, robustness, ablation, cost / quality frontiers, calibration - ECE, Brier, conformal - and a minimum-accuracy gate for CI), osr ope (off-policy evaluation of logged decisions: IPS, SNIPS, doubly robust), and the public routing-overhead measurement. Python API reference has every call.

Putting it together

A team adopting all eight at once does this: keep the instruction in the prompt registry and reference it by label; let the harness run the plan with fallback and limits; let compaction, retrieval and knowledge fit the window; turn conversation memory on so clients send only the new turn; compose plans and hand the rest to people; connect the stack's MCP servers and frameworks; set the data-protection mode and budgets; and point a trace export at Langfuse or Phoenix while quality gates hold the line and datasets go to Ragas or DeepEval for the metrics those tools compute best. Every one of those is a setting or a request field, not a rewrite.