# Enterprise Architecture ## Design principles | Principle | How it shows up | |---|---| | **Hexagonal / ports and adapters** | `StateStore`, `Telemetry`, `AuditSink`, `Policy`, `Strategy`, `SignalExtractor` are ports. In-memory and file adapters ship in `enterprise`; `enterprise.stores` adds Redis, SQL, versioned, encrypted, batched and namespaced stores; `adapters` provides the OpenAI-compatible client, agent harnesses, MCP / A2A / framework glue, sentence-transformers and OpenTelemetry. | | **Separation of concerns** | Signals (what is asked), Policy (what is allowed), Strategies (what is good), Ensemble (decide), Execution, Learning. Each stage is a module with one job. | | **Fail fast** | `RouterBuilder.build()` validates configuration (no targets, no strategies, duplicate names) before serving traffic. | | **Fail safe** | Hard constraints are never traded off; if nothing is admissible you get `NoRouteError` with the rejection reasons, not a silent bad route. | | **Explainability by default** | Every decision carries a `RouteTrace`; every strategy returns a rationale. | | **Inductive learning** | Every strategy works for a brand-new target from its declared capabilities; learners start silent (confidence 0) and gain weight with evidence. | | **Zero-dependency core** | Pure stdlib. Deterministic given a seed. Sub-millisecond per decision. | | **Twelve-factor** | Config via YAML / env (`OSR__` overrides every tunable in `opensmartroute.settings`), secrets via env or `*_FILE`, stateless router plus pluggable state store. | ## Design patterns used - **Strategy**: `Strategy` subclasses are interchangeable scorers. - **Chain of responsibility**: `Middleware` (guard, tenant, fair share, cache, timeout, shadow, router). - **Builder**: `RouterBuilder` fluent assembly with validation. - **Observer**: `Telemetry.on_decision / on_outcome / on_error`. - **Registry**: `TargetRegistry`, `HealthRegistry`. - **Circuit breaker / bulkhead / rate limiter**: `realtime`. - **Template method**: `Policy.check` with `HealthPolicy` decorating the base policy. - **Decorator**: `HealthPolicy(inner)`, `CostAwareBandit(inner)`, every `*StateStore(inner)` wrapper. - **Facade**: `EnterpriseRouter`, `AsyncRouter`. - **Memento**: `state()/load()` on every learner; `AutoLearner` atomic snapshots. ## Component view ```mermaid flowchart TB subgraph Edge API[HTTP / gRPC / SDK call] end subgraph Middleware chain G[GuardMiddleware
size, gadget, redaction] T[TenantMiddleware
defaults, limits] F[FairShareMiddleware
dominant-resource fairness] C[CacheMiddleware
LRU + TTL] TO[TimeoutMiddleware] SH[ShadowMiddleware
shadow / A-B + SPRT] end subgraph Core router S[Signals] P[Policy + HealthPolicy] RT[Retriever
BM25 + dense, RRF] ST[Strategies
rules, capability, similarity, task table, irt, preference, linucb, markov, health, queue, bandit] E[Ensemble + Objective + calibration] D[Decision + Trace + Plan + candidate set] end subgraph Learning loop O[Outcome] AL[AutoLearner
fan-out, drift, persist] H[HealthRegistry
breaker, rate, budget] IF[InflightTracker] end subgraph Ports SS[(StateStore
memory, file, Redis, SQL
versioned, encrypted, batched)] TEL[Telemetry] AU[AuditSink] FB[(FeedbackStore)] end API --> G --> T --> F --> C --> TO --> SH --> S --> P --> RT --> ST --> E --> D D --> TEL D --> AU O --> AL --> ST O --> H --> P IF --> ST AL --> SS O --> FB ``` ## Builder surface | Method | Wires | |---|---| | `with_rules(rules)` | `RulesStrategy` (pin / prefer / avoid) evaluated first | | `with_defaults()` | `CapabilityStrategy`, `SimilarityStrategy`, `BanditStrategy` | | `with_strategy(s, weight)` | any `Strategy`, optional ensemble weight | | `with_auto_learning(state_dir)` | `IRTStrategy`, `PreferenceStrategy`, `LinUCBStrategy`, `MarkovStrategy` behind an `AutoLearner` with drift detection and atomic persistence | | `with_state_store(store, save_every)` | persist learner state through any `StateStore` instead of files | | `with_health(latency_slo_ms)` | `HealthRegistry`, `HealthPolicy`, `HealthStrategy` | | `with_queue_awareness(slo_ms)` | `InflightTracker` + `QueueAwareStrategy`; `EnterpriseRouter.execute()` brackets each call with `acquire` / `release` | | `with_calibration(conformal_alpha)` | `TemperatureScaler` for confidence, `ConformalCalibrator` for `RouteDecision.candidate_set`; `router.calibrate()` replays remembered outcomes | | `with_retrieval(narrow_above, narrow_to, skill_set_size)` | `Retriever` (BM25 + dense, reciprocal-rank fusion) for large catalogues; submodular skill-set selection | | `with_fair_share(weights, ...)` | `FairShareMiddleware` | | `with_shadow(candidate, mode)` | `ShadowMiddleware` in `shadow` (log only) or `ab` (SPRT-judged) mode; `EnterpriseRouter.learn()` feeds outcomes to it | | `with_middleware(...)`, `with_telemetry(...)`, `with_audit(sink)`, `with_feedback(store)` | chain, observers, audit trail, outcome log | | `with_llm_judge(judge, escalate_below)`, `with_objective(obj)`, `with_policy(policy)`, `with_router_options(**kw)` | judge escalation, default objective, custom policy, any other `Router` argument | `build()` validates the configuration (no targets, no strategies, duplicate strategy names) and returns an `EnterpriseRouter` with `route()`, `execute()`, `run()`, `learn()` and `health_snapshot()`. ## State stores All stores implement the three-method `StateStore` port (`get` / `put` / `delete` of JSON dicts) and compose as decorators. External clients are duck-typed, so the core imports no driver. | Store | Backend | Notes | |---|---|---| | `InMemoryStateStore` | process memory | tests, single replica | | `FileStateStore` | one JSON file per key | keys hashed (no path traversal), atomic `replace` | | `RedisStateStore(client, prefix, ttl_s)` | any client with `get` / `set(ex=)` / `delete` (redis-py, valkey, fakeredis) | shared state across replicas | | `SQLStateStore(connection, table)` | any DB-API 2.0 connection (psycopg, sqlite3, pg8000, mysql-connector) | dialect detected; `ON CONFLICT` / `ON DUPLICATE KEY` upserts; table name validated | | `VersionedStateStore(inner, current, migrations)` | wrapper | stamps `{"_schema": n}`, runs forward migrations on read, refuses newer state | | `EncryptedStateStore(inner, key \| key_env)` | wrapper (`crypto` extra) | AES-256-GCM, per-record nonce, key id in the envelope for rotation, plaintext records still readable | | `BatchedStateStore(inner, interval_s, max_pending)` | wrapper | write-behind; reads see pending writes; `flush()` on shutdown or as a context manager | | `NamespacedStateStore(inner, namespace)` | wrapper | key prefix for several routers or tenants on one backend | ```python import redis from opensmartroute.enterprise import RouterBuilder from opensmartroute.enterprise.stores import BatchedStateStore, EncryptedStateStore, RedisStateStore, VersionedStateStore store = BatchedStateStore( VersionedStateStore( EncryptedStateStore(RedisStateStore(redis.Redis.from_url(url), prefix="osr:prod:"), key_env="OSR_STATE_KEY"), current=1, ), interval_s=2.0, ) app = RouterBuilder(registry).with_defaults().with_auto_learning().with_state_store(store).build() ``` ## Operational controls - **Shadow and A/B** (`ShadowMiddleware`, `ABTest`, `SPRT`): a candidate router runs beside production. In shadow mode only agreement rate and cost delta are logged. In A/B mode a deterministic hash of the request id assigns a traffic share to the candidate and outcomes are compared with Wald's sequential probability ratio test, so a bad candidate is stopped early and a good one is promoted with a controlled error rate. `on_verdict` fires once with the summary. - **Tenant fairness** (`FairShareMiddleware`): dominant-resource fairness across tenants over a sliding window; tenants above their fair share are steered to cheaper targets (cost weight boost) and, above the hard factor, capped out of premium targets. Nobody is starved. - **Queue-aware latency** (`InflightTracker`, `QueueAwareStrategy`): live in-flight counts and observed service times feed Erlang-C (when `metadata.concurrency` is set) or Kingman G/G/1 wait estimates, so the latency in the utility is the current time-to-first-token, not the catalogue number. ## Deployment topologies 1. **Library**: import `Router` / `EnterpriseRouter` in-process. Lowest latency; state on local disk. 2. **Sidecar**: the container image (`crosrua2c6onyidguw.azurecr.io/opensmartroute:`, built from `deploy/Dockerfile`) next to each app; shared state via a `StateStore` adapter. 3. **Central control plane**: the Helm chart in `deploy/helm/opensmartroute` behind a gateway (LiteLLM, Envoy, aisix); the router returns a decision, the gateway executes. Scale horizontally; state in Redis / SQL. See [deploy/README.md](https://opensmartroute.ai/docs/deploy.md). ### Scalability notes - Routing is CPU-bound and O(|targets| x |strategies|) with tiny constants; 10k decisions/s per core is typical for <= 50 targets. Above `narrow_above` admissible targets the retriever keeps only `narrow_to` candidates, so cost stays flat for catalogues with thousands of tools. - All in-memory structures are bounded (LRU caches, capped lists, session maps cleared past a limit). - Learner updates are O(1) (bandits, IRT, BT) or O(d^2) with d ~ 21 (LinUCB, Sherman-Morrison); value iteration is bounded to 30 sweeps over a small state space. - Multi-replica consistency: learners are *eventually consistent*. Two supported patterns: 1. **Writer / readers** - one replica learns and persists through a shared `StateStore`; the others call `AutoLearner.refresh()` on a timer and adopt the snapshot wholesale (no double counting). 2. **Independent replicas** - every replica learns from the outcomes it sees and snapshots are folded together with `learning.merge_learners(local, remote)`; every learner merges sufficient statistics (`merge()` on `ThompsonBeta`, `LinUCB`, `IRTModel`, `BradleyTerry`, `MarkovChain` / `RoutingMDP`, `BanditStrategy`, `TaskTableStrategy`). Merge deltas, not the same snapshot twice. - Corrupt or incompatible persisted state is quarantined on load (`*.corrupt-` file or `quarantine/...` store key) and listed in `AutoLearner.quarantined`; the learner starts blank and routing continues. ## Reliability - Circuit breaker per target; half-open probes. - Token-bucket rate limits and rolling budgets per target (and per tenant via `TenantMiddleware`). The policy only *checks* availability for each candidate; one token is consumed for the target that is actually chosen, so evaluating 16 candidates does not drain 16 tokens. - `HealthStrategy` demotes slow/unreliable targets softly; `HealthPolicy` excludes them hard. Each target keeps a rolling window of its last 256 latencies (`LatencyWindow`; p50 / p90 / p99 in `health_snapshot()` and on the platform's Health page); once five calls are in the window the strategy judges the **p90**, so a target with a good mean and a bad tail is penalised. The SLO is the request's hard `max_latency_ms`, else its soft `preferred_max_latency_ms` (penalise, never exclude), else `with_health(latency_slo_ms=...)`. - Drift detection flags targets whose quality degrades; wire `AutoLearner.drifted` to alerts. - Atomic state writes (`os.replace`): a crash never leaves a half-written model. - Per-task resource limits (`security.limits.ResourceLimitMiddleware`) cap steps, tool calls, depth, tokens, cost and wall-clock for agentic plans. ## Observability - Tracing: every stage of every request (signals, policy, rank, plan, execute, learn, health, cache, guard, shadow, fair share, autopilot) emits spans and events into the `Tracer` - `RouterBuilder.with_tracing(...)`, `MemorySink` / `MetricsSink` / `LoggingSink` / `FileSink` / `OpenTelemetrySink`, `GET /events`, `GET /trace/{request_id}`, `X-OSR-Trace-Id`. The hosted platform exposes the same buffer per workspace (`GET /api/v1/trace/{request_id}` with the reported outcomes and the request's audit records, `GET /api/v1/events`, `/platform/dashboard/events`). See [OBSERVABILITY.md](https://opensmartroute.ai/docs/OBSERVABILITY.md). - `LoggingTelemetry`: structured JSON (request hash, never raw text). - `MetricsTelemetry`: counters, p50 / p95 / p99 routing latency, mean confidence; expose via `/metrics`. - `OpenTelemetryTelemetry` (`otel` extra): spans and the `osr.route_latency_ms` histogram. - `FileAuditSink`: hash-chained audit log (tamper-evident); the chain resumes across restarts and `FileAuditSink.verify(path)` re-walks it, naming the first edited, removed or unparseable line. Records carry the `request_id`, so a decision and the outcomes reported for it can be joined back to the request (`with_audit(sink, outcomes=True)`; the platform shows them on the trace). - `RouteTrace.explain()`: human-readable decision explanation for support tooling. - `ShadowMiddleware.summary()`, `FairShareMiddleware.snapshot()`, `InflightTracker.snapshot()` and `EnterpriseRouter.health_snapshot()` for dashboards. ## Multi-tenancy `TenantMiddleware` enforces tenant presence, applies per-tenant cost ceilings, data boundaries and deny lists. `FairShareMiddleware` keeps one tenant from crowding out the others. `TargetConstraints.tenants` restricts a target to specific tenants. Learner state can be namespaced per tenant with `NamespacedStateStore` or a per-tenant `RouterBuilder`. ## Compliance hooks - Data boundary (`public` < `private` < `on_prem`) is a hard constraint. - Region allow-lists per target. - PII never reaches a `pii_allowed=False` target; `Redactor` masks before caching / logging. - Audit trail with request id, target, confidence, tenant; no content. - Learner state encrypted at rest with `EncryptedStateStore`; key from the environment or a `*_FILE` secret via `load_secret`. - Signed MCP manifests (`adapters.mcp.verify_manifest`) and an origin policy for sensitive tool parameters (`security.provenance.OriginPolicy`) for plans that execute state-changing tools.