Enterprise Architecture

Design principles

Design patterns used

• Strategy: Strategy subclasses are interchangeable scorers.
• Chain of responsibility: Middleware (guard, tenant, fair share, cache, timeout, shadow, router).
• Builder: RouterBuilder fluent assembly with validation.
• Observer: Telemetry.on_decision / on_outcome / on_error.
• Registry: TargetRegistry, HealthRegistry.
• Circuit breaker / bulkhead / rate limiter: realtime.
• Template method: Policy.check with HealthPolicy decorating the base policy.
• Decorator: HealthPolicy(inner), CostAwareBandit(inner), every *StateStore(inner) wrapper.
• Facade: EnterpriseRouter, AsyncRouter.
• Memento: state()/load() on every learner; AutoLearner atomic snapshots.

Component view

flowchart TB
    subgraph Edge
        API[HTTP / gRPC / SDK call]
    end
    subgraph Middleware chain
        G[GuardMiddleware<br/>size, gadget, redaction]
        T[TenantMiddleware<br/>defaults, limits]
        F[FairShareMiddleware<br/>dominant-resource fairness]
        C[CacheMiddleware<br/>LRU + TTL]
        TO[TimeoutMiddleware]
        SH[ShadowMiddleware<br/>shadow / A-B + SPRT]
    end
    subgraph Core router
        S[Signals]
        P[Policy + HealthPolicy]
        RT[Retriever<br/>BM25 + dense, RRF]
        ST[Strategies<br/>rules, capability, similarity, task table, irt, preference, linucb, markov, health, queue, bandit]
        E[Ensemble + Objective + calibration]
        D[Decision + Trace + Plan + candidate set]
    end
    subgraph Learning loop
        O[Outcome]
        AL[AutoLearner<br/>fan-out, drift, persist]
        H[HealthRegistry<br/>breaker, rate, budget]
        IF[InflightTracker]
    end
    subgraph Ports
        SS[(StateStore<br/>memory, file, Redis, SQL<br/>versioned, encrypted, batched)]
        TEL[Telemetry]
        AU[AuditSink]
        FB[(FeedbackStore)]
    end
    API --> G --> T --> F --> C --> TO --> SH --> S --> P --> RT --> ST --> E --> D
    D --> TEL
    D --> AU
    O --> AL --> ST
    O --> H --> P
    IF --> ST
    AL --> SS
    O --> FB

Builder surface

build() validates the configuration (no targets, no strategies, duplicate strategy names) and returns an EnterpriseRouter with route(), execute(), run(), learn() and health_snapshot().

State stores

All stores implement the three-method StateStore port (get / put / delete of JSON dicts) and compose as decorators. External clients are duck-typed, so the core imports no driver.

import redis
from opensmartroute.enterprise import RouterBuilder
from opensmartroute.enterprise.stores import (
    BatchedStateStore,
    EncryptedStateStore,
    RedisStateStore,
    VersionedStateStore,
)

store = BatchedStateStore(
    VersionedStateStore(
        EncryptedStateStore(RedisStateStore(redis.Redis.from_url(url), prefix="osr:prod:"), key_env="OSR_STATE_KEY"),
        current=1,
    ),
    interval_s=2.0,
)
app = RouterBuilder(registry).with_defaults().with_auto_learning().with_state_store(store).build()

Operational controls

• Shadow and A/B (ShadowMiddleware, ABTest, SPRT): a candidate router runs beside production. In shadow mode only agreement rate and cost delta are logged. In A/B mode a deterministic hash of the request id assigns a traffic share to the candidate and outcomes are compared with Wald's sequential probability ratio test, so a bad candidate is stopped early and a good one is promoted with a controlled error rate. on_verdict fires once with the summary.
• Tenant fairness (FairShareMiddleware): dominant-resource fairness across tenants over a sliding window; tenants above their fair share are steered to cheaper targets (cost weight boost) and, above the hard factor, capped out of premium targets. Nobody is starved.
• Queue-aware latency (InflightTracker, QueueAwareStrategy): live in-flight counts and observed service times feed Erlang-C (when metadata.concurrency is set) or Kingman G/G/1 wait estimates, so the latency in the utility is the current time-to-first-token, not the catalogue number.

Deployment topologies

1. Library: import Router / EnterpriseRouter in-process. Lowest latency; state on local disk.
2. Sidecar: the container image (crosrua2c6onyidguw.azurecr.io/opensmartroute:<version>, built from deploy/Dockerfile) next to each app; shared state via a StateStore adapter.
3. Central control plane: the Helm chart in deploy/helm/opensmartroute behind a gateway (LiteLLM, Envoy, aisix); the router returns a decision, the gateway executes. Scale horizontally; state in Redis / SQL. See Deploy with Docker and Helm.

Scalability notes

• Routing is CPU-bound and O(|targets| x |strategies|) with tiny constants; 10k decisions/s per core is typical for <= 50 targets. Above narrow_above admissible targets the retriever keeps only narrow_to candidates, so cost stays flat for catalogues with thousands of tools.
• All in-memory structures are bounded (LRU caches, capped lists, session maps cleared past a limit).
• Learner updates are O(1) (bandits, IRT, BT) or O(d^2) with d ~ 21 (LinUCB, Sherman-Morrison); value iteration is bounded to 30 sweeps over a small state space.
• Multi-replica consistency: learners are eventually consistent. Two supported patterns:
  1. Writer / readers - one replica learns and persists through a shared StateStore; the others call AutoLearner.refresh() on a timer and adopt the snapshot wholesale (no double counting).
  2. Independent replicas - every replica learns from the outcomes it sees and snapshots are folded together with learning.merge_learners(local, remote); every learner merges sufficient statistics (merge() on ThompsonBeta, LinUCB, IRTModel, BradleyTerry, MarkovChain / RoutingMDP, BanditStrategy, TaskTableStrategy). Merge deltas, not the same snapshot twice. The routing SLM is outside both patterns: ContrastiveStrategy / SLMStrategy declare persist = False, so the learner never snapshots, refreshes or resets their encoder - it is promoted and rolled back through its model file (osr improve, the autopilot, the platform's SLM service) and every replica reloads that file. A strategy of your own that owns its model's persistence sets the same attribute.
• Corrupt or incompatible persisted state is quarantined on load (*.corrupt-<ts> file or quarantine/... store key) and listed in AutoLearner.quarantined; the learner starts blank and routing continues.

Reliability

• Circuit breaker per target; half-open probes.
• Token-bucket rate limits and rolling budgets per target (and per tenant via TenantMiddleware). The policy only checks availability for each candidate; one token is consumed for the target that is actually chosen, so evaluating 16 candidates does not drain 16 tokens.
• HealthStrategy demotes slow/unreliable targets softly; HealthPolicy excludes them hard. Each target keeps a rolling window of its last 256 latencies (LatencyWindow; p50 / p90 / p99 in health_snapshot() and on the platform's Health page); once five calls are in the window the strategy judges the p90, so a target with a good mean and a bad tail is penalised. The SLO is the request's hard max_latency_ms, else its soft preferred_max_latency_ms (penalise, never exclude), else with_health(latency_slo_ms=...).
• Drift detection flags targets whose quality degrades; wire AutoLearner.drifted to alerts.
• Atomic state writes (os.replace): a crash never leaves a half-written model.
• Per-task resource limits (security.limits.ResourceLimitMiddleware) cap steps, tool calls, depth, tokens, cost and wall-clock for agentic plans.

Observability

• Tracing: every stage of every request (signals, policy, rank, plan, execute, learn, health, cache, guard, shadow, fair share, autopilot) emits spans and events into the Tracer - RouterBuilder.with_tracing(...), MemorySink / MetricsSink / LoggingSink / FileSink / OpenTelemetrySink, GET /events, GET /trace/{request_id}, X-OSR-Trace-Id. The hosted platform exposes the same buffer per workspace (GET /api/v1/trace/{request_id} with the reported outcomes and the request's audit records, GET /api/v1/events, /platform/dashboard/events). See Tracing and observability.
• LoggingTelemetry: structured JSON (request hash, never raw text).
• MetricsTelemetry: counters, p50 / p95 / p99 routing latency, mean confidence; expose via /metrics.
• OpenTelemetryTelemetry (otel extra): spans and the osr.route_latency_ms histogram.
• FileAuditSink: hash-chained audit log (tamper-evident); the chain resumes across restarts and FileAuditSink.verify(path) re-walks it, naming the first edited, removed or unparseable line. Records carry the request_id, so a decision and the outcomes reported for it can be joined back to the request (with_audit(sink, outcomes=True); the platform shows them on the trace).
• RouteTrace.explain(): human-readable decision explanation for support tooling.
• ShadowMiddleware.summary(), FairShareMiddleware.snapshot(), InflightTracker.snapshot() and EnterpriseRouter.health_snapshot() for dashboards.

Multi-tenancy

TenantMiddleware enforces tenant presence, applies per-tenant cost ceilings, data boundaries and deny lists. FairShareMiddleware keeps one tenant from crowding out the others. TargetConstraints.tenants restricts a target to specific tenants. Learner state can be namespaced per tenant with NamespacedStateStore or a per-tenant RouterBuilder.

Compliance hooks

• Data boundary (public < private < on_prem) is a hard constraint.
• Region allow-lists per target.
• PII never reaches a pii_allowed=False target; Redactor masks before caching / logging.
• Audit trail with request id, target, confidence, tenant; no content.
• Learner state encrypted at rest with EncryptedStateStore; key from the environment or a *_FILE secret via load_secret.
• Signed MCP manifests (adapters.mcp.verify_manifest) and an origin policy for sensitive tool parameters (security.provenance.OriginPolicy) for plans that execute state-changing tools.