Enterprise Architecture

Design principles#

Principle	How it shows up
Hexagonal / ports and adapters	StateStore, Telemetry, AuditSink, Policy, Strategy, SignalExtractor are ports. In-memory and file adapters ship in enterprise; enterprise.stores adds Redis, SQL, versioned, encrypted, batched and namespaced stores; adapters provides the OpenAI-compatible client, agent harnesses, MCP / A2A / framework glue, sentence-transformers and OpenTelemetry.
Separation of concerns	Signals (what is asked), Policy (what is allowed), Strategies (what is good), Ensemble (decide), Execution, Learning. Each stage is a module with one job.
Fail fast	RouterBuilder.build() validates configuration (no targets, no strategies, duplicate names) before serving traffic.
Fail safe	Hard constraints are never traded off; if nothing is admissible you get NoRouteError with the rejection reasons, not a silent bad route.
Explainability by default	Every decision carries a RouteTrace; every strategy returns a rationale.
Inductive learning	Every strategy works for a brand-new target from its declared capabilities; learners start silent (confidence 0) and gain weight with evidence.
Zero-dependency core	Pure stdlib. Deterministic given a seed. Sub-millisecond per decision.
Twelve-factor	Config via YAML / env (OSR_<GROUP>_<FIELD> overrides every tunable in opensmartroute.settings), secrets via env or *_FILE, stateless router plus pluggable state store.

Design patterns used#

• Strategy: Strategy subclasses are interchangeable scorers.
• Chain of responsibility: Middleware (guard, tenant, fair share, cache, timeout, shadow, router).
• Builder: RouterBuilder fluent assembly with validation.
• Observer: Telemetry.on_decision / on_outcome / on_error.
• Registry: TargetRegistry, HealthRegistry.
• Circuit breaker / bulkhead / rate limiter: realtime.
• Template method: Policy.check with HealthPolicy decorating the base policy.
• Decorator: HealthPolicy(inner), CostAwareBandit(inner), every *StateStore(inner) wrapper.
• Facade: EnterpriseRouter, AsyncRouter.
• Memento: state()/load() on every learner; AutoLearner atomic snapshots.

Component view#

flowchart TB
    subgraph Edge
        API[HTTP / gRPC / SDK call]
    end
    subgraph Middleware chain
        G[GuardMiddleware<br/>size, gadget, redaction]
        T[TenantMiddleware<br/>defaults, limits]
        F[FairShareMiddleware<br/>dominant-resource fairness]
        C[CacheMiddleware<br/>LRU + TTL]
        TO[TimeoutMiddleware]
        SH[ShadowMiddleware<br/>shadow / A-B + SPRT]
    end
    subgraph Core router
        S[Signals]
        P[Policy + HealthPolicy]
        RT[Retriever<br/>BM25 + dense, RRF]
        ST[Strategies<br/>rules, capability, similarity, task table, irt, preference, linucb, markov, health, queue, bandit]
        E[Ensemble + Objective + calibration]
        D[Decision + Trace + Plan + candidate set]
    end
    subgraph Learning loop
        O[Outcome]
        AL[AutoLearner<br/>fan-out, drift, persist]
        H[HealthRegistry<br/>breaker, rate, budget]
        IF[InflightTracker]
    end
    subgraph Ports
        SS[(StateStore<br/>memory, file, Redis, SQL<br/>versioned, encrypted, batched)]
        TEL[Telemetry]
        AU[AuditSink]
        FB[(FeedbackStore)]
    end
    API --> G --> T --> F --> C --> TO --> SH --> S --> P --> RT --> ST --> E --> D
    D --> TEL
    D --> AU
    O --> AL --> ST
    O --> H --> P
    IF --> ST
    AL --> SS
    O --> FB

Builder surface#

Method	Wires
with_rules(rules)	RulesStrategy (pin / prefer / avoid) evaluated first
with_defaults()	CapabilityStrategy, SimilarityStrategy, BanditStrategy
with_strategy(s, weight)	any Strategy, optional ensemble weight
with_auto_learning(state_dir)	IRTStrategy, PreferenceStrategy, LinUCBStrategy, MarkovStrategy behind an AutoLearner with drift detection and atomic persistence
with_state_store(store, save_every)	persist learner state through any StateStore instead of files
with_health(latency_slo_ms)	HealthRegistry, HealthPolicy, HealthStrategy
with_queue_awareness(slo_ms)	InflightTracker + QueueAwareStrategy; EnterpriseRouter.execute() brackets each call with acquire / release
with_calibration(conformal_alpha)	TemperatureScaler for confidence, ConformalCalibrator for RouteDecision.candidate_set; router.calibrate() replays remembered outcomes
with_retrieval(narrow_above, narrow_to, skill_set_size)	Retriever (BM25 + dense, reciprocal-rank fusion) for large catalogues; submodular skill-set selection
with_fair_share(weights, ...)	FairShareMiddleware
with_shadow(candidate, mode)	ShadowMiddleware in shadow (log only) or ab (SPRT-judged) mode; EnterpriseRouter.learn() feeds outcomes to it
with_middleware(...), with_telemetry(...), with_audit(sink), with_feedback(store)	chain, observers, audit trail, outcome log
with_llm_judge(judge, escalate_below), with_objective(obj), with_policy(policy), with_router_options(**kw)	judge escalation, default objective, custom policy, any other Router argument

build() validates the configuration (no targets, no strategies, duplicate strategy names) and returns an EnterpriseRouter with route(), execute(), run(), learn() and health_snapshot().

State stores#

All stores implement the three-method StateStore port (get / put / delete of JSON dicts) and compose as decorators. External clients are duck-typed, so the core imports no driver.

Store	Backend	Notes
InMemoryStateStore	process memory	tests, single replica
FileStateStore	one JSON file per key	keys hashed (no path traversal), atomic replace
RedisStateStore(client, prefix, ttl_s)	any client with get / set(ex=) / delete (redis-py, valkey, fakeredis)	shared state across replicas
SQLStateStore(connection, table)	any DB-API 2.0 connection (psycopg, sqlite3, pg8000, mysql-connector)	dialect detected; ON CONFLICT / ON DUPLICATE KEY upserts; table name validated
VersionedStateStore(inner, current, migrations)	wrapper	stamps {"_schema": n}, runs forward migrations on read, refuses newer state
EncryptedStateStore(inner, key | key_env)	wrapper (crypto extra)	AES-256-GCM, per-record nonce, key id in the envelope for rotation, plaintext records still readable
BatchedStateStore(inner, interval_s, max_pending)	wrapper	write-behind; reads see pending writes; flush() on shutdown or as a context manager
NamespacedStateStore(inner, namespace)	wrapper	key prefix for several routers or tenants on one backend

import redis
from opensmartroute.enterprise import RouterBuilder
from opensmartroute.enterprise.stores import BatchedStateStore, EncryptedStateStore, RedisStateStore, VersionedStateStore

store = BatchedStateStore(
    VersionedStateStore(
        EncryptedStateStore(RedisStateStore(redis.Redis.from_url(url), prefix="osr:prod:"), key_env="OSR_STATE_KEY"),
        current=1,
    ),
    interval_s=2.0,
)
app = RouterBuilder(registry).with_defaults().with_auto_learning().with_state_store(store).build()

Operational controls#

• Shadow and A/B (ShadowMiddleware, ABTest, SPRT): a candidate router runs beside production. In shadow mode only agreement rate and cost delta are logged. In A/B mode a deterministic hash of the request id assigns a traffic share to the candidate and outcomes are compared with Wald's sequential probability ratio test, so a bad candidate is stopped early and a good one is promoted with a controlled error rate. on_verdict fires once with the summary.
• Tenant fairness (FairShareMiddleware): dominant-resource fairness across tenants over a sliding window; tenants above their fair share are steered to cheaper targets (cost weight boost) and, above the hard factor, capped out of premium targets. Nobody is starved.
• Queue-aware latency (InflightTracker, QueueAwareStrategy): live in-flight counts and observed service times feed Erlang-C (when metadata.concurrency is set) or Kingman G/G/1 wait estimates, so the latency in the utility is the current time-to-first-token, not the catalogue number.

Deployment topologies#

1. Library: import Router / EnterpriseRouter in-process. Lowest latency; state on local disk.
2. Sidecar: the container image (ghcr.io/isathish/opensmartroute:<version>, built from deploy/Dockerfile) next to each app; shared state via a StateStore adapter.
3. Central control plane: the Helm chart in deploy/helm/opensmartroute behind a gateway (LiteLLM, Envoy, aisix); the router returns a decision, the gateway executes. Scale horizontally; state in Redis / SQL. See deploy/README.md.

Scalability notes#

• Routing is CPU-bound and O(|targets| x |strategies|) with tiny constants; 10k decisions/s per core is typical for <= 50 targets. Above narrow_above admissible targets the retriever keeps only narrow_to candidates, so cost stays flat for catalogues with thousands of tools.
• All in-memory structures are bounded (LRU caches, capped lists, session maps cleared past a limit).
• Learner updates are O(1) (bandits, IRT, BT) or O(d^2) with d ~ 21 (LinUCB, Sherman-Morrison); value iteration is bounded to 30 sweeps over a small state space.
• Multi-replica consistency: learners are eventually consistent. Two supported patterns:
  1. Writer / readers - one replica learns and persists through a shared StateStore; the others call AutoLearner.refresh() on a timer and adopt the snapshot wholesale (no double counting).
  2. Independent replicas - every replica learns from the outcomes it sees and snapshots are folded together with learning.merge_learners(local, remote); every learner merges sufficient statistics (merge() on ThompsonBeta, LinUCB, IRTModel, BradleyTerry, MarkovChain / RoutingMDP, BanditStrategy, TaskTableStrategy). Merge deltas, not the same snapshot twice.
• Corrupt or incompatible persisted state is quarantined on load (*.corrupt-<ts> file or quarantine/... store key) and listed in AutoLearner.quarantined; the learner starts blank and routing continues.

Reliability#

• Circuit breaker per target; half-open probes.
• Token-bucket rate limits and rolling budgets per target (and per tenant via TenantMiddleware). The policy only checks availability for each candidate; one token is consumed for the target that is actually chosen, so evaluating 16 candidates does not drain 16 tokens.
• HealthStrategy demotes slow/unreliable targets softly; HealthPolicy excludes them hard. Each target keeps a rolling window of its last 256 latencies (LatencyWindow; p50 / p90 / p99 in health_snapshot() and on the platform's Health page); once five calls are in the window the strategy judges the p90, so a target with a good mean and a bad tail is penalised. The SLO is the request's hard max_latency_ms, else its soft preferred_max_latency_ms (penalise, never exclude), else with_health(latency_slo_ms=...).
• Drift detection flags targets whose quality degrades; wire AutoLearner.drifted to alerts.
• Atomic state writes (os.replace): a crash never leaves a half-written model.
• Per-task resource limits (security.limits.ResourceLimitMiddleware) cap steps, tool calls, depth, tokens, cost and wall-clock for agentic plans.

Observability#

• Tracing: every stage of every request (signals, policy, rank, plan, execute, learn, health, cache, guard, shadow, fair share, autopilot) emits spans and events into the Tracer - RouterBuilder.with_tracing(...), MemorySink / MetricsSink / LoggingSink / FileSink / OpenTelemetrySink, GET /events, GET /trace/{request_id}, X-OSR-Trace-Id. The hosted platform exposes the same buffer per workspace (GET /api/v1/trace/{request_id} with the reported outcomes and the request's audit records, GET /api/v1/events, /platform/dashboard/events). See OBSERVABILITY.md.
• LoggingTelemetry: structured JSON (request hash, never raw text).
• MetricsTelemetry: counters, p50 / p95 / p99 routing latency, mean confidence; expose via /metrics.
• OpenTelemetryTelemetry (otel extra): spans and the osr.route_latency_ms histogram.
• FileAuditSink: hash-chained audit log (tamper-evident); the chain resumes across restarts and FileAuditSink.verify(path) re-walks it, naming the first edited, removed or unparseable line. Records carry the request_id, so a decision and the outcomes reported for it can be joined back to the request (with_audit(sink, outcomes=True); the platform shows them on the trace).
• RouteTrace.explain(): human-readable decision explanation for support tooling.
• ShadowMiddleware.summary(), FairShareMiddleware.snapshot(), InflightTracker.snapshot() and EnterpriseRouter.health_snapshot() for dashboards.

Multi-tenancy#

TenantMiddleware enforces tenant presence, applies per-tenant cost ceilings, data boundaries and deny lists. FairShareMiddleware keeps one tenant from crowding out the others. TargetConstraints.tenants restricts a target to specific tenants. Learner state can be namespaced per tenant with NamespacedStateStore or a per-tenant RouterBuilder.

Compliance hooks#

• Data boundary (public < private < on_prem) is a hard constraint.
• Region allow-lists per target.
• PII never reaches a pii_allowed=False target; Redactor masks before caching / logging.
• Audit trail with request id, target, confidence, tenant; no content.
• Learner state encrypted at rest with EncryptedStateStore; key from the environment or a *_FILE secret via load_secret.
• Signed MCP manifests (adapters.mcp.verify_manifest) and an origin policy for sensitive tool parameters (security.provenance.OriginPolicy) for plans that execute state-changing tools.