FinOps for AI: A Guide to Cost and Governance
OpenSmartRoute provides a decision layer for LLMs that includes a FinOps module. It tracks per-request provider costs and customer charges side by side, enforces budgets per workspace, tenant and API key, and analyzes provider invoices. This guide explains how the system handles current costs and what advanced features are planned.
What FinOps Features Exist Today
The platform offers a range of tools to manage AI spend. These tools are designed to give visibility into costs and enforce limits at the workspace level.
Per-Request Cost Tracking
The system tracks costs per request. It displays the provider cost and the customer charge side by side. This allows for a direct comparison of what is billed by the provider versus what is charged to the customer.
Budgets and Limits
Budgets can be defined per workspace, tenant, and API key. These budgets support both daily and monthly timeframes. When a limit is reached, the system enforces a hold on further requests. Additionally, per-key spend limits can be set in cents to prevent unexpected overages.
Provider Invoice Analysis
The system includes a bill analysis feature for provider invoices. This allows for a detailed review of the provider's billing data, helping to identify discrepancies or areas for negotiation.
Savings Ledger
A savings ledger is available to track savings against a named baseline. This helps in understanding the financial impact of routing decisions and cost-saving measures.
Insights and Anomaly Detection
The system provides insights findings to identify anomalies. For example, a day above the mean plus two standard deviations is flagged as an anomaly. The system also names the driver of the anomaly, helping teams to quickly identify the cause.
Advanced FinOps Features on the Roadmap
While the current FinOps module covers the basics, the roadmap includes advanced features for deeper financial control and forecasting.
Cost Allocation Dimensions
Advanced FinOps plans to add cost-allocation dimensions beyond tenant and key. These dimensions include cost centers, projects, and tags on a request. This allows for more granular reporting and showback and chargeback reports per dimension.
