Manju Rajashekhar explains how the actor model solves runtime challenges for production AI agents. He argues that treating agents as distributed systems improves reliability and state management.
Manju Rajashekhar is the founder and CEO of Mad Labs. He previously worked on Twitter's early engineering team. He helped solve the fail-mail problem there. He scaled caching systems from 50 million to 200 million users. He co-founded Blackbird AI before it was acquired by Etsy. He spent eight years leading search and recommendation engines. He now runs an agentic platform focused on reliable execution.
QCon AI is a practitioner-led event focused on engineering discipline. It focuses on scaling workloads safely. It provides access to architectural playbooks used in production. It shares failure metrics that peer organizations use.
Manju Rajashekhar started his career more than a decade ago. He worked at Twitter on distributed caching systems. He was a prolific open-source contributor around that time. He wrote software that is still used today at companies like Uber. He also contributed to Wikipedia to scale their caching systems. Google Cloud and Azure document his work as a reference implementation.
After Twitter, he founded a company in the deep learning space. His product was search as a service and recommendation as a service. That company was acquired by Etsy. He spent eight years as a VP of engineering there. He ran all of their AI and machine learning functions. This included search, ads, recommendation, and the personalization engine.
Most recently, he started a company called Mad Labs. He has a couple of wonderful colleagues of his. They are building agents with reliable execution over long horizons. They are very token efficient. They have a strong governance structure around their work.
Manju Rajashekhar is excited to talk about actors for agents. He realized not much is spoken about this topic. He feels this is crazy because the actor model fits well to agent runtime. He hopes to convince you that the actor model is not an old concurrency idea. He believes it maps well to agentic system design. You can start using it as a tool in your day-to-day work.
Instead of starting with boring definitions, he starts with a story. The story is when your demo leaves the demo environment. It makes contact with production. Many people built a demo with agents over the past 12 months. The architecture for these demos is usually simple. There is a prompt chain over the agentic loop. There is some application glue code. There is a cue for slow work, model calls, and tool calls. You finally get an answer.
The architecture feels pretty natural for a demo. It proves the models can reason and tools can be called. It produces a demo that makes agentic systems feel real. Now the demo gets super popular. Friends, colleagues, and relatives start using it. The demo makes its first contact with production.
Production actually changes the shape of the problem. Tool calls now take longer than expected. Anthropic decided to tell you that you exhausted your token. They are going to rate limit you. Users start asking follow-up questions while your agent is still working. Workers get restarted halfway through the execution. Suddenly the hard part is no longer about reasoning. The hard part becomes the runtime of production agents.
That is exactly what Manju wants to talk about today. He wants to know how to model the runtime of production agents. Really, what happens once your demo goes from demo to production. The shape of the problem changes.
Let us look at the shapes that change. The first thing that changes is the notion of time. In the demo environment, your tool returns pretty quickly. It feels like a normal request-response cycle. Everything is happening really quickly. That is no longer true in the production setting. You get rate limited. Tool calls can take longer. Sometimes you have to get back to the user for feedback. The user can only respond back tomorrow because they are not available. A function call ends up being a poor abstraction for work that begins now and results later.
The second thing that changes is just the interaction. Users do not stop existing while your agent is still working. They ask for follow-up questions. They ask for status. They request for partial results. They change the execution of the agent based on new information. Your agent loop is no longer a single request-response loop. It is a running process that is just receiving information.
The third thing that changes is delegation. Agentic systems do not merely execute a series of steps. They are creating the entire steps on the fly. They are learning more information and coming up with the execution of work. The structure of execution no longer follows a straight line. It ends up becoming a tree, or more precisely, a graph.
The fourth thing that changes is just failure. There are different failure models that happen with agents. Tool calls fail. Model calls fail. Worker processes get restarted. External APIs get rate limited. Queues can re-deliver messages. The list goes on and on. In the demo environment, you are treating exceptions as an afterthought. In the production setting, it is just a part of the normal operating environment.
Because the production changed the shape of the problem, the architecture has started to grow sideways. Your chat lives in one place. Your workflow state lives somewhere else. The queue knows that the work exists. But it does not know what the user really intended. The database records progress. But it does not always record the continuation of work. The retry state is scattered across different places. The recovery condition just becomes a second implementation of your workload.
The code still works, but the system becomes hard to reason about. The problem is not any of these single components is wrong. Queues make sense. Databases are useful. You do need workers for parallelism and concurrency. The problem is that the system does not have a single execution model. The agent gets treated as a chat in one part of the system. It gets treated as a workflow or a background job in a different part. It gets treated as an entire recovery process in yet another part.
That is the production pain of the runtime of these agents. The conceptual unit of work that agents end up carrying out is actually split across multiple runtime concepts. This happens for one main reason. We design these agents as a reasoning problem with LLM and tool calls. This ends up hiding the actual problem. The actual problem is the runtime of these production agents.
The moment we start looking at production agents as a runtime problem, a familiar system starts showing up. Many of you might be familiar with this. Production agents are just distributed systems in disguise. They are stateful. They are concurrent. They are long-running. They communicate with other components. So they are distributed in nature. They can fail in parts. They can restart independently. They must accept new information while the work is still ongoing.
Once you see agents as a distributed system, a more fundamental question starts showing up. The question is how do you model the runtime of a system like this. Our answer to that is the actor style reasoning fits really well with the runtime of production agents. I would like to convince you that is the case today with this talk.
Before we talk about agents, let us walk through the actor idea from first principles. The actor idea is not new. It was discovered as part of the AI and concurrency research in the 1970s. It is Erlang and OTP that made the actor style thinking operational in a serious way. There is a great story about Erlang. It is the story of nine nines, or nine nines of availability. Erlang proved it by running a system over 2 million lines of Erlang code. The system was available for 20 years with nine nines of availability.
That is crazy to think about it. What Erlang showed is that you can build really reliable systems. These systems are extremely concurrent, fault tolerant, and always on. We want to bring the same idea to agents. To understand the actor model from first principles, let us ask how we normally think about computation. Most of us, when we learn to program, think about software as a single flow of control. A function calls another function. You receive results. You update state. Then you move on. This model is not wrong. It is a great mental model. It is super useful. It quietly assumes a world where control flow is local. Failure is either absent or is handled by the caller. Real systems are not like that. Real systems end up having multiple flows of control. Many things happen at once. Parts need to communicate without sharing all of their internals.
Parts can fail independently. The moment we enter this world, which is the real world, the hard question changes. The hard question is not merely, which function should I call? The hard questions are, who owns the state? How do other parts of the system communicate with it? What happens if one part fails? The actor model is one answer to these questions.
In the actor model, the basic unit is actually not a function. It is not a class. It is not a thread. The basic unit is an actor, which is an independent unit of computation. The operative word here being independent. The actor has a private state that only the actor has access to. The actor can receive messages. Because it can receive messages, it must be addressable. This gives it a mailbox. When it receives messages, the actor decides completely based on its local state what to do next.
It can update its state. It can send messages to a different actor. It can create child actors. It can fail. It can stop. It can restart. All of those are options that are available to the actor. The actor ends up starting to behave like this little machine. A machine that has an address, which gives it a mailbox. A machine that has a private state. The machine has an encapsulated behavior of how it should act purely based on its internal state and the messages it receives.
This leads us to the first foundational design principle of an actor. The state has an owner. We are all familiar with shared memory. But in the actor model, state is owned by a single actor. This prevents conflicts and makes reasoning about state much easier.
Foundational Design Principles of an Actor
The second principle is message passing. Actors communicate only by sending messages. They do not share memory directly. This makes concurrency safe by design. If two actors try to update the same data, they must do it through a message. This ensures that the state update is atomic.
The third principle is supervision. Actors can be supervised by other actors. If an actor fails, the supervisor can restart it. This allows the system to recover from failures automatically. It does not require manual intervention. The system keeps running even when parts fail.
The fourth principle is isolation. Each actor runs in its own isolation. It cannot see the internal state of other actors. This limits the damage if one actor fails. It also makes debugging easier because you can focus on one actor at a time.
These principles make the actor model a strong fit for production agents. Agents need to handle long-running tasks. They need to manage complex state. They need to recover from failures. The actor model provides a natural way to do all of this.
Why it matters
Reliability is the main reason this matters to the reader. Production agents need to run without constant manual intervention. The actor model helps achieve this reliability. It reduces the time engineers spend fixing broken workflows. It allows teams to ship agents faster with confidence.
Cost is another major factor. Agents often run for long periods. Wasting tokens or running inefficient models costs money. The actor model helps optimize resource usage. It ensures that only necessary work is done. It prevents wasted computation on failed paths.
Safety is also critical. Agents handle sensitive data and user inputs. The actor model enforces strict boundaries on state access. This reduces the risk of data leaks or prompt injections. It makes the system safer by default.
New capabilities emerge from this approach. Teams can build agents that handle complex, multi-step tasks. They can create systems that scale to millions of users. They can deploy agents that are resilient to network issues. The actor model unlocks these capabilities for production use.
How OpenSmartRoute helps
A team that routes its requests through OpenSmartRoute gains significant benefits from this news. They can treat their agent infrastructure like a distributed system. They can apply actor-like patterns to their routing logic. This improves the reliability of their entire agentic workflow.
The team would set up a catalogue of models, agents, tools, and skills. They would define rules for how requests are routed. They would score every candidate on quality, cost, speed, and safety. They would set weights per request to prioritize the best fit. Hard rules would pin specific requests to specific models. For example, text with personal data would stay on an on-premises model. A region or a cost cap would never be crossed.
The system would learn from outcomes. A model that answers well would get more traffic. One that fails would get less. A new model would be one catalogue entry. It would compete on the next request. Nothing else would change in the app. The system would work with any OpenAI-compatible provider. It would support open-weight models served locally. It would support MCP tools and A2A agents.
A savings ledger would show what each routed request cost next to what the most expensive model would have cost. An input guard would spot prompt injection and personal data before a request leaves. The 'osr eval' tool would measure routing accuracy on the team's own prompts. It could fail a build when it drops below a threshold. The hosted platform would keep a models catalogue with prices and public rankings built from real traffic.
What to do
Start by reading the presentation by Manju Rajashekhar. Look for how he describes the shift from demo to production. Identify the four key changes in the runtime. Note how the actor model solves the concurrency and state problems.
Next, audit your current agent architecture. Check where state is stored. Check how failures are handled. Check if your system has a single execution model. Look for scattered retry states or conflicting views of the workflow.
Then, consider adopting the actor model for your next agent project. Evaluate if your team has the skills to implement it. Look at libraries that support the actor model. Compare them against your current stack.
Finally, integrate OpenSmartRoute into your pipeline. Define your catalogue of models and tools. Set up your scoring rules for quality, cost, speed, and safety. Configure your hard rules for data privacy and cost caps. Run 'osr eval' to ensure your routing is accurate. Monitor your savings ledger to track costs. Use the input guard to protect your data. This setup will give you the reliability and control needed for production.
How OpenSmartRoute helps
Teams using OpenSmartRoute gain immediate reliability by treating agents as distributed systems. The router scores every candidate on quality, cost, speed and safety before sending a request. Manju Rajashekhar argues this approach improves state management and reduces runtime failures.
The platform sets up a catalogue where a new model is one entry that competes on the next request. An input guard spots prompt injection and personal data before a request leaves the system. The savings ledger shows what each routed request cost next to the most expensive model.
This setup allows the team to learn from outcomes without changing the app. A model that answers well gets more traffic, while one that fails gets less. The hosted platform keeps a models catalogue with prices and public rankings built from real traffic.