Agents Need Smarter Retrieval to Save Context Space

Add a Retrieval Tool for Semantic Search

Grep is great when you know what words to look for. It only matches the exact text you give it. That is a problem for searching a news archive for anything new. If an agent greps for "Company A funding," it misses specific articles. An article titled "Company A closes $50M Series B" is exactly what it wants. It misses this because the words do not match exactly. To search by meaning instead of exact words, you add a retrieval tool. This tool is backed by a vector store. A retrieval tool is any tool the agent can call to fetch information. It takes a query and looks it up in some data source. It returns what it finds, like the closest matches. How a retrieval tool matches depends on the data source behind it. Backed by a vector store, it matches by meaning. A retrieval tool can sit on top of other data sources too. It can use a SQL database for structured lookups. It can use a search API for fresh information from the web. It can use an MCP server for tools someone else already built. For example, an MCP server might handle a documentation search. Many retrieval setups search once at the start, then answer. This works for an FAQ bot using 2-step RAG. It does not work for our competitive analysis agent. Let us say halfway through researching Company C, it finds an article about a product launch. It did not know about this product at the start. Now it needs that product's pricing. With only one search, the report goes out without this information. Research agents need agentic retrieval instead of 2-step RAG. The agent decides when to search. It decides what to search for. It decides whether it needs more information. With Deep Agents, this works by default. The agent runs in a loop. It can call any tool as many times as it needs. To help the agent search well, you can package your search know-how as a skill. A skill is a set of instructions for a specific task. It includes which index to use. It includes how to phrase queries. It includes how to format citations. The agent only sees each skill's name and a short description until a task calls for it. Then it loads the full instructions. Niche know-how is there when it is needed. It does not take up context the rest of the time. This technique is called progressive disclosure.

The Retrieve Offload Delegate Pattern

However, despite optimizing the search process, even the right search can return too much. Our agent needs just one number from the market data export. It needs Company B's quarterly revenue. Loading all 40,000 rows to find it would fill its context window. Our retrieval tool has the same problem. If it returns every news article straight to the model, four results per search across five companies adds up fast. The filesystem tools are built to return as little as possible. Read_file reads a file one page at a time. It reads 100 lines by default. The agent can set an offset to choose which line to start from. It can set a limit to choose how many lines to read. This allows the agent to jump to the part it needs. So to find Company B's revenue, the agent greps for it first. Then it reads just the few lines around the match. Let the harness offload oversized results. Sometimes a tool returns more than expected anyway. When a tool result is too large, over about 20,000 tokens, the harness saves it to a file. It saves the file under /large_tool_results/. It gives the agent the path plus a short preview. The preview includes the first and last few lines. The agent can then read back just the parts it needs. It uses read_file's offset and limit to do this. Have your retrieval tool return file paths. You can apply the same idea to your own retrieval tool. Do not wait for the size limit. Have it write each result to a file. Have it return only the paths. This is the first part of a pattern called retrieve, offload, delegate. LangChain's RAG guide walks through this in full. Here is a simplified retrieval tool. It runs the search and saves each result to a file. It hands back only the list of file paths. Now the main agent sees four short paths instead of four full articles. Small results do not help if they are the wrong results. Any of those four files could be about a different company with a similar name. A story from two years ago looks just as convincing to a model as the right answer. Off-topic or outdated text looks just as convincing. The goal is to keep it out of the main agent's context entirely. However, the main agent still has to read those files to find out which ones matter. Every off-topic article it opens does land in its context. That is where the delegate part of the pattern comes in. A subagent reads each retrieved file and returns only the key facts. Define the subagent. The description tells the main agent when to use it. It says one file per call. The system prompt tells the subagent what to do with that file. Give the main agent the retrieval tool and the subagent. Pass the same backend you used in search_news. This ensures the subagent can read the files the tool saved. Search: the main agent calls search_news and gets back four paths. Delegate: it sends each path to a chunk-analyst subagent. They can run in parallel. Synthesize: each analyst returns a short summary with its source. The main agent combines them. It searches again if something is missing. Raw results go to files. The noisy reading happens in subagents with their own context windows. The main agent only ever sees conclusions. Anything off-topic stays in the subagent's context. The main agent's context window stays clear!

Why it Matters

Filtering does keep off-topic results out. But it cannot fix results that are out of date. Our competitive analysis agent runs every Monday. Last month's index will not catch this week's news. A solution is a search API or MCP server. These could give the agent current information. They provide information instead of whatever was indexed months ago. If your data has dates, filtering by date in the retrieval tool would also keep old results out. On the flip side, relevant current results can cause problems. Since retrieved text is data, not instructions, anything you retrieve could contain a prompt injection. No prompt or delimiter strategy fully prevents this. Make sure to validate outputs. Check that answers cite the sources you expect. Check that claims match them. Here is what to look out for regardless of whatever harness you are building on. Use the most efficient tool that works. If you know the words, grep beats an embedding lookup. Save semantic search for when you actually need it. Page instead of loading. Read large files in pieces instead of all at once. Return pointers instead of payloads. Write retrieved content to files and hand the agent the paths. Delegate the reading. Let subagents read and summarize so the main agent only sees conclusions. Verify grounding. Check answers against what was retrieved. Treat retrieved text as untampered data.

How OpenSmartRoute Helps

Teams that route requests through OpenSmartRoute gain from this news by managing tool efficiency. OpenSmartRoute sends each request to the best-fit model, agent, tool, or skill from a catalogue. The team defines this catalogue. It scores every candidate on quality, cost, speed, and safety. The team sets the weights per request. Hard rules pin a request to specific constraints. Text with personal data stays on an on-premises model. A region or a cost cap is never crossed. It learns from outcomes, so a model that answers well gets more traffic. One that fails gets less. A new model is one catalogue entry. It competes on the next request. Nothing else changes in the app. It works with any OpenAI-compatible provider. It works with open-weight models served locally. It works with MCP tools and A2A agents. A savings ledger shows what each routed request cost next to what the most expensive model would have cost. An input guard spots prompt injection and personal data before a request leaves. 'osr eval' measures routing accuracy on the team's own prompts. It can fail a build when it drops. The hosted platform keeps a models catalogue with prices and public rankings. These rankings are built from real traffic.

What to Do

Start by auditing your current agent search patterns. Identify where agents open too many files or load too much data. Look for inefficient searching that wastes tokens. Check if your tools return full payloads instead of paths. Implement the retrieve, offload, delegate pattern for large results. Save results to files and return only the paths. Create subagents to read and summarize those files. Ensure your retrieval tools use semantic search only when needed. Use grep for keyword searches to save context. Set up date filtering to prevent stale data from entering the context. Validate all outputs to check for prompt injections. Use OpenSmartRoute to manage these tools and models efficiently. Monitor your savings ledger to track cost reductions. Run 'osr eval' to ensure routing accuracy remains high.

How OpenSmartRoute helps

Teams using OpenSmartRoute gain immediate savings by routing agent searches to cheaper models. The router scores every candidate on cost, speed, quality, and safety before sending a request. It learns from outcomes so a model that answers well gets more traffic. A new model is just one catalogue entry and competes on the next request. Nothing else changes in the app when the team updates its catalogue.

The input guard spots prompt injection and personal data before a request leaves the system. This protects sensitive information while agents search for context. The savings ledger shows exactly what each routed request cost next to the most expensive option. Teams can see real money saved by avoiding full file loads in the context window.

To set this up, the team defines its catalogue with prices and public rankings. It runs 'osr eval' to measure routing accuracy on its own prompts. The router works with any OpenAI-compatible provider or open-weight models served locally. Agents use built-in tools like grep to find text without loading full files. The retrieve, offload, delegate pattern keeps the main agent context clean and efficient.

Source: https://www.langchain.com/blog/context-engineering-starts-with-retrieval