Tool-call optimization: stop using context as a data transfer layer

Tool calls execute correctly, yet the Agent still needs many requests and its context keeps growing. What should we optimize next?

Start with the path the data takes.

One tool returns a large dataset. The model copies it into another tool’s arguments. That tool produces intermediate output, only a small part of which matters for the answer. The rest stays in history. While a job runs, more requests ask whether it has finished. Every call may succeed while the workflow does unnecessary work.

The previous article covered reliability. This article examines what comes next: let the model read the information needed for its decisions, while the application handles data transfer, storage, processing, and waiting.

We recently applied these ideas to market-analysis tools. Thousands of candles make the issue easy to see, but log analysis, reporting, and batch document processing can face the same design problem. The market workflow below is implemented; the other scenarios illustrate how the ideas could transfer.

First ask who needs the data

Before changing a large result’s format, identify its next consumer.

Purpose	Suitable treatment
The model needs evidence to interpret or choose a next step	Return relevant fields, excerpts, and coverage
Another tool needs the data as input	Store it on the backend and pass a resource reference or query parameters
Details might be needed later	Return a result identifier and a way to retrieve them
Data only describes execution progress	Let the application maintain it and display it when useful

For example, a tool can count errors across a full log and return statistics with representative samples. The model can retrieve surrounding lines when investigating a particular error. Reporting tools can filter, group, and aggregate before the model explains the results.

The task determines the boundary. If the user asks for a paragraph-by-paragraph document review, the text is input to the model’s work. A file ID cannot replace reading it. References eliminate copying; they do not give the model knowledge of unseen content.

Keep long inputs on the backend and pass references

Consider a request to calculate moving averages and crossings over a year of hourly candles. Our old workflow returned all the candles to the model, which copied them into calculation arguments. If the user only needs the signals, that adds an unnecessary step. Most of those values are calculation inputs, not evidence the model needs to read to decide what to do next.

The Agent now gives Compute the source, instrument, timeframe, date range, and calculation logic. The server reads the requested market data, pins a snapshot, stores the long input in backend segments, and calculates across them. Indicator windows and recurrence state are saved with progress; earlier candles do not need to enter the next model request.

Before: fetch all candles → model context → copy candles into tool arguments → calculate
Now: submit data-location parameters and calculation logic → receive task ID
     → backend captures snapshot and calculates in segments → read needed results

References work at two levels. Market parameters identify the input range when creating a calculation. After creation, a task ID and result cursors address the saved execution and its output. The internal queue also carries task identifiers; the Worker loads the saved request from the database. Neither the model nor the queue needs to keep carrying the full market dataset.

The general pattern is for a producing tool to return a readable resource identifier and for the next tool to accept that reference plus processing parameters. An existing data source can instead be addressed through query parameters. Long logs, uploaded files, and query results can follow this design without being copied through the model.

The backend holds working data without relying on one Worker’s memory. The current implementation stores input snapshots, calculation checkpoints, and result segments in PostgreSQL. A replacement Worker or retry can continue from committed progress. The model receives task identifiers, summaries, and result pages needed for inspection. Raw candles and unrelated intermediate data do not enter conversation context merely because the calculation used them.

Working storage needs clear ownership, versioning, retention, and expiry behavior. Each read must enforce authorization; possession of a reference cannot bypass access checks. An expired reference must not silently resolve to fresh data. Storage choices depend on size and lifetime, but must not require one process to stay alive.

Compact the data the model actually needs to read

Once data needs to reach the model, examine its representation. Query results, log lists, product catalogues, and time series can all repeat the same field names on every row. A JSON array of thousands of candles, for example, repeats timestamp and price-field names throughout. Removing indentation and whitespace leaves those repeated keys and object structures consuming context.

The relevant size is the text the model reads. Compressing JSON for HTTP transport still leaves the same large payload after decompression. Moving repeated column names into a header makes more of that context carry actual values.

Choose a representation that fits the shape: clear fields for single records, shared columns for repeated rows, and minimal wrapping for long text. Our Service Tool implementation uses QRP1, a versioned text protocol. It has five shapes: record for individual records, table for tables, series for time series, text for a body of text, and receipt for a compact execution receipt. The model reads it directly without converting it back to JSON. QRP1 is our implementation choice; other systems can use existing tabular or compact-text conventions that the model understands and the application can validate.

The current K-line tool uses fixed dt_ms,o,h,l,c,v,closed columns. Compute tables use columns declared by the calculation. Generic nested objects are flattened into path,value_type,part,parts,value. All can use QRP1 without sharing the same compression behavior.

The fictional candles below illustrate the specialized series format. Only key fields are shown; in actual output, t0, columns, source, snapshot, and paging fields share one QRP1|... header line:

# Object array
[{"time":"2026-09-08T00:00:00Z","open":100,"high":103,"low":99,"close":102,"volume":25.4,"closed":true},
 {"time":"2026-09-08T01:00:00Z","open":102,"high":104,"low":101,"close":103,"volume":21.8,"closed":true}]

# Illustrative compact series data section
 t0=2026-09-08T00:00:00Z
 columns=dt_ms:i64,o:f64,h:f64,l:f64,c:f64,v:f64,closed:bool
 0,100,103,99,102,25.4,1
 3600000,102,104,101,103,21.8,1

dt_ms is a millisecond offset from a fixed t0, which stays unchanged across pages. closed preserves whether a candle is confirmed. Nulls, booleans, and special-character escaping follow shared rules; making a result shorter must not make null indistinguishable from an empty string.

The savings come from repeated structure. Source, coverage, error details, and execution status still matter. Small records may not beat JSON, and flattening deeply nested results does not guarantee savings either. Long tables with repeated columns are the clearer use case. The experiment below measures bytes and tokens separately; fewer characters alone do not establish token savings.

Return enough to proceed, then retrieve details on demand

The first response should support the next decision and state coverage and how to retrieve more. Log analysis might return error counts and sample locations; a report might return totals and a detail reference. Retrieve further evidence when the model needs to verify a claim or the user asks to inspect it. Summaries must not imply completeness, and samples must be labelled.

In the market example, the Agent can retrieve details if the user asks to inspect every crossing. Even compact results can fill the context. Each response has a byte limit, with next_cursor for further pages. This limits one response, not the total coverage of a query, calculation, or file-processing task.

Paging must also preserve a fixed snapshot. If data changes between the first and second pages, a live query can repeat or skip rows. Our cursors bind the query, result revision, ordering, and snapshot. During retention, replaying a cursor returns the same logical page. Expired or mismatched cursors fail explicitly instead of silently starting from new data.

If the Agent automatically reads every page, paging merely divides one large response into smaller ones. It does not reduce the total information entering context. Apply backend filtering or aggregation first when the task allows it.

Let the application handle long waits

Calculations, batch exports, and log scans can outlast one tool execution. When nothing has changed, repeated “still running” responses provide no evidence for the next model decision. Having the model keep asking for progress adds requests and fills history with identical status messages.

Such tasks can return an identifier, let the application display progress, and resume reasoning once the needed result is ready. Our server-tool implementation saves what the conversation is waiting for in the database and ends the current execution. When the result is ready, the system starts another model request with that result. There are four steps:

1. Register the wait. If the background task is still running, save the session ID, original toolCallId, background task ID, and related metadata. The session enters awaiting_server_tool; the current execution ends while the calculation continues.
2. Record a terminal event. When the task completes or fails, write an event to a database outbox—a list of notifications awaiting delivery. It records that a result needs to reach the corresponding conversation and survives a Worker restart.
3. Deliver the tool result. A Worker reads the event and actual result, encodes the result as QRP1, and appends a role: tool message to the conversation using the original toolCallId. The next model turn can pair the answer with its earlier call.
4. Queue the continuation. The system creates a successor execution of type server_result. The model receives the previous conversation plus the newly delivered result and can continue its analysis, call another tool, or answer the user.

This does not keep a model request open throughout the wait. A background Worker still checks pending delivery events, but it does not call the model to ask for progress or repeatedly add “not finished yet” to the context.

Database transactions and state checks prevent duplicate delivery. Appending the result, creating the successor execution, and updating the waiter and event states commit together. An already delivered event is skipped. If the user has interrupted the wait, an old event cannot restart that continuation. Delivery can therefore be retried while the tool call receives its result only once.

There is also a timing gap to handle: a task might finish before its waiter is registered. Registration checks whether the task is already terminal and creates the delivery event if needed. An existing result must not leave the conversation waiting indefinitely.

Keep failure details even when results get smaller

A compact result still needs to state the outcome, resource or version, completeness, and available next action. Failures need stable codes and safe, useful causes rather than a generic “processing failed.”

Failures follow the same result path. We fixed a misleading case where a task had already failed but a bad cursor caused the result read to report a paging error instead. The task’s terminal status and safe diagnostic now take precedence, so the model can repair its code or arguments. If a side effect remains uncertain, the result retains outcome_unknown; automatic retries cannot resolve that uncertainty safely.

A reproducible comparison using the current codec

We measured deterministic synthetic payloads using the current code. No user data was read and no model was called. Token counts below come from encoding the actual text with o200k_base. They are not live model billing counts, latency measurements, or success-rate results.

How savings change with data size

The K-line fixture grows from 1 to 10,000 rows, with up to 500 rows per page. Both representations preserve the same values and source, range, snapshot, and paging semantics. The baseline is minified JSON; QRP1 uses the current K-line codec. Totals include all page headers. Both sides use fixed 384-byte synthetic continuation cursors where needed.

Each original JSON baseline is normalized to 100%. Bytes and tokens share one percentage axis. Colored bars show the retained QRP1 size; the gray remainder shows savings. Values above 100% mean expansion.

Rows	JSON bytes	QRP1 bytes	Byte reduction	JSON tokens	QRP1 tokens	Token reduction
1	649	589	9.2%	253	258	-2.0%
500	61,200	23,910	60.9%	26,163	13,416	48.7%
5,000	617,470	249,591	59.6%	265,311	140,360	47.1%
10,000	1,235,675	503,006	59.3%	531,332	281,709	47.0%

Reduction means 1 − QRP1 size / JSON size. At one row, QRP1 uses about 2% more tokens: header overhead has few rows to spread across. At 5,000 rows, bytes fall 59.6% while tokens fall 47.1%. File size alone would misstate the context savings.

Generic flattening can make results larger

A control fixture contains 500 product-shaped records with ID, label, price, and enabled state. The current generic adapter expands these into 2,000 path/value rows, encoded in pages of 300 flattened rows. Its baseline is the complete minified JSON object. The chart includes path, type, fragment, and paging overhead.

The generic fixture grows from 34,968 to 82,029 bytes, and from 11,405 to 38,357 tokens. The representation preserves explicit paths and types and supports segmented reads. It does not guarantee compression. The K-line result cannot support a claim that every tool result gets cut in half.

For tools that repeatedly return homogeneous lists, this suggests evaluating dedicated result columns instead of repeating a path for each scalar. That is a potential improvement identified by this fixture, not a change claimed as already implemented.

Compact results still leave the next data copy

A final controlled illustration uses 5,000 rows and compares three input-transfer paths: JSON results followed by raw-data arguments; QRP1 results followed by the same raw-data arguments; or direct data-location arguments that let the server read the input.

This counts input-transfer payloads only, not complete tasks. Calculation logic is represented by fixed placeholder text, and no calculation is executed. Prompts, tool definitions, lifecycle receipts, final results, and history replay are excluded. The 64-token number describes these example location arguments; it does not mean a calculation task costs 64 tokens.

The narrower comparison still exposes the design issue. Compacting a result leaves substantial copying if the model must then emit all the raw data as another tool’s arguments. Letting the tool retrieve the data by parameters removes that long input from model context.

Download the measurements as CSV or JSON. These fixtures measure representation and transfer costs; production gains still need task-level validation.

Measure the workflow, not just the format

Each technique removes a different cost. References reduce copying, backend processing removes intermediate data, compact formats remove repeated structure, on-demand reads avoid unnecessary details, and asynchronous continuation avoids model requests during waits. Adopt them where the data path needs them; a small result does not require an entire background-job system.

Three groups of evidence matter:

• Context load: bytes and actual tokens in results and call arguments, including whether long inputs still get copied into later requests.
• Work performed: model requests and tool calls per task, model polling during waits, and total completion time.
• Correctness: segmented processing matches processing the same complete input, pages neither skip nor repeat rows, failures retain their causes, and results are delivered once.

In our market implementation, the data path and continuation mechanism have changed. This experiment measures synthetic-payload bytes and tokens. Full-task token and latency savings still need measurement on the same tasks with the target model. A shorter format alone does not establish a faster workflow.

When we encounter a large Tool Result now, the first question is whether the model needs to read all of it to make its next decision. If another tool is the consumer, the backend can pass a reference. Conversation context can then carry task requirements, evidence, and the results that actually need explaining.