Eager Tool Calling: How We Cut Agent Latency by 50% on Long Tool Chains

The latency nobody was measuring

An eight-tool agent task in our staging environment clocked in at twenty-four seconds. The model was fast. The individual tools were fast. The wall clock was slow. So where did the time go?

We dropped the trace into a flame graph and the answer was embarrassing. For the first four seconds the model streamed tokens while every tool sat idle. Then the model finished, fired all eight tools in parallel, and spent another twenty seconds waiting for them. Two phases, stacked end to end, with no overlap between them.

We rewrote that codepath. The same request now finishes in roughly five seconds: the four-second stream and the tool execution happen in the same four seconds instead of one after the other. Across a representative sample of production traces, median end-to-end task completion is 50% faster, and longer tool chains see proportionally larger wins.

The technique is called eager tool calling (internally, tool-call pipelining). Think of it as CPU instruction pipelining for agents: a modern CPU doesn't wait for one instruction to retire before decoding the next, and an eager runtime doesn't wait for the model to finish before starting the tools it has already described. The idea is small. The engineering wasn't. Here's what it is, how it works, the production bugs that taught us what not to do, and the numbers we measured.

---

Why parallel tool calling didn't fix it

The common intuition is that parallel tool calling already solved this. It didn't. Parallel tool calling is necessary but not sufficient.

The classic agent loop

A single agent turn looks like this:

1. The model reasons over the context and begins streaming a response.
2. The response contains one or more tool_use blocks.
3. The runtime waits for message_stop.
4. The runtime executes the tools.
5. The runtime collects the tool_result blocks and starts the next turn.

Total wall clock for one turn = stream time + tool time, added serially. The tool layer is a passive consumer that won't start work until the producer has completely finished.

The parallel half-fix

Modern APIs (Anthropic, OpenAI, Bedrock) let the model emit multiple tool_use blocks in a single assistant message, and mature runtimes run those blocks concurrently.

That moves the tool phase from the sum of tool durations to the max. On an eight-tool turn with tools averaging 2.5 seconds, parallel tool calling takes about 2.5 seconds for the tool phase instead of 20. A big win.

But the stream phase still happens first. The tools still wait for message_stop. If the model takes four seconds to emit its response, those four seconds are dead time on the tool side. The structural reason is that most runtimes treat streaming as a UX feature (tokens appearing live in a chat window) and parse tool calls only from the completed message. The information needed to start a tool arrives seconds before the runtime looks at it.

What we wanted

Tools that start running the instant the model finishes emitting them, while the model continues streaming the rest of the message. Not tools parallel with tools. Tools parallel with generation.

---

Three eras of tool calling

01Sequential

stream + sum(tools)

Each tool waits for the one before it, and all of them wait for the stream.

02Parallel

stream + max(tool)

Tools overlap each other, but none starts until message_stop.

03Eager

max(stream, tools)

Each tool starts the moment its block is complete, while the model keeps streaming.

The move from sequential to parallel collapses one dimension of latency. The move from parallel to eager collapses the other. Time goes from sum to max in both directions.

---

How eager tool calling works

The key insight: the seal

Each tool_use block the model emits has a stable tool_call_id. During streaming, the runtime sees chunks arrive one at a time. The first chunk of each tool call carries its tool_call_id. Partial argument chunks carry only an index.

Here's the observation that makes eager tool calling possible: the moment a chunk arrives with a tool_call_id different from the previous one, the previous tool call is complete. The model has moved on. Its arguments are fully accumulated. It's safe to execute right now, even though the assistant message is still streaming.

We call that transition the seal event. A tool is sealed the instant the model starts streaming a different tool, or when the message itself ends. (Providers that emit an explicit end-of-block event, such as Anthropic's content_block_stop, give you the seal directly. The id-transition rule works for streams that don't.)

01Stream

Chunks for A

Tool A’s id arrives, then argument fragments by index.

02Transition

New id: B

The first chunk carrying a different tool_call_id.

03Seal

A is complete

The model has moved on, so A’s arguments are final.

04Dispatch

Run A now

A goes to the worker pool while B keeps streaming.

What the detector looks like

A simplified sketch of the seal logic, in the shape of an OpenAI-style streaming delta. It isn't our production code, which adds the idempotency check, cancellation scope and spans described below.

current_id = None
id_by_index = {}
args = {}

async for chunk in stream:
    for call in chunk.tool_calls or []:
        if call.id:                                  # first chunk of a new tool call
            if current_id is not None:
                dispatch(current_id, args[current_id])   # seal the previous call
            current_id = call.id
            id_by_index[call.index] = call.id
            args[call.id] = ""
        tool_id = id_by_index[call.index]            # route partial chunks by index
        args[tool_id] += call.arguments or ""

if current_id is not None:
    dispatch(current_id, args[current_id])           # message end seals the last call

dispatch hands the call to a background worker and returns immediately. The stream loop never waits on a tool.

Visualizing the overlap

In a correct eager execution, the tool lanes aren't end to end. They overlap:

stream : [==================================]
tool A :   [=========]            <- fires when B's id arrives
tool B :       [=========]        <- fires when C's id arrives; runs concurrent with A
tool C :           [=========]    <- fires at message_stop; runs concurrent with A + B

Drop a vertical line at the 50% mark and it cuts through all three bars. That's the signature of eager tool calling. If your "eager" diagram shows tools end to end, you've implemented sequential tool calling under a different name.

---

Inside CloudThinker's stream handler

01Input

Stream reader

• SSE parserIncremental, chunk by chunk
• Index-to-id mapRoutes partial argument chunks
• Per-call buffersArguments accumulate here

Never blocks on a tool

02Decision

Seal detector

• Last-seen idOne comparison per chunk
• Idempotency checkUnsafe tools wait for message_stop
• seal_latency_ms spanEmitted on every seal

Fire and forget

03Execution

Worker pool

• Cancellation scopeLives as long as the stream
• Exception boundaryPer dispatch, never shared
• Error tool_resultFailures surface on the next turn

One failure stays one failure

The dispatch logic is simple to describe, even if the implementation took care.

We track the most recent tool call ID seen in the stream. The moment a new ID arrives, the previous tool is complete, and we dispatch it to a background worker pool without waiting for the rest of the message. That's the entire seal trigger: one state comparison per chunk.

Partial argument chunks don't carry an ID, only a positional index. So we keep a small index-to-ID map that routes each partial chunk back to the right accumulation buffer, keeping arguments intact even when multiple tools stream interleaved.

The dispatch is fire-and-forget by design. If a background worker raises an exception, it's caught and surfaced as an error tool_result on the next turn. The stream reader never sees it, and the other tools keep running.

---

Production lessons

01Retraction

Idempotent only

A fired tool can’t be un-fired, so writes stay on the classic path.

02Cancellation

Scoped to the stream

Stream dies, scope cancels, in-flight tools abort.

03Isolation

One boundary per tool

A flaky S3 list once took down a whole turn. Not anymore.

04Observability

Seal spans

Sealed-at versus finished-at answers “why is tool X slow?”.

The first surprise was tool retraction. The model occasionally emits a tool call mid-stream, reconsiders and replaces it. Rare, but real. A tool that has already fired eagerly can't be un-fired. We added a per-tool idempotent flag: non-idempotent operations (payments, destructive commands, outbound emails) fall back to the classic path and only fire after message_stop. The eager fast path is reserved for reads and safe operations.

The second was cancellation. When the user interrupts or the model emits a stop sequence mid-stream, every in-flight eager tool needs to cancel cleanly. Leaking a tool execution into a dead conversation is both a correctness bug and a resource leak. We tied each dispatch to a cancellation scope whose lifetime matches the stream reader: stream dies, scope cancels, tools abort.

Then a flaky S3 list operation crashed an entire agent turn for a minute before we realized we had no exception boundary around individual tool dispatches. One failing tool was bringing down the stream reader. Now every sealed dispatch runs in its own exception boundary. A failure produces an error tool_result and nothing else propagates.

Finally, eager execution hides behavior inside the stream in a way classic execution doesn't. "Why is tool X slow?" can't be answered without knowing when it was sealed versus when it finished. Every seal event now emits an observability span with seal_latency_ms, tool_call_id and conversation_id. When something misbehaves, the timeline tells you which stage was slow.

---

The numbers

We benchmarked 16 workloads representative of real agent traffic, comparing a sequential baseline, parallel tool calling and eager tool calling. P50 wall clock from request received to final assistant message. The numbers below come from the open-source benchmark in eager-tools: synthetic, deterministic and reproducible by anyone via make bench.

The synthetic harness removes network jitter so the comparison isolates the dispatch strategy. Speedup versus parallel ranges from 1.20× to 1.50×, median about 1.28×. In production, where tail latency and provider variance compound, end-to-end task completion is about 50% faster on the median trace, and longer tool chains pull further ahead. Treat the OSS bench as the lower bound anyone can reproduce on a laptop. Production wins are larger.

Cost impact

• Output tokens per task: effectively unchanged (same reasoning, same answer).
• Net cost reduction: about 35% on tool-heavy workloads, from fewer retries after timeouts, more conversations completing inside the cache TTL window, and faster task completion freeing up agent slots.

Where the speedup comes from

Two stacked effects:

1. Generation and execution overlap. The biggest chunk. Every millisecond of stream time that used to sit idle on the tool side now has a tool running in parallel.
2. Fewer turns. A task that used to take three model turns often completes in one. Each saved turn saves prefill, a network round trip and reasoning overhead.

---

When not to use it

The limits are real:

• Fast tools (under 50ms). If your tools finish in milliseconds, there's nothing to hide behind the stream. Seal and dispatch overhead exceeds the latency saved.
• Sequentially dependent tools. If tool B needs tool A's result to even be formulated, the model won't emit B until A returns. No pipeline opportunity, so eager and classic are identical.
• Non-streaming backends. You can't seal per block without incremental parsing. If your provider or gateway buffers the full response before forwarding it, eager tool calling is impossible until that changes.
• Non-idempotent tools. Covered above. Destructive operations, payments and outbound messages stay on the classic path.

That last rule matters most for operations agents. In CloudThinker, reads (metrics, logs, inventory, queries) take the eager path, and every write still goes through Auto Mode: in Manual it waits for a named person's approval, and in Auto it runs only if a policy you wrote covers it. Firing early never changes what an agent is allowed to do.

---

A mental model that helps

Eager tool calling is CPU instruction pipelining for agents. The agent runtime is the CPU, the model stream is the instruction fetch, the tools are the execution units, and the seal event is the register-ready signal that lets execution begin before the rest of the batch is decoded. Once that analogy clicks, the rest of the design falls out of it.

---

Try it yourself

Building this requires a streaming SSE parser, a seal detector, an async executor pool tied to the stream's lifetime, per-tool idempotency flags, and observability spans on every seal event. The open-source eager-tools library ships all of it, and make bench reproduces the table above.

Eager tool calling isn't a new idea in the abstract. CPUs have pipelined instructions since the 1980s. What's new is applying it to agents, after two years of the ecosystem treating streaming as a way to make tokens appear live rather than a chance to run execution against generation.

For a production agent running several tools per turn, that's the difference between an impressive demo and a system fast enough to replace a human operator's first twenty minutes of an incident.

Try CloudThinker free and cut your agents' latency in half on long tool chains.