Prompt Cache Reuse Dropped From 99% to Zero—Here’s Why

An Agent can send almost the same long context to the same model twice and get two very different cache results. In one Strat Thread production trace, reuse climbed from zero to almost 99%, stayed there for one request, and then fell straight back to zero. Another trace, recorded less than a minute away, stayed between roughly 96% and 99%.

The first guesses are natural: perhaps the cache expired, the request reached another machine, or the provider sneezed. Those things can happen. But the traces—request travel logs that record timing and token use—pointed back to the application: it rebuilt the beginning of the request whenever it loaded another capability.

The useful mental model is simple:

Prompt caching does not ask whether two prompts mean the same thing. It starts at the first token and checks how long the order stays identical. A token is a small piece of text as the model reads it.

Progressive loading is compatible with excellent cache reuse when new capability blocks are appended. It becomes destructive when each load rewrites the system prompt or inserts tools ahead of context that the model has already seen.

These are representative production paths, not a controlled before-and-after benchmark. The unstable path has a token-weighted hit rate of 36.26%; the stable path reaches 97.64%. They prove that both behaviours existed in the same deployed system. They do not, by themselves, prove that a later code change improved production.

What OpenLIT recorded

The unstable trace contains five model requests:

Request	Input tokens	Cached input tokens	Request hit rate
1	29,168	0	0%
2	31,337	0	0%
3	31,558	31,232	98.97%
4	31,635	31,232	98.73%
5	48,556	0	0%

Across the trace:

62,464 cached / 172,254 input = 36.26%

This is not gradual decay. It is 0% → 0% → 98.97% → 98.73% → 0%. The fifth request contains plenty of familiar material, yet none of its reported input is reused.

The stable trace tells a different story:

Request	Input tokens	Cached input tokens	Request hit rate
1	49,002	48,640	99.26%
2	49,181	48,640	98.90%
3	50,896	48,640	95.57%
4	51,401	50,688	98.61%
5	52,806	50,688	95.99%

Across this trace:

247,296 cached / 253,286 input = 97.64%

Input grows as tools run and results enter the conversation, while almost all earlier context remains reusable.

Because these traces came from the same version at nearly the same time, the responsible conclusion is that the application could produce both a stable and an unstable prefix. This is diagnostic evidence, not release acceptance.

It caches work, not answers

LLM inference can be simplified into two phases:

1. Prefill processes the input tokens and computes the per-layer attention state needed by the model.
2. Decode generates output autoregressively, reading previous state and appending state for each new token.

The Transformer architecture introduced in Attention Is All You Need makes the input more than text that a server merely parses. A long prompt passes through many layers of large matrix operations. Recomputing the same prefix is real accelerator work.

When a new request begins with an identical token sequence, a serving system can reuse Key/Value tensors—or an equivalent provider-managed prefix state—already produced for that sequence. Only the new suffix needs fresh prefill before decoding continues. OpenAI documents that prompt caching may keep encrypted key/value tensors in GPU-local storage; its exact production implementation remains private.

The PagedAttention paper shows why this state is a first-class serving concern. KV cache grows with sequence length, consumes substantial accelerator memory, and benefits from block-based allocation and prefix sharing. Reuse saves compute, but it still needs memory, lookup, routing, expiry, and eviction.

A prompt cache is not a box of old answers. The model still creates a fresh output. What it reuses is the calculation already completed for an identical beginning—like continuing a long sum from the first new line instead of starting again at the top of the page.

It matches the same beginning; it does not guess meaning

Think of a request as one path from a root:

hidden provider context
  -> tool definitions
  -> stable developer policy
  -> capability block A
  -> user message
  -> tool call
  -> tool result
  -> next user message

The cache looks for the longest identical beginning. A computer scientist may model this with a trie or radix tree, but the simple picture is a roll call. Compare two lines from the front, one position at a time. Once one person does not match, everyone after that point stands behind a different history, even if some names appear again.

Common prefix breakers include:

• merging a newly loaded capability back into the first developer message;
• preserving the same tools but changing their array or JSON serialization order;
• recomputing active capabilities from an empty set on every turn;
• putting timestamps, page state, session values, locale, or user values before stable policy;
• replacing history with a compacted summary;
• changing model, reasoning options, region, or provider request construction; and
• using volatile cache keys that scatter otherwise identical traffic.

That is why cache loss often looks like a cliff. Everything before the first mismatch may be reusable; everything after it needs new computation.

Why step-by-step loading can break the beginning

Progressive loading is meant to keep irrelevant tools out of the initial context. An Agent starts with a foundation and loads strategy, market-data, backtest, or operations capabilities only when needed.

A naïve implementation rebuilds the catalogue:

turn 1: foundation + tool A
turn 2: regenerated foundation + tool A + tool B
turn 3: regenerated foundation + tool B + tool A + tool C

The product sees increasing capability. The cache sees a moving prefix. Inserting tool B before tool A can shift the position of history, calls, and results that were previously reusable.

The safer shape is append-only:

stable foundation [explicit cache breakpoint]
  + capability block A
  + first-turn messages and results
  + capability block B
  + second-turn messages and results

“Progressive” should describe how capability is appended, not how the complete prompt is regenerated.

How I now keep the beginning stable

The design I now use has seven parts:

1. Keep one deterministic foundation at the front. Identity, authority, safety, data freshness, tool use, and delivery rules have stable text and order.
2. Place an explicit breakpoint after that foundation. GPT‑5.6 can preserve this prefix even when later user content changes.
3. Store capability blocks separately and append them. A batch may use canonical ordering, but a block already present in history never moves.
4. Let the session own the retained activation set. Do not rediscover capabilities from an empty set every turn; reset only through an explicit clear operation.
5. Make the cache key describe the prompt contract. Version, language, mode, and controlled sharding are useful. Random request/session values and raw user identifiers are not good defaults.
6. Fingerprint the complete request envelope. Model, reasoning settings, tool catalogue, provider options, and retry paths all affect reproducibility. A new-model optimization must not silently change a legacy route.
7. Observe reads and writes separately. cached_tokens alone can overstate savings. GPT‑5.6 requires cache_write_tokens, ordinary input, output, latency, and final cost.

The current OpenAI prompt-caching guide makes the same operational points: retain conversation history, keep tool definitions stable, append multi-turn context, use explicit breakpoints after stable content, and use prompt_cache_key to influence routing without treating it as a guaranteed machine pin.

Why cached input costs less

Cached tokens are cheaper because the expensive repeated prefill has already happened—not because the tokens disappear. A later request still pays for new suffix prefill, all output decoding, cache memory and lookup, and any miss or routing overflow.

As of 30 August 2026, OpenAI’s GPT‑5.6 launch and pricing page prices a cache write at 1.25× the ordinary input rate and a cache read at 0.1×. For a stable prefix of length P:

two requests without caching:       2P
one write plus one complete read:   1.25P + 0.1P = 1.35P

ten requests without caching:       10P
one write plus nine complete reads: 1.25P + 9 × 0.1P = 2.15P

The first full reuse already compensates for the write premium. Repeated reads push the average prefix cost towards the cached-input rate. But this only applies to the reusable input prefix. Dynamic suffixes and output remain fully priced, while repeatedly writing low-reuse content can cost more than leaving it uncached.

At the GPT‑5.6 Sol price listed that day—$5 / 1M ordinary input tokens—the corresponding write and read rates are about $6.25 / 1M and $0.50 / 1M. Production code should not hard-code those figures. Prices change; cost should come from provider usage and billing data.

Measure tokens and money, not average percentages

An unweighted mean of request hit rates is misleading. A ten-token request at 100% and a 100,000-token request at 0% average to 50%, while almost all spend comes from the latter.

Use a token-weighted rate:

sum(cached_input_tokens) / sum(input_tokens)

Then calculate cost explicitly:

ordinary input × input rate
+ cache reads × cached-input rate
+ cache writes × cache-write rate
+ output × output rate

Latency is separate. Cache reuse usually reduces prefill work, but queueing, network time, tool execution, output length, and reasoning effort may dominate the end-to-end trace.

A controlled acceptance run should hold the model, provider route, foundation, tool definitions and order, user task, and request count constant. Exclude one warm-up, aggregate baseline and candidate tokens separately, and require final cache_write_tokens rather than filling missing usage with estimates.

What this evidence proves—and what it does not

The OpenLIT evidence proves that:

1. a production Agent path can fall from almost 99% reuse to zero after its prefix changes;
2. the same application can sustain a 97.64% token-weighted stable path; and
3. the goal is not to hide necessary context from the model, but to serialize it in a stable, append-only, observable form.

It does not yet prove that the Strat Thread candidate turns every purple request green. At the time this article was published, the new prefix contract had not entered production. The green trace is a real reference for the desired path, not a post-release acceptance result. A controlled workload will compare cache reads, writes, total cost, and latency after that release.

That distinction is part of the engineering work. A cache chart becomes untrustworthy the moment two different workloads are renamed “before” and “after.”

The deeper design lesson

Prompt caching for a long-running Agent is not mainly a provider switch. It is a question of how the application arranges context.

The foundation should behave like an immutable header. Capabilities should behave like append-only events. Tool definitions need stable identity and order. Volatile values belong near the suffix. Breakpoints should reflect rates of change. Telemetry must separate cache writes, reads, ordinary input, and output.

With that structure, progressive loading and high cache reuse are compatible. An Agent can gain tools as it works without discarding tens of thousands of tokens of completed computation. The expensive part is not that a prompt is long. It is making the model process the same long prefix as if it had never seen it before.