Zero cache reads across 7.6 million tokens

Prompt caching is the cheapest performance win in an LLM application. If the start of a request is byte-for-byte identical to one you sent a few minutes ago, the provider reads it back from cache instead of processing it again. On Anthropic’s pricing, a cache read costs a tenth of a normal input token.

An agent is the ideal case for it. Every turn resends the same tool definitions, the same system prompt and the whole conversation so far, and only the last message is new. In a long agent turn with many tool calls, almost all of the input is repeated.

So when I checked the token accounting in the agent runtime behind FAC Cloud, the hosted service for Frappe Assistant Core, I expected most of our Claude input to be cached. The real number, across roughly 7.6 million input tokens, was zero.

The runtime is closed source, so the code in this note is simplified illustration rather than the real thing. The bugs, the numbers and the lessons are real.

Nothing had failed. Every request succeeded and every answer was correct. We were simply paying full price for the same context over and over, and the only place it showed was a column of zeros in a usage table.

How the cache sees a request

The rest of this note depends on one fact. A request is cached as a single prefix, in this order:

Two things follow from that order. A change to the tool definitions invalidates everything after them, including the entire conversation. And a change anywhere in the prefix is not a partial miss. The cache is not a diff; one different byte and the whole prefix is written again, at 1.25 times the normal price.

Each of the four bugs below broke that prefix in a different way.

Bug 1: the cache point never reached the provider

We build agents with the Strands Agents SDK. Our runtime marked the system prompt with a cache point, passed as a structured content block. The version of the Strands Anthropic provider we used at the time accepted the system prompt as a plain string, so the structured block, and the cache point inside it, was silently discarded on the way out.

The fix was to stop relying on the system prompt and put the cache point where the provider does honour it: inside the message content. We wrapped the model so a single rolling cache point is added to a copy of the messages just before each request.

import copy

def with_cache_point(messages):
    """Return a copy of the messages with a cache point on the second-to-last one."""
    if len(messages) < 2:
        return messages  # first turn: nothing to reuse yet
    marked = copy.deepcopy(messages)  # stored history never sees the marker
    marked[-2]["content"].append({"cachePoint": {"type": "default"}})
    return marked

The point goes on the second-to-last message, not the last. A point at the very end covers the new message too, so every request writes a fresh cache entry that no later request will ever match. One message back, the next request finds the entry this one wrote.

Bug 2: caching was never even attempted

While fixing the first bug I found a second one that made it irrelevant. The check deciding whether a model supports cache blocks compared the model’s class name against an allowlist:

if type(model).__name__ in CACHE_CAPABLE_MODELS:  # never true in production

In production every Anthropic model ran through our own subclass of the provider class, and that subclass name was not on the list. So the runtime never asked for caching in the first place. The fix was to unwrap the model and use isinstance against the real provider classes, which also keeps working for any future wrapper or subclass.

An exact type-name check is fine until someone subclasses, and then it fails without a sound. The same pattern was hiding in the code that applies per-turn reasoning settings. A regression test caught that one before it shipped.

Bug 3: a clock in the system prompt

With caching switched on, the numbers improved, but one environment still showed more cache writes than reads on every cycle: about 44,700 tokens written against 35,700 read. That is the signature of a cache that is being filled and never hit.

The cause was one line. To help the model with questions like “invoices due this week”, the system prompt included the current time, to the minute:

f"- Current time: {now:%H:%M} ({tz})\n"

The system prompt sits inside the cached prefix, so a new minute meant a new prefix. Our runtime also rebuilds the agent on every cycle of a turn, and a turn runs one cycle per round of human approval. A turn with eight approval rounds paid for the full context eight times.

The fix keeps what the model actually needs, which is the date and roughly the time, and says so honestly:

f"- Current time: between {now.hour:02d}:00 and {(now.hour + 1) % 24:02d}:00 ({tz})\n"
"Call a tool if you need the exact time — this window is deliberately coarse."

The prompt now changes once an hour instead of once a minute. Building it at 11:11 and at 11:15 used to produce two different prefixes. It now produces the same hash twice, and a test pins that property so the clock can’t drift back in.

Bug 4: tool order followed the last edit

The runtime connects agents to several MCP servers and merges their tools into one list. The servers were loaded with Frappe’s frappe.get_all, which sorts by modified DESC by default. Editing any server, or even writing its connection status, moved it to the front, reordered the tool definitions, and invalidated every cached conversation for that user.

servers = frappe.get_all("MCP Server", fields=["name", "url"], order_by="creation asc")

One argument fixed it. A second test now checks that editing a server or updating its status leaves the tool block’s fingerprint unchanged, and that only adding a server changes it.

The trap we decided not to build

This last bug matters most when you are tempted to load tools lazily, which we were. FAC exposes 32 tools, about 9,000 tokens of definitions, and connecting more MCP servers adds more. A popular fix is a small “tool finder” step that picks the relevant tools for each turn and sends only those.

Given the prefix order, that is exactly backwards. A stable tool list costs almost nothing after the first request, because it is read from cache at a tenth of the price. A tool list that changes every turn moves the start of the prefix every turn, so the system prompt and the whole conversation are written again at full price. The tokens you save by trimming the list are smaller than the cached context you throw away.

So we kept the tool list fixed for the length of a conversation. Tools are chosen when the agent is built, the order is deterministic, and turning a server on or off takes effect from the next conversation. If we ever need to expose hundreds of tools, the plan is a fixed pair of tools, one to search the catalogue and one to call a tool by name, so the list itself never changes.

Results

On our development environment, across the direct Anthropic models, the runtime went from zero cache reads to about 63% of input tokens read from cache, 18% written to it and 18% sent uncached. Using Anthropic’s published multipliers for reads and writes, that brings the cost of the input to about 47% of what it would be with no caching at all. The development data is a small sample, so treat the exact figure as indicative. The direction is not in doubt.

What I’d tell anyone building an agent

• Check your cache read count before anything else. Caching fails silently. A request with a broken cache point succeeds, and only the usage numbers show it.
• Treat the prompt prefix as an interface. Timestamps, unsorted lists and per-user details placed before the cache point each cause a full miss.
• Pin the property in a test. Build the prompt twice and assert the bytes match. Both the clock and the sort order drifted in because nothing checked them.
• Keep the tool list stable. A large stable list is cheap. A small list that changes every turn is expensive.
• Don’t trust type-name checks. Use isinstance, and unwrap wrappers first.