Replaying a real session to find the memory cost we never counted

On October 26 we shipped a round of session memory fixes and closed the write-up by saying the before-and-after numbers were still to come. They weren't good. Long sessions were still being reset, and a single session could still peak at 200 MB. This is what happened next: we nearly hid the problem with an easier fix, then replayed a real session locally and found a cost we had never counted.

Big sessions still died

Agent sessions run on Cloudflare Durable Objects. The last round removed the extra copies of large tool outputs and cut the baseline from 60 MiB to 38 MiB, yet long sessions kept getting reset. Our models offer a 1M token context window, so the context was the obvious suspect.

The first idea: shrink the context

Cap the context at 150k tokens and compact earlier. Memory would certainly drop, and the change was small.

We didn't do it. The 1M window is something people use, and cutting it would trade the model's capability for our memory budget. It was also still only a guess: we had never measured how much memory the context itself takes.

We also suspected a framework leak. Our earlier 50-round run showed about 3 MiB of resident growth, but that was a lab fixture, not a real long session, so it couldn't rule a leak out.

Turning a real session into a fixture

A fixture only contains the cases you thought of. So we exported a session that had really hit the limit and added a recipe to the memory lab: seed that history and run real steps at its production context window. The export stays in a local cache and never enters the repository.

Heap snapshots only show what is alive right now, and we wanted to see what each step did. So we added allocation sampling through the inspector, which records allocations by function, including ones already collected.

await cdp.send("HeapProfiler.startSampling", { samplingInterval: 16384 /* ... */ });
// run real steps
const profile = await cdp.send("HeapProfiler.stopSampling");

Two results came back:

• The context of this 2.2 MB session added only about 3.4 MiB of memory. The capacity theory was wrong.
• 71% of allocation came from measureContext. It tokenized the whole request twice after every model step, about 117 MiB per step.

Why tokenizing matters

Our last post showed that what kills a session is data alive at the same moment, not total allocation. So 117 MiB of allocation can't be read as peak memory; the real effect had to be measured after the change.

There was clearly room to improve. A piece of text always has the same token count, and we were recounting the whole request twice a step.

What we changed

The constraint was the same as last time: no change in behaviour and no loss of precision. Token counts decide when to compact, so an estimate wouldn't do. We could only count less often.

Each message part now resolves through three layers:

1. A WeakMap keyed on object identity.
2. Counts stored by content hash in the object's own SQLite, capped at 4,096 rows, which survive a restart.
3. Real tokenization, only when both miss.

The tokenizer itself now walks the text with lastIndex and slice instead of building intermediate string arrays.

Token counts match the previous tokenizer, and with model usage available, totals and compaction decisions don't change.

A killed session leaves nothing behind

The worst part of this hunt was having no evidence after each reset. The platform gives no callback, and code can't read the isolate's memory.

So we record ahead of time. Each loop boundary now writes the shape of the step to the lifecycle record: step number, message count, history and request bytes, request tokens, the current tool and its output size. It shares a write with the activity state, so it costs no extra row. The next boot reads it and logs agent.session.died with the object id, which joins to the platform's memory buckets. A reset can now be traced to a session and a step. The platform's memory-limit log line also has an alert and a runbook.

Users get a better answer too. During a reset, session reads retry twice at 500 ms, then return a retryable 503 SESSION_RESTARTING instead of an opaque internal error. Writes are not retried.

We misread the data once

Our first look after the fix said nothing had improved and sessions were still dying about five times a day. We nearly went off to change something else. The query window turned out to include data from before the fix. Split at the deploy time, the conclusion reversed. Before comparing a before and an after, check that each window holds only its own side.

Where things stand

As of November 2:

24 hours isn't long, and we're still watching.

Three things we don't know yet:

• Two resets had only about 1.1 MB of request and 8 MB resident in the black box, with no evidence of a memory limit. We haven't found their cause.
• How much of the 144 MB peak belongs to the session itself. As we noted before, several objects can share one runtime.
• Where the real kill line is. We still only know it falls between 254 and 300 MB.

The real sample was more honest than any fixture we had thought up. "The context is too big" sounded entirely reasonable, and without the replay we would probably have shipped the cap.