On October 25 a user filed feedback about two things: context compaction had failed, and the agent "keeps stopping halfway through." This post is the record of how we tracked that down. We guessed wrong twice along the way, and learned a few things about the platform that we only found out by measuring, so those are in here too.

## Background: where a session runs

Our agent sessions run on Cloudflare Durable Objects, one object per session (DO below). The session runtime is built on Cloudflare's Agents SDK and the AI Chat framework, and on top of that we implement our own event protocol for the console and API callers. Model requests go through our own model router to the upstream provider, which for this session was DeepSeek.

## Compaction: a swallowed error

The compaction failure was the easy one. Our router's ledger showed the manual compaction request waiting 118.9 seconds without receiving response headers from upstream. Our limit for compaction was 120 seconds, so it ended as a timeout and was recorded as a 502.

The user, though, saw "No output generated. Check the stream for errors." That came from the order of our code: we awaited `result.text` first and only then checked the error captured from the stream. When the stream has already failed, `result.text` throws the framework's generic error and hides the real cause. On top of that, compaction ran with `maxRetries: 0`, so one failure was final.

The fix was straightforward: wait up to 300 seconds for the first byte and 480 seconds for the whole compaction request; retry twice on timeouts and 5xx; when the context overflows the window, drop the oldest turns and try again; check the stream error before reading the text.

## "Keeps stopping": no cause of death in the logs

This one was much harder. In our logs, the session's DO started 17 times across two periods. On 4 of those starts, it found a turn still marked as running and had to settle it. The last log lines before each of those four looked alike: the model finished a step, a tool started running, and 20 to 70 seconds later the DO was simply gone.

No exception, no error, the log just stopped. Our first suspicion was a deploy replacing the instance, but the times did not line up with any deploy.

## Finding exceededMemoryErrors in the platform metrics

The clue was in Cloudflare's GraphQL analytics API. `durableObjectsPeriodicGroups` reports, per object and per five-minute bucket, the peak memory and the number of memory-limit errors:

```graphql
durableObjectsPeriodicGroups(filter: { namespaceId: $ns, datetimeFiveMinutes_geq: $from }) {
  dimensions { datetimeFiveMinutes objectId }
  max { memoryUsageBytes }
  sum { exceededMemoryErrors }
}
```

The four settlements fell exactly in four buckets with memory errors:

| 5-minute bucket (UTC) | Peak memory | Memory errors | Settlement time |
|---|---|---|---|
| 01:15 | 141.7 MB | 2 | 01:16:01 |
| 01:30 | 157.0 MB | 2 | 01:30:33 |
| 01:35 | 165.8 MB | 2 | 01:38:29 |
| 01:45 | 166.8 MB | 2 | 01:49:32 |

Across production over the previous seven days, 66 active sessions had recorded 30 memory errors, with a per-bucket peak P99 of 147 MB.

At that point our first hypothesis was that long conversations were slowly filling memory with history. It did not survive long: this session was first killed at 00:33:56, during its very first turn, when the history was almost empty.

## Two traps in the platform's memory numbers

Reading these numbers, we fell into two traps.

First, **the fatal allocation is never sampled.** The process is reset before the next sample, so the recorded peak is the last sample before death, not the moment of death. The 141 to 167 MB range is a lower bound, not something you can derive a limit from.

Second, **several DOs can share one runtime's memory.** On our test environment we saw seven different sessions report exactly the same peak in the same five-minute bucket. They were all reading the memory of the runtime they shared. So one misbehaving session can take its neighbors down with it, and a single session's reading is not necessarily its own usage.

## Measuring first: how much memory actually gets you killed

The documented DO memory limit is 128 MB, yet we had recorded peaks above that without every session dying. Rather than guess, we deployed a throwaway Worker to measure it, and deleted it afterwards:

| What we did | Result |
|---|---|
| Allocate 2 GB of collectable garbage in one request, keeping only 4 MB alive | Survived |
| Accumulate about 240 MB of live data over several requests (platform reading: 254 MB) | Survived |
| Hold 300 MB at once within one request | Reset, with a memory-limit log line |

This changed where we looked. What decides life or death is not how much you allocate in total, but how much data is alive at the same moment. Garbage is harmless as long as it can be collected; a few large copies alive together are what cross the line. So the question was not "what allocates a lot" but "when, and which copies, are alive together."

The method and more scenarios are written up separately in [Measuring the memory limit of Durable Objects](/en/blog/durable-objects-memory-limit-measured).

The platform does log an explicit cause, `Durable Object's isolate exceeded its memory limit and was reset.`, but not in our session's own log stream, so we missed it at first.

## Heap snapshots: where the memory goes

We ran the session DO in a local workerd with the inspector attached and took heap snapshots. To make runs reproducible, model replies came from recorded frames and tool outputs from prepared fixtures, with no real model calls. One gotcha: workerd does not respond to `HeapProfiler.collectGarbage`, so taking a snapshot was the only way to force a full GC.

The snapshots showed three things.

**The baseline was high.** Before any session work, code alone used 57.4 MiB, an empty session 58.4 MiB, and 72.7 MiB after the first turn. Most of the first-turn increase was the tokenizer loading. The BPE vocabulary used for token counting stayed resident at about 12 MiB, and the bash command permission table at about 12.5 MiB; both had been expanded into huge numbers of small objects for fast lookup.

**Tool output had no cap and was copied several times.** Bash stdout crossed the RPC boundary into the DO whole, and was then held four to five times at once while being parsed, joined and saved as evidence, plus a base64 copy on a fallback path. One 20 MB output was enough to push a process already at 60 MB over the line, which matches "the DO vanishes tens of seconds after a tool starts."

**There was no leak, but every step re-expanded the whole history.** After 50 consecutive turns, post-GC residency had grown by only about 3 MiB. But on every model step we converted the full history, estimated its tokens, diffed it against the previous version and serialized it, with several copies alive at once. On top of that, the AI SDK keeps every step's full request body until the turn ends by default, so more steps meant more full copies of history held.

## What we changed, and the numbers after

We ordered the fixes by memory saved versus risk, and measured each one before and after in our memory lab:

| Change | Scenario | Peak before → after |
|---|---|---|
| Command output stays on the machine that ran it; the DO gets the head and tail, and full evidence streams from the machine to object storage | One 20 MB output | 122 → 76 MiB |
| Tokenizer vocabulary and permission table as compact binary data, parsed on demand | Empty-session baseline | 60.6 → 38.0 MiB |
| Stop keeping per-step request bodies (`experimental_include: { requestBody: false }`), cache token estimates per event, only build records for changed messages | 5 commands per turn, 100 KB output each | 177 → 154 MiB |
| Fork sessions by reading and importing history in batches instead of passing one array of the whole history | Forking a 36 MB session | 131 → 95 MiB |

The last one previously could not work at all: the data exceeded the size limit for a single transfer between services, and the fork simply failed.

All of these changes had one hard constraint: no change in behavior. For each change, the model request, the persisted content and the events pushed to clients were compared byte for byte, before and after, against the same recorded sessions, and had to be identical. After switching the permission table to binary we checked 130,000 permission decisions one by one, and after switching the tokenizer format we checked token counts on 3,036 text samples.

A couple of detours along the way:

- The original generator script for the permission table was gone, so we treated the existing generated output as the source of truth, derived the binary format from it, and added a drift check.
- The binary data files need a dedicated loader in tests. Our own proof commands always included it, so we never noticed. Just before the release, a colleague ran a plain `bun test` and saw 25 of 26 compaction tests fail. Production was unaffected, since the bundler handles these files, but the test configuration was missing the loader. We fixed that before the release went out.

## Looking back: some of this the framework already did

Partway through, we realized that many of the copies alive at once came from work we were repeating outside the framework: re-reading messages the framework had already stored, re-diffing changes it had already diffed, and running our own stall detection and recovery alongside its own.

So we went through the source of Agents SDK 0.30.1 and AI Chat 0.12.2 item by item, listed 15 overlaps, and moved most of them back to the framework. Stall detection now uses the framework's, treating 300 seconds of silence as stalled.

A few we tried and decided to keep for now. The clearest case is resuming after a restart: when we handed it to the framework, a single tool approval made the model get requested twice (2 requests in a real DO test, against 1 with our implementation). The framework also has no public way to cancel a scheduled recovery when the user presses stop, and its "give up when there is no progress" check only runs when the next recovery attempt runs. We have written these up as requests upstream and will switch once they land.

The split we ended up with: how a session runs inside (message storage, request assembly, stream resumption, recovery) goes to the framework wherever possible; the external event stream, sequence numbers, cursors and permission filtering stay ours, because our public protocol is an event stream while the framework's model is a full message list.

## A rough memory budget

Putting it together, we now estimate a session's peak memory like this:

```
peak ≈ baseline + resident window + max(single inbound payload cap, request assembly unit) × copy factor + GC headroom
```

Looking back, every problem we found was one of these terms without a bound, or a copy factor out of control: tool output had no cap, history was copied several times per step, and the baseline carried two expanded tables. The formula is not precise. Its use is the question it lets you ask while changing code: "which term does this new path's content fall under, and what bounds it?" We recorded it in an architecture decision record and backed it with a few machine checks: no full history reads and no whole-conversation serialization inside a session, a bundle size budget, and a memory lab that measures, for fixed scenarios, how residency grows with turns and how per-step peak grows with history length. Both curves have to stay flat.

For monitoring, we hooked up the platform's memory metrics: an alert fires immediately on a memory error, and a warning fires when a session keeps approaching the limit. Each session DO also records, on every start, how its previous life ended (a deploy, a platform reset) and what it was doing last. Next time someone says "it keeps stopping," we can at least see where and why.

## Where things stand, and what we don't know yet

These changes reached production on October 26. In the seven days before, 87 active sessions in production recorded 36 memory errors, with a per-session peak P99 of 183 MB. A fair comparison needs at least three days of data after the release; we will add it here when we have it.

Two things we still don't know:

- Where the platform's real kill line is. We only measured that it falls between 254 MB and 300 MB, so our design target keeps a margin and no longer assumes the documented 128 MB.
- When the framework will cover what we need: reading the raw, uncompacted history, cancelling a scheduled recovery on user stop, and not repeating requests on approval continuations. Until then, those parts remain our own implementation.

The biggest lesson for us was that a problem like "it keeps stopping" cannot be solved from your own logs alone. The cause of death was in the platform's metrics, the real rule (how much data is alive at once) had to be measured, and where the memory went took reproducible heap snapshots. This time a user's feedback pushed us into it. Next time we would like the alert to fire first.