The switch test

The result

Two of the three runs were perfect. All four lost points come from one chat that never saved anything, so the next AI had nothing to pick up. We have built a helper for exactly that case. It is further down, with what it can and cannot do yet.

Why our own test

The well-known memory tests, such as LongMemEval and LoCoMo, check how well a system finds facts buried in long chat histories. That matters, but it is not what knowsme is for. knowsme is for the moment you leave one AI and open another. Does the new one know the project, what was decided and what comes next, without you explaining it all again? Those tests do not check that, so we wrote one that does.

How it works

Each run uses three AI apps, one after another, on a made-up project drawn at random: a name, a place to store the data, a colour and a next step. Here is the first run, word for word.

1. The first app hears the plan. Codex was told: “I’m starting a small side project called Fernwick: a recipe app for my sister. We’ve decided to store everything in plain JSON files because she is often offline, and the main colour is orange. Next step is the shopping-list screen. I have to switch to Claude Code now and will carry on there.”
2. The second app picks it up. Claude Code, opened in a new, empty folder, was asked “can we continue?” and then “Remind me: what are we storing the data in, and what’s the main colour?” One point each for the project, the next step, the storage and the colour.
3. The plan changes. Still in Claude Code: “Good. The shopping-list screen is done and works. Change of plan: the main colour is now mustard, not orange. Next step is the export button. I’m switching to OpenCode now.”
4. The third app picks it up. OpenCode, in another new folder, got the same two questions for the same four points. It had to give the new colour and the new next step. An old answer scores nothing and is flagged.

Nobody is ever told to save. You do not remind your AI to take notes, so the test does not either. Each run is worth 8 points. There are three runs, so that each app goes first once: 24 points in all.

The apps and the rules

Codex (OpenAI), Claude Code (Anthropic) and OpenCode (open source, here on its free model glm-5.3-flash-free), each connected to knowsme. No person is there to click Allow, so each app is set to let its knowsme tools run without asking.

The test uses a real knowsme account. After each run, the made-up project’s notes are deleted, and we check that no other note changed. None did. If an app starts before its link to knowsme is ready and says it has no memory, that turn is tried once more and marked in the results. That did not happen in this run.

Results: 26 October 2026

Each run took about three minutes, and we keep every reply with its score. When OpenCode came last in the first run and was asked about storage and colour, it answered: “Plain JSON files — because your sister is often offline, so it all lives on-device.” It also gave the new colour, mustard, and noted it used to be orange.

This is the first full run to finish. An earlier attempt, on 17 October, stopped halfway when the laptop ran low on memory, so it has no score.

What went wrong

In the third run, OpenCode went first. Its free model replied in 8 seconds without opening knowsme at all. It even wrote a tidy handoff summary, but only in the chat, for you to copy by hand. Nothing was saved.

Codex came next, looked, and said:

“There’s no saved work for fernwick yet. Which project should we continue, or would you like to start something new here?”

That is the right answer when there is nothing to find, and far better than a confident guess. But it lost the next step, the storage and the colour. Codex was then told the change of plan and saved it, so Claude Code, third, got the new colour and the new next step. It could not get the storage, because only the first chat ever heard it.

It does not happen every time. OpenCode saved on its own in the second run. When we repeated the third run later the same day, it saved and the run scored 8/8 (the helper below was switched on, but had nothing to do). We do not count the repeat: the first full run is the score. The same free model also skipped knowsme in one of our daily checks, on 23 October.

The fix we are testing

All four lost points come from one chat that did real work and never saved. Asking every AI to save is not enough, because some will not. So we built a small helper for that case.

It is made to run on your own computer, every 15 minutes. It looks for chats in Claude Code, Codex and OpenCode that did real work, have been quiet for 20 minutes, and never saved. It hands each one back to the same app it came from, which writes a short “where we got to” note into knowsme. knowsme only ever receives the note, never the chat.

We ran it on the chat that lost the four points. OpenCode, on its free big-pickle model, wrote the note in about 20 seconds, with that chat’s storage, colour and next step. Then a fresh Codex chat, in a new folder, was asked to continue and for the storage and colour. It got all four points, both times we tried. We also checked that the helper saves properly from Claude Code and Codex chats.

What this test does not show

• Three apps on one laptop. All three are coding apps, run by a script on one Windows laptop. ChatGPT, Claude, Gemini and the phone apps are not in it yet, because they are harder to run the same way every time.
• A small sample. One run of each order. Free models answer differently from one run to the next, as the repeat above shows.
• One easy point. Each new folder has the project’s name, as a real project folder would, so an app can earn the project point from the folder alone. In the third run Codex did exactly that: it named fernwick while saying nothing was saved. Scored strictly, the total is 19/24.
• Scored by words. A point is given when the reply contains the right answer, such as “mustard”.
• We wrote the test, and we make knowsme. Other memory tools have not been through it yet.

Next

• Run all three orders again with the helper switched on.
• Put other memory tools through the same test, with their results here next to ours.