Don't trust large context windows

I’m working on a tiny agent harness at home to learn and the process of taking human speech and turning it into agent tool calls that output something generally deterministic depending on how the tool is defined is so interesting.

One of the big takeaways is you really only have to rely on the non-determinism<->determinism translation layer once when you switch between the two domains. You can obviously rely on it more if you want, and that’s probably faster because determinism is hard, but you don’t need too do that.

Can't help but feel like a lot of people who are deep in IT made it there because they hated working with humans.

But people are now individually acting this way on their desks on an hour by hour basis. LLMs make cargo-culting inevitable because they are inscrutable and opaque.

There is always this sense in the LLM-proponent world that LLMs are at any moment as bad as they are ever going to be; line goes up.

But it seems clear that the gap between perceived and measurable productivity is still likely spent in poking entrails with a stick.

We are so used to probabilistic tools that have significant setup time before they become valuable and save us loads of time that we're at risk of repeatedly writing off that setup time without seeing the rewards, believing that one day it will actually work out that way.

(Which is most recognisable from the early JS frontend frameworks era.)

Meantime here we have an article that shows that a thing (longer context windows) that people thought would functionally solve a problem so we would get the value from all that setup does not, in fact, very meaningfully kick it down the road, and the comments are still full of entrails-and-stick work.

Have you ever tried doing evals on moderately complex but bounded tasks?

I spent some time doing it when testing these "token reducing" tools like Headroom, RTK etc. as well as customizing my Pi tools. What I found interesting was that despite LLMs being deterministic, for a given toolset and prompt, the results were highly consistent for a given eval, across multiple models (I tested at the time using GPT 5.4 mini, 5.5, 5.3 codex, Gemini 3 flash, initially running sets of 5 evals on each task but once I realized how consistent the results were, dropping to sets of 3.

Aside: in my tests, RTK and Headroom made the overall context use higher for roughly equivalent results. The context use for those specific toolcalls went down but the number of model turns and overall context use went up.

Just because it was nowhere near perfect, and a lot of "religion" and hand waving was involved, doesn't mean it was illusory.

It's enough that it existed quite more than it does in the LLM era for what the parent said to make sense.

But those technologies are layers, and there are reliable things that sometimes bubble across the boundaries — type hints, better code patterns to trigger compiler optimisation, interesting tricks with key column selection — and someone with expertise from that layer below can explain why, and their advice will always work in situations that are sufficiently similar.

You are right about AI personalities. Obvious even with the open weights models. Gemma and Qwen write code and documentation like people from different cultures. Because I guess they are a bit like that.

The LLM providers fine tune the models with some kind of information retrieval tasks, but to do so you must provide some non relevant context to bootstrap the session for the long context tasks.

It would be very easy to do this in ways that train the sequence model to treat early history as noisier than it really is, or to weaken its relationship to late context.

You’re also probably stacking more contexts together with long contexts (start with task A, then detour to solving B and C before you can complete A).

Training sequence lengths probably decay super linearly with length creating far fewer samples at long length during training.

Defeatism doesn't do anything positive for the world. You're trying to convince the people pushing for a marginally better world that they should give up because it'll never happen. That is not a useful contribution.

Because that's what OP is talking about. Superstition presented as factual advice instead of the technically rigorous and scientific fact that already exists.

You're being downvoted because you don't understand this fact, or indeed understand what you're saying at all.

I'll spell it out for you: technically and scientifically rigorous facts do actually exist, even in regards to LLMs. We can, in fact, obtain scientific and objective facts about how LLMs perform. It can be rigorously proven that certain context habits affect certain tasks positively or negatively. Your argument is that none of this matters more than superstition. And you're surprised that arguing to a room full of engineers and scientists that science is dead and superstition is the one true way forward gives you negative response.

I usually don't have to worry about compiler optimisations because compiler experts do that; sometimes they appear in a thread about code and say "compiler guy here — if you write your code like this the compiler can optimise it".

And that person will be provably right (or wrong), in that context. And it'll be the same each time you run the test!

I just… ehh. You make a good point and I worry you are not wrong. It's all so different.

I like my 3D printing analogy much more than I wish I did.

I don't think OP is claiming that prior to LLM coding everything in the software development world was super rigorous (I assume that's effectively what you mean with the "rose-tinted glasses" comment). But rigor was actually possible and in a deterministic way too, which is fundamentally impossible with LLMs. You can build all kinds of guardrails and processes around LLMs that make it somewhat approach rigor again, but it's still fundamentally based on a bunch of statistical probabilities instead of deterministic, repeatable results.

All of the methods I see to mitigate the fundamental and inherent issues of LLMs seem roughly equivalent to the kind of crap you see in astrology groups or palm reading etc. You need Venus and Mercury to be in alignment while Mars is retrograde if you want to be able to get the right results from your token predictor.

In addition, you can co-author a plan for a biggish chunk of work, divided into stages, have it launch a subagent for phase 1 and check its work, then ESC-ESC to go back to just after you wrote the plan and have it do phase 2. Repeat until done. This keeps the overall goal in the main context for the review, but clears out previous reviews. Kind of like a workflow but with more control.

I imagine most harnesses should have a way to do this today, if they don't, get a new one. OpenCode i.e. is highly customizable, Claude and VS Code both support a ton as well including custom agents (though unclear if you can create custom top-level in claude-code)

https://opencode.ai/docs/agents/

https://code.claude.com/docs/en/sub-agents

https://code.visualstudio.com/docs/agent-customization/custo...

AI vendors still need to compete with each other both in terms of token cost and competency. An agent that is costly and less effective by wasting tokens is less competitive.

* push("What I'm about to do"),

* pop("What I've achieved").

"Push" marks the position in current context after the call and returns "Proceed", "pop" erases everything after the matching call and replaces "Proceed" in it's result as what was passed to the argument to the "pop", effectively pruning long-winded head even inside one reasoning stream. In the end the model only sees how it decided to do work on something, and that it was already done, forgetting everything it between, except what it itself decided was important.

Gemma 4 31B QAT successfully uses it when navigating a maze, marking positions at intersections, exploring them, navigating back and pushing again if necessary. Smaller models often fail to mark positions and forget to backtrack as well, instead they try to rely on themselves to track their paths and navigate back (and also fail).

I think it should work for long-running deep research tasks, but I was too lazy to test it, because it all required a lot of code to glue this up, since most tools and libs are not designed to work like that, and now I'll need even more code to test it, without a purposeful task.

I don't bother warning it in the system prompt anymore. It's pointless. I let it bump its head as required. A few hundred tokens and the agent is back on track each time.

  [User] Actual human prompt
  [Agent] Attempted use of tool & hand slap
  [Agent] call(projection of user's prompt relative to discovered tool constraints)
    ["User"] Prompt from above call
    [Agent] Legal tool use
    [Agent] ... until satisfied 
    [Agent] return(summary that satisfies the prompt for this level of execution)
  [Agent] Additional call() invokes possible depending on returned summary
  [Agent] Final return(summary) from root ends this turn of conversation and user sees summary
  [User] Next turn of conversation initiated by actual human

The only tools permissible to root in my scheme are call() and return().

60k isn't much bigger than the system prompt.

I usually see this when the context gets "tainted" as I call it. The model gets stuck on a bad path and there's no way to bring it back without clearing the context and starting again.

Frequently it'll be something as small as 1 sentence of a prompt many messages ago.

When cases like that happen, I reset the context and try to be explicit about assumptions and requirements to keep it off the "tainted" path. Other times it's actually useful and agents will do things they normally wouldn't do once the state is tainted. For instance, if you're testing a chat bot's ability to stay on topic, you can seed the context early with what you want it to do. It generally will refuse initially but later on in the conversation it will still silently take that seeded context into account almost "subconsciously" and become more likely to do the thing it originally refused.

Do you have any old documentation that it's picking up and referencing? If you set all claude settings back to default do you see the same issue?

Which drugs?

Relax, I acknowledged this in my comment...

But, it does a good job following existing conventions in a codebase, as long as they're really consistent. So the more actively you enforce that consistency the more likely it is to do the right thing without memories or prompting.

I don't like "never do" or "always do" type rules in AGENTS.md or in memory, as it often over-interprets them and ties itself in knots trying to satisfy an impossible set of goals.

- Work Mode - HITL/AFK

- Problem Statement

- Who It Affects - Primary / Secondary User

- User Stories

- Business Case

- Why Now

- Success Critera

- In Scope/Out of Scope [Out of Scope v. important)

- Thinnest Slice (This I've found super valuable, means you max out the amount of 'product' for your buck and avoid diminishing marginal returns or overbuilding. Often I will build this)

- Eigenfeature - What is the larger feature we _could_ (but probably won't) which would solve for this use case and other stuff I might not have thought of

- Technical Notes

- Deps

- Schema Changes

- Risks

- Final Recommendation [go / no go, including on scope]

There's a note in my Claude / Agents MD which says no net new feature gets introduced without this and I get it to move through a pipeline of folders (active, approved, shipped, proposed etc). All runs in a system of MD files and have even created a little MD Kanban from the metadata!

In an interactive session, adding "Fine, but make the button red" after the model generated a first solution more than doubles the tokens used. As the model now not only gets the original code and the feature request but also the updated code plus the change request as input tokens.

Sending a feature request to an LLM and then sending the feature request again with "The button shall be red" only doubles the tokens used.