It is counterproductive to force cache on all tools of an agent

Posted by istvan_intrnd 16 hours ago

Counter1Comment1OpenOriginal

To make agentic cache work, LLM providers force frameworks to add all tools into the cached prompt. This is the main reason why I cannot properly use cache in my own agentic framework, where the set of tools changes in function of objectives. It confuses most base models, and however many times they say it doesn't my observations contradict this. How about yours?

Comments

Comment by sshadmand 10 hours ago

Don't they have to cache in order to access the tools quickly enough to run the agents? I imagine not doing slow would slow down the progress considerably. Or, maybe I misunderstand which part of the flow/agent/work you mean by "cache on tools"