Agentic Monitoring

Posted by bcongdon 3 days ago

Counter120Comment22OpenOriginal

Comments

Comment by nevon 2 days ago

I feel like this is a bit of a "draw the rest of the owl" problem. With a couple of terabytes of logs per day, god knows how many metric timeseries, thousands of cloud environments, thousands of repositories, hundreds of thousands of build pipelines running every day, it's not something you can just point an agent at and call it a day. Segmenting and correlating the data is _hard_. Even something as seemingly simple as "p95 latency of this endpoint has increased - what repo contains the responsible code" is difficult to do, even if you are willing to give an agent read access to everything.

Comment by jeffbee 2 days ago

Yeah something about this whole concept is way, way off. The energy, and therefore cost, of an LLM analyzing every data point of every time series is many orders of magnitude too high, unless your operations are trivial.

Comment by srdjanr 2 days ago

I don't think the idea is to give agents raw data, but for it to run many queries. Humans also don't look at terabytes of raw logs.

Comment by jubilee33 2 days ago

I encounter this problem a lot in my work. Agents being run on a cron for instance, where the task is not dynamic and a simple script can be more than sufficient. As agentic use increases i think it will get worse, as scripting is hard to do elegantly, even if the agent is writing the script. But having an agentic call for every single detail is just so inefficient.

Comment by antupis 2 days ago

I think there running both will be winning combination scripts/classical ML do large scale crunching and when they find some anomaly or some threshold creates alert. Agents will create initial report and maybe create first pr for developer.

Comment by ibraryunus 2 days ago

[flagged]

Comment by suncemoje 2 days ago

> With a couple of terabytes of logs per day, god knows how many metric timeseries, thousands of cloud environments, thousands of repositories, hundreds of thousands of build pipelines running every day, it's not something you can just point an agent at and call it a day. Segmenting and correlating the data is _hard_.

In my experience, this is exactly what LLMs / a simple agentic system with one or more sub-agents and a few tools are extremely good at.

Comment by cloudrkt 2 days ago

Could a system one model like Jev not provide some sort of "pre-classification" on these logs to make this in to something workable?

Comment by 2 days ago

Comment by t0mas88 2 days ago

Interesting idea. I guess the main challenge is to create some kind of harness to make the input volume appropriate for an LLM to evaluate. Billions of the same log events from a raw stream will be cost prohibitive and not find you much interesting output. But a trend in some value changing would.

As a human I look at Datadog or Grafana plus some log searches, maybe those need to become tools for the agent.

Comment by suncemoje 2 days ago

> In my iteration on this, I’ve found that a pure agent loop is not the optimal shape for agentic monitors. Rather, a harness that allows the agent to write a spec of how to monitor, along with a deterministic system for running those specs, produces significantly cleaner signals with dramatically lower token usage.

Would be curious:

1. How / at what interval does it run those specs?

2. Are these basically just queries? Or what's included in a "spec" that an agent executes on?

Comment by solomonb 1 day ago

I made a project for this earlier this year: https://github.com/solomon-b/friendly-ghost/

You can can read from Loki or Journald and use regex or alert levels to filter what you pass to the LLM.

I've been using it with deepseek for my homelab. Its pretty helpful.

Comment by yegle 2 days ago

I have been doing something similar but much smaller scale: having a daily scheduled task from Antigravity to ADB connect to my phone and dump all logs and do analysis to let me know if there's any app behaving weirdly (e.g. repeated crashes).

The adb connection is intentionally over Tailscale so in the morning after I connect to some random WiFi (Android remote debugging is only enabled w/ WiFi connection) I can start reading the report, then quote and ask Gemini to fix some issues.

Comment by skyfine 2 days ago

We have been doing this recently after some security incidents. Constant monitoring of all security related events, and being able spot anomalys which our normal system isn’t able to catch. The active part is just emailing us right now, but I am considering adding some light lock down actions.

Comment by strixgov 1 day ago

[dead]

Comment by jDucky 1 day ago

[flagged]

Comment by kelseydh 2 days ago

It feels like the real value here will be from monitoring services like DataDog adding this functionality to their own internal products, rather than companies rolling their own agentic monitoring. Datadog's Bits AI is already quite helpful in scanning monitoring data for insights.

Comment by theshrike79 2 days ago

Without even knowing what it is, I'm kinda guessing Datadog's AI also has a nice hefty price tag attached to it?

And if not today, it will have one soon.

Comment by 2bird3 2 days ago

Feels like the author hits the nail on the head but leaves out higher cardinality data. Would love to know how Anthropic goes about ingesting and analyzing their logs, of which they presumably have many.

Comment by appplication 2 days ago

Ingesting? Presumably Kafka. Analyzing? Presumably a fairly large analytics org.

Comment by rihegher 2 days ago

Claude code is regularly calling some datadog API

Comment by forgotmypw17 2 days ago

This is already a product: https://www.tasksmind.com/

(Disclosure: Not mine, but a friend's.)

Comment by esafak 2 days ago

How are you setting the alerts? Are you using anomaly detection? If so, what kind?

Comment by renegade-otter 2 days ago

This is like tokenmaxxing for observability?

Comment by simianwords 2 days ago

This is a thought terminating cliche. There’s a tendency to sneer at any proposed change to how the industry works because apparently we are already at the global optimum and every new change is just marketing and hype.

It’s so obvious that we will have round the clock “intelligent” monitoring.

Token costs are getting cheaper by the month. There’s a 2x drop in prices every month or so.

It’s more useful to predict where the industry will go forward.

Comment by simianwords 2 days ago

https://news.ycombinator.com/item?id=50134310

I made a comment touching on similar things a while back. This is obviously the future.

Expect a lot of techno conservatives and greybeards pushing back on this.

“Just have better unit tests”

“Just have traditional alerts using SLO budgets”

Comment by productgrowthai 16 hours ago

[flagged]

Comment by khalidsaidi 1 day ago

[flagged]

Comment by radim_h 1 day ago

[dead]