Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod

if any guardrails fails, we suspend probes till cooldown.

also 1. every probe is bounded by hits/expiry time (whichever comes earlier) 2. hit budgeting happens with a token bucket at a global level, per probe was an overkill (numbers are configurable) 3. we even measure the execution time that probes have when active and suspend if that that takes longer than threshold (again configurable) 4. we even have budgets for the network bandwidth it would take (approximated by the size of payloads) 5. collection itself is bounded by max no of total snapshots we can keep in memory. 6. every snapshot has a size limit as well, every variable has a size limit as well. 7. depth of objects, no of objects, size of lists is capped by default.

latency delta varies by platform under load but is mostly negligible

nodejs: ~7-10ms python: ~4-9ms java: 1-2ms

the main reason for this is guardrails suspending probes, having loosened guardrails will increase this under load

regarding localization of failures.. absolutely we even report the error in the probe snapshot (confirmed by adding side effects in an expression and commenting out the guardrails during testing)

huge payload size doesnt matter.. we limit the objects depth, list length, remove duplicate refs from data etc.. even string length is truncated., but even if it happens, your request would still survive.

also, even if the collector dies or there's a network failure, your service remains unaffected, we just are unable to collect telemetry

we are boring under extreme conditions :)

These work only on either uncaught exceptions or wrapping up caught exceptions with their sdk. These tools will not help you with silent failures, like logic bugs where code executes cleanly without throwing, but produces the wrong business state. If every problem in your app ends up as an exception, sure you'll be able to catch the symptoms of where the exception got thrown. we can deal with these too, but these tools cant deal with the messy bugs where no exception fires.

> Such very mature tools exist that auto-instrument, collect variables from the call stack, and pinpoint error causes.

That is true for python using frame.f_locals (we use this as well)

nodejs only gives it only till the lasy async boundary, after that v8 itself drops this data. java only gives you the current frame, to get variables beyond that you would needs JDI/JVMTI which would block your threads, usually unnacceptable in production

To get around this safely, we add multiple probes all across the call chain and collate collected data using the traceId from the context (or thread id as a fallback);

> Does this tool only exist to shore up poor system design?

Returning 200 OK on a silent failure is 100% bad system design, I completely agree. But real-world production systems are full of legacy edge cases. (if that weren't true, L1/L2/L3 support team shenanigans wouldn't exist)

Also, the exception will tell you that an exception occured in order service in GET /orders/{id}/payment, your trace will tell you payment service is giving 404 for that order ID

what it wont tell you it happened becuase the webhook endpoint that your payment gateway calls is now receiving a new payment state called 'PENDING' and that you dont handle but still mark the payment as 'processed' for idempotency check. and now your order service is calling the payment service and its giving 404 because it never got written

Bad design. 100% Agree, but has happened IRL.

> putting engineers in a situation where debugging requires accessing unknown amounts of live sensitive customer data is generally considered bad practice (even if it happens often IRL)

I think tells that teams would go to these extents to fix issues. Not ideal. I agree.

> in a hurry to debug, it's easy to miss that a property should have been redacted; by then it's too late and sensitive data is exposed

fair critique. we currently use in-process rule engines to filter known sensitive patterns, and users can add on to it. but we are also building out-of-process secondary checks (using NER/classifiers) to sanitize payloads before storage. It requires strict rules, but getting verified runtime evidence is far safer and faster than blindly guessing and shipping trial-and-error hotfixes to production. or waiting to be too sure.. a luxury that might not be possible everytime.

> Rollbar etc aggregate errors and captured data to identify patterns before a human (or agent, or tool) ever takes a look at it.

There is merit in that as well, if you are looking at so many logs/traces, you kinda have to do it. We have a different approach, we use hypothesis driven conditional probing instead. probes are dropped dynamically as the understanding of the bug evolves in a session

exmaple:

console.log('hello');

const x = await getThisValueSomehow();

if (condition A) {

console.log('i m in condition A');

// do something;

} else if (condtion B) {

console.log('i m in condition B');

// do something;

}

You can also place a probe before the branch to capture variable state when neither condition evaluates to true. You gather precise data on demand rather than paying to store petabytes of static trace data.

> A single captured instance can also be very misleading as to the true cause.

We collect multiple snapshots per probe run. However, because we capture full variable state at the exact execution line, a single snapshot frequently reveals the root cause for that specific failure path. If that snapshot raises new questions, you/your agent simply drops more probes deeper down the call chain

Thanks! This was very insightful

we work at the application layer by hooking into production grade tooling if available (inspector in v8, sys.monitoring in python) or bytecode manipulation(jvm)

since we arent controlling the application for a different process, we dont need to freeze the app to get the current app state

requirements we had in mind in order of importance

1. safety -> user app needs to function as usual no matter what happens, there shouldnt be an error in the user's app because of us

2. zero idle footprint -> if no probe is active, cpu/memory differency in the user app should be immeasurable

3. zero latency footprint at non probe paths while other probes are active

4. measure mem/cpu footprint directly or via a proxy like eventloop lag and have guardrails around it. suspend probes or even lose snapshot data if guardrail conditions meet

5. minimal mem/cpu footprint for active probes

6. minimal latency foot print for active probe paths

You're correct inpector API is not available in many non-v8 targets. Bun also has somewhat of a partial support for inpector API but at least has a programmable debugger interface. It's not going to be as fast as native inspector but its better than nothing i guess :P

for python sys.monitoring. for JVM, we do bytecode manipulation itself.

bundlers are not an issue because we support sourcemaps. We just need mappings, not code in the sourcemaps and we do sourcemap resolutions out of process so that your app doesnt spend ~200 MB of memory for parsing sourcemaps

all the rules get compiled into a single regex pattern, that lets us save on iterations.

You're correct, serverless is a bit tricky. CPU gets suspended the moment your function returns. The way it works is that you wrap your functions with a wrapper in our sdk.

that wrapper is supposed to track if there's telemetry to be sent, if so.. it sends it, otherwise, return as usual

this makes sure that when there's no active probe, there's no latency added. But when there's an active probe.. ~100-200ms could be added in the worst case if the probe is just before the return.

again, this isnt a problem in non serverless worloads because the CPU is always on.

but since probes are bounded by time and count, this will go away as soon as the time or count condition meets. beats adding new logs and redeploying in my opinion

Although, our primary sell is debugging, the context from production on how things work currently helps ai agents during feature development and code reviews as well.

what we wanted to convey is that sometimes people confuse "the symptom went away" with "the root cause was fixed"

I have seen that a rollback, a quick redeploy, or a temporary drop in tenant load makes the alerts go away and issue is considered resolved. specially true for larger teams with many engineers and services

a real example: a dev got OOMed after a release that coincided with a flash sale. he increased memory limits, and containers stopped crashing and it was "fixed". Actualy, a newly introduced internal module had a memory leak. adding RAM just hid the leak until the next traffic spike.

hyperprobe exists to capture actual in-memory runtime state during live traffic so you can prove the root cause before changing code or scaling infra in this case

But when the pipeline fails (bugs happen that's fine) re-running the exact same process may not be the solution.

Where does the additional intelligence that wasn't there before come from? We ran the pipeline that got it to prod on the same exact models you have access to. So the value prop is that you read the logs automatically instead of a developer directing a debug session?

Also thanks for the candid feedback! (And fair call on the design — we definitely prioritized shipping core functionality over UI polish, but point taken on the orange/brown palette, we'll change it)

a probe is basically just a definition it contains the file and line number of the code you want to inspect

that is safe by default. there's no custom userland function that you can call

if you want conditional probes, then it get's tricky

here's how it works

NodeJS: v8 natively blocks if the code even tries to produce side effects using `throwOnSideEffect: true`

in python and java: conditions can be set on local variables only without accessors as of now. you can call functions, can only a subset of comparison operators.

so, you can set `total > 50` as a condition but not `order.total > 50` as it could in theory trigger a getter (which ideally should be fine, but devs/agents can make getters with side affects so we wont allow it for now)

We will support these use cases with a custom DSL for agents + AST parsing in the future which will let us safely evaluate these expressions.

This is readonly and safe by default.

expressions come into play for conditonal probes

read only safety guarantees here depend on the runtime

NodeJS: handled implicitly by using `throwOnSideEffect: true` any side possible effects are prevented using this

Python and Java: As of now, we don't let conditions have method invocations at all and only allow a subset of comparator operators. no assignment allowed

usually property can invoke getter which usually should be safe to execute by design, but since we cant guarantee how it would have been written, we dont allow that as well for now.

order.total > 50 => not allowed

total > 50 => allowed

to get around this we use multiple probes, agrregated by the current context's traceId (if avaliable)

we plan to eliminate this problem by adding a custom DSL + AST parsing which can act as the policy layer to dissallow condtional probes

Audit trail is in our roadmap. As of now, you can delete the data that's collected by probes. the only problem we have with audit trail is what if you capture something sensitive and that remains in your audit trail.. so we need some immutability that registers audit trails.. but then have enough flexibility to remove the data collected.. can be done

It could work, the technology isnt the limitation.

But we were clear from day one that we cant let our sdks change the memory. Even if it helped solve a real problem.. say for example resetting a bad env variable or a feature flag without redeployment.

I might be biased from my experience, but i would prefer having a bug in my system for longer that i can reliably reason with than having it solved dynamically within the app which adds another thing to keep in my mind.

for me, bug -> fails -> good bug + dynamic patch -> works -> bad

also we dont think that we ourselves wont have any downtime ever, so we design for it. we'd not want to become as critical for your app as say your database.

As of now, your app works even if our servers are down/blocked/slow, adding the ability to change memory on the fly could change this

trying to debug using our tool might even lookup some memleak candidates in your primary container, but there wont be conclusive evidence for it and it would say so.

for in app errors all we do is hypothesize and either prove/disprove that using data from running system.

and whenever we do report something we give have the evidence for it. its not fool proof but just asking does this hypothesis gets proved with this evidence in a subagent mostly does the trick