Kuna: Decompiler Development in the Age of Coding Agents

Posted by matt_d 4 days ago

Comments

Comment by mahaloz 4 days ago

Creator here! Since I saw some questions below, I'd like to clarify some things.

Kuna is originally Ghidra ported to Rust, but it has changed quite a bit since. The results shown in the screenshot on the site are not a feature in Ghidra, which includes the Switch layout and some of the variable simplification. It's from Ghidra, previously talked about here [1].

> It would be cool to see agentic interpretation of function and variable names.

On DecBench, which I also created last week, you can see just that for Codex and Claude when given only assembly [2]. They dominate, but also have some significant recall problems.

> Why not train an LLM to be a decompiler instead of make one

Over the last two decades, this has been attempted _many_ times [3]. To date, there has been no successful project/paper that can compete with traditional decompilers on fundamental metrics. I speculate this is due to their inability to abstract the many-to-one mapping of optimization to code. This can change, but I desired to take a different approach.

If frontier labs continue to get better, they will become the de facto decompilation users. Why not get LLMs to help LLMs? Design a core decompiler that helps them more than others :).

[1]: https://www.youtube.com/watch?v=VP29biKLoSw

[2]: https://decbench.com/leaderboard/?dataset=sample-set

[3]: https://decompilation.wiki/fundamentals/neural-decompilation...

Comment by comandillos 4 days ago

Haven't used IDA much lately, but after looking at the screenshot with that IDA PRO decompiled code in their website I feel like Ghidra is already ahead of them in this area :D

Comment by mahaloz 3 days ago

Interesting. A lot of data shows IDA Pro is significantly better than Ghidra: https://decbench.com/

Based on today's results, IDA Pro is ahead by 15 percentage points, which would mean, statistically, IDA Pro will recover perfect source code for 15% more functions than Ghidra on average.

Comment by Orphis 4 days ago

I have seen some code where either Ghidra or IDA is producing the better output. It's not cut and dry, but in general, I do prefer Ghidra.

Comment by fishfasell 4 days ago

It would be cool to see agentic interpretation of function and variable names. It can see and track the flow of data a lot faster than a human can so if given some context, maybe it could synthesize names for them.

Comment by snoopen 4 days ago

I've used Claude with Ghidra MCP and had good success. It correctly identified and named MD5 functions, file IO functions, and the whole chain from a UI dialogue function. And when prompted it named all the interesting variables too.

Comment by ChickeNES 4 days ago

I'm not sure you even need the MCP tbh, I just have Claude use pyghidra scripting.

Comment by hackcasual 4 days ago

I was working on adding better WASM support to ghidra, but now I can just point Claude to binaryen

Comment by nneonneo 4 days ago

Well, that's too bad; as the main developer of https://github.com/nneonneo/ghidra-wasm-plugin it would have been really nice to get better support for all the new features that have been added into the spec over the last few years.

I also think that, for larger projects, it's still useful to have real decompilation support; for example, Il2CppDumper works in tandem with ghidra-wasm-plugin to enable decompilation of Unity Il2Cpp projects, which are often enormous (100MB wasm files are not uncommon) and for which the symbols + types are invaluable.

Comment by bpavuk 4 days ago

I wonder, how many such interesting and universally beneficial developments did emergence of tools like Claude Code kill?

Comment by billypilgrim 4 days ago

idalib with Claude Code already works really well. But honestly, despite what people have been saying, LLMs have been very good at decompiling for at least two years now, I have been using it for that purpose regularly. Even just copying disassembly from the current function and all nested called functions from IDA into ChatGPT is already unexpectedly good.

Comment by saidnooneever 4 days ago

this is really amazing work thank you and in my opinion (as its explained) a very good example of how to use AI powered development and research to advance the tools we have. decompilation is really hard, and better algorithms for it are very valuable contributions for many areas of tech.

Comment by ur-whale 4 days ago

Not sure why someone would want to use an LLM to build a new decompiler instead of training an LLM to BE a decompiler.

Especially given the fact that you have an infinite training set to train that LLM from (compilers can generate as much training data as you could possibly want).

Comment by Tepix 4 days ago

LLMs are famously inefficient, expensive and unreliable.

Comment by ur-whale 3 days ago

> LLMs are famously inefficient, expensive and unreliable.

Maybe, but here I am suggesting an LLM specifically trained to do only machine code to high-level code reverse engineering.

I suspect this can be made both efficient and very reliable.

Especially given the fact that there is very often a path the verify if what the LLM produces is a valid answer (by testing if the high level code produced by the LLM, when compiled, behaves like the original machine code).

Comment by mahaloz 3 days ago

Answered above, but the gist is people have tried: https://decompilation.wiki/fundamentals/neural-decompilation..., and yet general LLMs from Anthropic and OpenAI appear to match or beat all of these works :(

Comment by gopalraja 4 days ago

Yeah, fair question.

I read this less as “build a decompiler with an LLM instead of training one to decompile,” and more as a different loop: you still get an artifact you can measure and improve against IDA / Ghidra / angr. Training on infinite compiler data is powerful for the first path; this post is mostly about the second one (i think, but i could wrong).

Comment by Rexxar 4 days ago

There is no point to have an LLM do what can be done faster and deterministically by standard algorithms.

Comment by ur-whale 3 days ago

> There is no point to have an LLM do what can be done faster and deterministically by standard algorithms

I think using the word "deterministically" in the context of decompilation is not a particularly good idea.

The code generated by a compiler can be arbitrarily complex and arbitrarily varied even before you take into account willful attempts at obfuscation to protect against reverse-engineering (for example, try futzing with compiler optimization flags to see how much the ASM changes).

The "high-level code to assembly" operator is thus not even really a function (same high level code can generate infinite variation of functionally equivalent assembly code) and any attempt at producing an inverse of this operator ... this really does not strike me as an environment where the word "deterministic" seems appropriate.

Comment by mahaloz 3 days ago

I would call any traditional decompiler deterministic, though these can rely on probabilistic models to infer how to undo these optimizations.

A talk I did on this at Reverse Con (clipped to the important part): https://youtu.be/VP29biKLoSw?t=891

Many of these decompilers can infer what optimizations occurred by hints/patterns that are left behind.

This can be fundamentally different at times from LLMs, which have shown a degradation on unique architectures. I still believe the best approach will be a mixed method. Get 80% of the way there with traditional methods, and take the last 20% (which is the hardest) home with LLMs.