Can you reverse engineer an ASIC?

This is absolutely not how reverse engineering a digital logic ASIC works.

Either the story got embellished through retellings, or this person was a fantasist.

There are people who hack on GPUs but it’s done at the software level.

I did get a kick out of imagining a scene where someone is trying to connect an oscilloscope to a circuit board to reverse engineer the CPU opcodes. That’s like the CSI: Miami version of what this would look like.

You'll have more luck reverse engineering the software driver first. They're not hidden, you can just open the driver files in Ghidra, the almost-universal tool for open-sourcing proprietary code. Hidden opcodes can be discovered first by just trying all the opcodes you couldn't discover any other way. You only need to go to the physical level if they're really hidden.

You can't tease out the right information by applying random inputs. Which input would you even use? The PCIe interface? You'd first "randomly" need to get past its complex training sequences...

Your postdoc probably wrote micro-benchmarks of some sort. That is a common technique.

Modern systems tend to use "standard cells" for logic which is a lot more digestible.

This is just a job posting in disguise for Jane Street: solve the puzzle, get a call from one of their recruiters.

Circuit designers use these tools basically daily for two reasons - the first is Layout Versus Schematic. We want to make sure that the physical layout matches the schematic, so the tool turns the layout GDS into a netlist and compares that to the netlist created from the schematic (basically a diff, but more complicated). The second is so we can run simulations that take into account the "parasitic" resistances and capacitances of the wires and metal shapes in the physical layout. It's basically the same procedure as LVS with an extra step that analyzes the metal shapes to determine said R's and C's.

Theoretically Yes, as an ion-beam-mill and electron-microscope combination machine can slice up semiconductors layer-by-layer. Given these machines can often also give precise x-ray analysis material data, the exact makeup of the chip can be extracted by competitors given enough time. =3

They weren't trying to steal IBM's design, they already had one of their own, what they wanted to know was "is there any as yet unannounced functionality in the IBM VGA chip?" they didn't want to be caught out when some unknown registers popped up. Of course it turned out there was as yet unannounced functionality - but not by IBM's design, more because of the orthogonality of the design - MODE-X was discovered and became the basis for DOOM's speed, anyone who didn't support it lost out