Show HN: High-Res Neural Cellular Automata

Posted by esychology 6 hours ago

Counter141Comment35OpenOriginal

Neural CAs model self-organizing pattern formation.

Now they can generate patterns at HD resolution in real-time, enabled by turning each CA cell into a Neural Field.

Try 3 demos: grow a pattern from a seed (and damage it, it heals), synthesize PBR textures that can regenerate, or create 3D textures like clouds.

Comments

Comment by whilenot-dev 3 hours ago

The automata just completely destroys the image if I draw too much over the stabilized image with the brush. 5 horizontal swipes are enough to destroy the kitty, is that to be expected?

EDIT: video here: https://imgur.com/a/ItZGd5X

Comment by esychology 3 hours ago

The NeuralCA both generates and maintains the pattern. Because the NCA was not exposed to damage or erasure during training, its regeneration capability is a purely emergent phenomenon. However, this ability remains somewhat brittle, particularly when the central regions of the pattern are erased.

Comment by mackenney 2 hours ago

I would love to see two seeds competing for space in the grid

Comment by WhiteNoiz3 2 hours ago

With the old model (and I suspect this one too) it's trained to generate from a single 'seed' pixel in the center of the image. If you erase the center of the image, that's when it completely collapses.

Comment by oersted 1 hour ago

It must be more general than that, otherwise the cells wouldn’t be able to repair their area if the damage came from the wrong direction (repair is not center-out).

The model generally learns to generate each pixel from its surroundings, even if the surroundings are partially missing.

Comment by WhiteNoiz3 1 hour ago

There's hidden state in the model which presumably it uses to communicate position, ie there's the 3 colors but then a bunch of other channels that the model can use how it wants.

Comment by cl3misch 1 hour ago

Have you actually tried that? If you specifically erase the center, the image does change a lot at first, but rebuilds itself eventually (albeit to a slightly different final state). It's uncanny how "biological" is feels!

Comment by WhiteNoiz3 1 hour ago

I have yes.. You need to erase a larger amount of the center, but it almost always results in a collapse wheras erasing around the center typically regrows.

Comment by Mithriil 1 hour ago

If you hold the eraser for a second at the center, I find that it destroys the image more often than not.

Comment by zcw100 1 hour ago

I've always loved the original work and it's nice to see they're still working on it. I've always wondered if there was a way to connect this back to infrastructure rather than images. Something you could run on a cluster and if portions of it failed it would regenerate the system.

Comment by WhitneyLand 2 hours ago

At a glance it looks like it could be just iterative texture sampling.

The difference is when creating each pixel, there’s no coordinate to look up, instead it’s using only a set of rules like Conway’s game of life.

But the rules come from a neural network trained on the image, so… it’s kind of memorizing enough information to effectively do the same thing as texture sampling, but using only local information.

I’m sure I’m missing something about how it works or what makes it interesting…

Comment by oersted 1 hour ago

To me, it is intriguing as a toy model for how cells are able to grow into complex tissue and organisms based only on local information, and how they are able to repair and recover harmed tissue.

Of course, this is as close to cells, as neurons from neural networks are to real neurons. And I have no idea what it could be applied to (inpainting/outpainting?), but it’s interesting as exploratory research.

Comment by menno-sh 10 minutes ago

Oooh, this made it click for me. Thank you

Comment by esychology 1 hour ago

[flagged]

Comment by jekude 4 hours ago

The abstract implies that strictly local updates are a hinderance to high res, however i would have thought there would be an interesting way to get speed up gains from neighbor-only traffic on GPUs CAM-style. am i making that up?

Comment by esychology 2 hours ago

I think performance is not the only issue for scaling to larger grids. CUDA Convolution implementation already utilizes coalescing to improve performance. The main bottleneck is that in larger grids, cells are further apart, and it takes more steps for them to be able to communicate.

Comment by hidelooktropic 3 hours ago

For the unfamiliar, could someone explain what I'm looking at? The abstract was a little too concrete (heh) for me to follow.

Comment by esychology 3 hours ago

If you're familiar with CAs (e.g. Conway's Game of Life), you can think of a NeuralCA as a CA where the update rule is given by a neural network. Here we optimize the neural net weights so that it behaves a certain way (e.g. grow a lizard from a single seed).

Comment by flir 3 hours ago

What are the inputs to the NN? The whole grid, or just nearby cells? What happens if two NNs overlap on the same grid? (Gonna go read the paper).

Comment by esychology 2 hours ago

The input to the NN is just the 3x3 neighborhood around a cell. We can overlap two NNs on the same grid (through interpolation). Checkout https://meshnca.github.io to see the effect. When the brush is in graft mode, it basically allows you to paint some regions that will follow a different NN rule.

Comment by flir 2 hours ago

> The input to the NN is just the 3x3 neighborhood around a cell.

Well that sounds like black magic. Nice. Thanks for the reply.

Comment by soraki_soladead 1 hour ago

The original NCA is probably a helpful intro: https://distill.pub/2020/growing-ca/

Comment by embedding-shape 4 hours ago

Really interesting demo, nicely done :) Would be fun if switching the "Target Image" when using the second brush mode in the Growing Demo didn't erase/reset the existing canvas, so we could "stamp" new things on top of other images. Small thing perhaps but I got sad when it disappeared when I wanted to merge a kitten on top of the chameleon but couldn't :(

Comment by esychology 1 hour ago

That's not possible in the current demo but this sounds like an interesting feature to work on and add!

Comment by bfmalky 3 hours ago

You can, just enable the 'transition' switch.

Comment by embedding-shape 1 hour ago

That seems to be something else? It takes the current image and "transforms" it into the new target.

Comment by WithinReason 4 hours ago

You can make the centipede grow longer, which makes sense given how this works. Or grow a 2nd centipede for extra points.

Comment by esychology 4 hours ago

haha yes, also the same with the worm

Comment by moralestapia 47 minutes ago

@esychology this is phenomenal work, thank you so much for sharing it. I am working in a similar thing and might reach out about it soon.

Also, what's going on? Why would the community flag and kill this comment[1], from the creator itself. If you're jealous of what the guy built, take it elsewhere. HN will implode with that attitude.

1: https://news.ycombinator.com/item?id=48571171

Comment by esychology 43 minutes ago

Thank you for the kind comment! Please reach out, I'm happy to have a chat.

Comment by bjourne 11 minutes ago

It's the stupid spam filter going haywire: https://news.ycombinator.com/item?id=48321198

Comment by amelius 4 hours ago

Why are the images always generated in the same orientation (upright)? Do the cells have awareness of what is "up"?

Comment by WhiteNoiz3 1 hour ago

IIRC training starts with the initial state and the end state, and the end state is always oriented the same way. It would be interesting to see what would happen if the end state was rotated randomly though I suspect it wouldn't work so well.

Comment by esychology 4 hours ago

yeah normally NCAs have a sense of up and left. There are some isotropic variants that make the perception fully rotation-invariant.

Comment by bjourne 1 hour ago

I found your previous work here: https://distill.pub/2020/growing-ca/ For someone (like me) who wants to understand the basics its probably better. It's very well written.

Comment by mirekrusin 3 hours ago

So the goal is to evaporate it with minimum number of shots?