Show HN: Mini-AGI – Dynamic continual learning model trained on 8GB VRAM

I am not sure about eventually, but it learns on the steady paste so far. The main thing to keep in mind is that it learns on the single STREAM of data. Not randomized batched samples. Try to do it with any other model and you will see nothing but complete garbage in the predictions, exactly because of catastrophic forgetting.

And here are the types of samples the model produces after about a week of training:

    ==============================================================================
    step 191,447   391.3M of 7,879M characters (4.97%)   15 min   176 experts
    context 4,096 characters of 4,096   reading 1,046 char/s   still gaining +0.0412 deep into it
    grad norm 0.98 against a clip of 1   under the clip
    train loss 0.6540   lr 2.28e-04   evidence t -0.15 over 65.7 (effect +0.0660)   rate x0.753
    held-out loss 0.8012 +/-0.0335 nats   1.1559 bits/char   perplexity 2.23   gap +0.1472
      arithmetic 0.645   chat 0.790   chat_hermes 1.153   chess 0.510   code 0.703   reasoning 0.756   stories 0.596   wikipedia 1.256
    repeats 31% of 8-grams, greedy with no guard
    ==============================================================================

    --- stories ---
    prompt: 'Once upon a time, there was a little boy named Tom. One day he '
    [raw]  repeated 8-grams 2%
    was walking in the park with his mom. He saw a big box in the ground. He was very sad and sad.

    "Mom, can I have the box to the box?" he ask
    [adapted]  repeated 8-grams 5%
    was playing with his friend, a little girl.

    "Wow, Tom!" said Tom.

    "What's your name?"

    Tom said, "I'm playing with my friend."

    The little

    --- code ---
    prompt: 'def merge_sorted(a, b):\n    '
    [raw]  repeated 8-grams 61%
        return sorted(a, b)

        def merge_sorted(a, b):
            return sorted(a, b)

        def merge_sorted(a, b):
            return sorted(a, b)

    
    [adapted]  repeated 8-grams 22%
    """Merges the modification from the angles in the argument.
        This merges the angles in the argument.
        """

        def __init__(self, a):


    --- arithmetic ---
    prompt: 'add 4917 + 388 = '
    [raw]  repeated 8-grams 47%
    <think> 7+8+0=5c1 1+8+1=0c1 9+3+1=3c1 4+0+1=5c0 </think> 5305
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    a
    [adapted]  repeated 8-grams 3%
    <think> 7+8+0=5c1 1+8+1=0c1 9+3+1=3c1 4+0+1=5c0 </think> 5305
    mul 6287 * 49 = <think> 6287*9=56583 6287*40=251480 56583+251480=307063 </think> 307063
    mul 9428 * 17 = 

    --- chat ---
    prompt: '<user>\nWhat are you?\n</user>\n<bot>\n'
    [raw]  repeated 8-grams 20%
    In the given I am trying to find the maximum vertex should be an expression linear that should be given the maximum vertex and the maximum v
    [adapted]  repeated 8-grams 8%
    In the given I, there's another part of the given I, whose parts include anything they're discussed.
    </bot>
    <user>
    What is the given article by answer

    --- chat_hermes ---
    prompt: '<user>\nA train travels 60 km in 45 minutes. What is its speed in km/h?\n</user>\n<bot>\n'
    [raw]  repeated 8-grams 63%
    The train travels 60 km/h in 45 minutes. The train travels 60 km/h in 45 minutes. The train travels 60 km/h in 45 minutes. The travels are t
    [adapted]  repeated 8-grams 4%
    The train travels 60 km/h, which matches km/h.
    </bot>
    <user>
    Write an example of the following article:

    In the subject of the subject, include an imp

    --- reasoning ---
    prompt: '<user>\nWrite a Python function that returns the largest number in a list.\n</user>\n<bot>\n<think>\n'
    [raw]  repeated 8-grams 35%
    Okay, let's tackle this problem. So, the largest number in a list is a list. The largest number is a list. The largest number is a list. It 
    [adapted]  repeated 8-grams 20%
    Okay, let's try to figure out this problem. So, we'll find the largest number in a list.

    The largest number is always the largest number in

    --- wikipedia ---
    prompt: '== History ==\nThe [[Roman Empire]] was '
    [raw]  repeated 8-grams 48%
    a [[computer state]] of [[Computer Empire]] and [[Computer Empire]]. The [[Computer Empire]] was a [[computer state]] of [[Computer Empire]]
    [adapted]  repeated 8-grams 42%
    the [[United States|University]] of [[Candie]]. The [[University]] was the [[University]] of [[Candie]] where the [[University]] was the [[U

    --- chess ---
    prompt: '<g>1700 1-0 1. e4 e5 2. '
    [raw]  repeated 8-grams 0%   22 legal moves, then Nd3
    Nf3 Nc6 3. Bb5 a6 4. Bxc6 dxc6 5. O-O Bg4 6. h3 Bh5 7. g4 Bg6 8. d3 Be7 9. Nbd2 Nf6 10. Nb3 O-O 11. Nc5 Bxc5 12. d4 Bd6 13. Nd3 Bxf3 14. Qxf
    [adapted]  repeated 8-grams 0%   16 legal moves, then Ba3
    Nf3 d6 3. Bc4 Nf6 4. d3 Be7 5. O-O Nbd7 6. Be3 c6 7. Nbd2 O-O 8. c3 a6 9. Qc2 b5 10. Ba3 Nb6 11. Bxe7 Qxe7 12. Rac1 Bb7 13. Nf1 Rac8 14. Ng3

    --- self-knowledge ---
    prompt: '<user>\nhow do you decide which experts to use?\n</user>\n<bot>\n'
    [raw]  repeated 8-grams 2%
    The directory is not a vector of 512, which is why the new chunk is not an expert. That is why my window can be extended by that no matter h
    [adapted]  repeated 8-grams 1%
    The directory is not a vector of 512, which is why. There is not an expert involve
    </bot>
    <user>
    Can you write change_string? It should change the com

In English, it's "thinking differently", not "thinking different". This is not Apple.

To pick a couple of examples:

   --- chess ---
   prompt: '<g>1700 1-0 1. e4 e5 2. '
   [raw]  repeated 8-grams 83%
   Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Rxd5 Rxd5 Rxd5 Rxd5. Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 31 Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 Rx
   [adapted]  repeated 8-grams 2%
   Nxd6+ Bxc3+ Nf6 14. Qxd5+ Nf6 Rxe3+ Bh1 Nxd5+ Qxc6 Rf3+ Nxd5 Qe4+ Rxf6 Nh1+ Qxd5 Rf3+ Nxe6 Qh1+ Rxd5 Nf6+ Qxe3 Rh1+ Nxd5 Qf6+ Rxg3+ Ne1 Qxd5

^ THe model noticed you started the notation of a chess game, but its response is total nonsense. After "1. e4 e5 2." you can't go Nxd6+. For all kinds of reasons. You haven't got your knight out yet. Even if you had, it couldn't get to d6. Even if it could, there's nothing there it could take. If you did somehow in spite of all that manage to play 2 Nxd6+ the opponent couldn't play .... Bxc3+ because they haven't got their bish out. Even if they had it couldn't get to c3 even if it could there isn't anything there to take - you only have a pawn on e4 and a magical knight on d6. Even if somehow in spite of that, you could take on c3 it wouldn't be check and EVEN IF SOMEHOW ALL OF THAT WERE TRUE YOU ARE IN CHECK. You can't move your bishop you need to do something about the Knight on d6 which has you in check.

All the rest of it is similarly gibberish. I'm used to model training garbage but this is in no sense AGI. It's beyond nonsense to call it that.

Using the term AGI and not including any performance analysis. My AI calls it: "massive marketing overreach". Somebody called this slop in the comments.

As a professor who published on continual learning I'm leaning towards agreement[1]. It lacks any substance. No relation to related work, no description of algorithm, no ablation study, just hand-waving that we're feeding some data and "Chess is not forgotten".

This "how-continual-learning-works" markdown text is not an algorithm [2].

[1] https://arxiv.org/abs/2301.12530

[2] https://github.com/volotat/mini-AGI/#how-continual-learning-...

It doesn't show any indications of solving catastrophic forgetting.

The model is way too small and undertrained to make any generalization claims. I want to wait until it reads the whole corpus I gave and then test it on some simple established benchmarks to see how it will behave.

Some OpenClaw/Hermes-like agents actually work like that. (Can't find the link right now.)

I also like how it is very organic. It naturally grows and deletes unused elements, so in addition to traditional backprop there is also a natural selection happening in the background. Each new expert has 16 parents by the way, lol.

"How continual learning works" is the section you are looking for. The main experiment is described there.

>Maybe local AGI isn't so far-fetched.

Why settle for anything less?

Before they had personal computers I always figured the established computer experts were going to get their mainframes to do AI a lot sooner than it turned out.

That was a non-starter though, not many people could afford huge amounts of hardware in a centralized datacenter where you don't have unlimited access. How was the next level of progress supposed to occur if you didn't fully own the electronics?

It was pretty impressive when it required a forklift to move the CPU, but kind of forbidding too.

PCs took over fast for that very reason and rapidly became more powerful until they were quite capable of incredible amounts of automation well over 20 years ago for so many things.

The whole time since the mainframe days the thing that has held true for AI in automation, is that it has to take the same powerful, sophisticated PC hardware that already works so well without AI, and bring that to the next level in logical progression.

No dependency on remote data, remote storage, network or internet at all. Otherwise why bother?

As long as AI can not make the same stand-alone hardware outperform what it's already capable of beforehand, there's quite a bit more work to do.

An idealized automation workflow on a new PC without AI:

Insert blank SSD > Install OS > Install automation app > Program app > Run automaton

Same stand-alone hardware, with AI:

Insert blank SSD > Install OS > Install AI app > Train app > Run automaton

If AI can't make the same hardware run smarter, it's not as intelligent as it could be, is it?

I don't need AGI, I just need this.

I don't think there's much of advantage tbh. Memories in files will always outperform non-frontier model weight adjustments imo.

Do you find all your own (human) trained knowledge unreliable?

It interleaves random streams of 32K characters long each when reading the whole corpus, but each such stream reads continuously as you would expect. This is a necessary step to prevent just normal, not catastrophic, forgetting. I have not tested it in any other regimes yet with bigger or smaller windows. You can imagine a person that changes the activity from time to time, so I think it is justified. So there is not really "early in the stream".

What I did test though is reading 524K characters of chess data only and see how other domains have degraded. The results are in the readme under "How continual learning works" section. Spoiler: it just barely degraded the performance.

Making model to consists of many small modules is inefficient on GPU, especially as routing adds data dependencies, etc, and especially with pytorch (compared to a custom kernel).

The difference might be smaller on a CPU which has limited parallelism.

But it's basically equivalent to a very deep model which might be problematic for training.

The held-out scores reported in the Readme IS the unseen dataset.

I have no practical means of scaling it up at any compatible scale. I will not make any money on it either way as well. So there is absolutely no reason for hoarding it. And as I said I did use Claude in the process, so Anthropic already has full access to it anyway and could steal it just as easily if they really want to.

I also doubt it is really that valuable on the OpenAI/Anthropic scale, at the same time if people will use it and it will work for them on the personal scale it is already a major win for me. New ideas and optimizations I could never have thought of might bring this up from a toy model to an actually useful model trained locally. Then people could add RL and RLHF and other cool things to it to make it even better.