Promethevs: Full DeepSeek-v4.1-Flash-1017, locally, on 12GB of VRAM

Posted by ar-nelson 1 day ago

Counter13Comment5OpenOriginal

Comments

Comment by suiwen2020 10 hours ago

[flagged]

Comment by rvz 1 day ago

If you were wondering:

> Up to 7.4 tokens/s.

It's not really "Flash" if it is 7 tokens per second.

Also don't bother with any GPU lower than an RTX 5080 or even a DGX Spark (it is a scam) if you are seriously considering running frontier local models.

Comment by pu_pe 22 hours ago

It's apparently running at ~11 tok/s on a 4090: https://github.com/Magnvs7/Promethevs/issues/7

Strata runs considerably faster than that on similar hardware, though this one doesn't seem to be a fork.

Comment by Ancapistani 23 hours ago

Oh, I don't know - it could be useful for long-running and/or background tasks in some cases.

The problem is that this particular model is so cheap, compared to others. Optimizing to host it locally with relatively poor performance just doesn't make sense when yesterday I burned ~180M tokens for $2.20.

Comment by cassianoleal 19 hours ago

Surely local hosting is not so much about cost savings as it is about data ownership and risk management. e.g. can't leak a secret if you're not calling out.

Comment by a3w 1 day ago

Strata for running from HDD seems really popular, to get big models running.

Is that faster?