GPT‑NL: a sovereign language model for the Netherlands

1. Free of controversy like unlicensed training materials

2. Free of exploitative rlfh loops by people in low-wages countries

3. The leasons learned (and published) from going through the entire training process on "European" hardware: "AI factories" (the term for Slurm HPC/HTC systems with lots of heavy GPU nodes, heavily subsidized by our government [0])

1 and 2 are strong counter-LLM arguments at the moment, and hold back some groups of potential users. Another is energy/water use, so going for maximum green energy would be a nice boon as well. 3 is something I consider to be highly useful for our European identity and "way of the ninja" (for you Naruto fans out there).

[0 https://hpc-portal.eu/funding-opportunities]

Assuming us Europeans finally get our act together, I think it is better for our long-term future (and the ethical problems) if we manage to get a baseline of training input and data ourselves, from scratch, with everything being ethically sourced.

Oh and, while we're at it, the EU has 24 official languages plus a host of minority languages. Most LLMs focus on the English, German, French and Chinese languages, but everything else is... left behind at best. An European model with actual funding and proper data sources might be able to significantly reduce that.

[1] https://www.taiwannews.com.tw/news/6245677

[2] https://www.theguardian.com/technology/2024/apr/16/techscape...

The problem is that there are a lot, at least 30, of these small projects scattered around, funded for a few years as some ad-hoc temporary coalition of universities and businesses. Those simply cannot compete with businesses spending tens of billions on developing these. Especially when you have to bring a spoon to a gunfight restricting to "clean" data.

Multilinguality is essentially a solved problem, and restricting too much on one language with more limited resources is gonna make the model worse in that language too.

Sure… they can, except at the end of the day it’s a bit late, regulatory burden will make it comparatively useless, and because of that nobody will ever use it. It will be spending a bunch of taxpayer dollars for press releases.

The running joke is that when these “sovereign” EU models launch, they’re going to refuse to answer anything that might involve personal information such as Elon Musk’s birthday.

When you reward the most selfish, corrupt, and antisocial behaviours with wealth and power, you're guaranteed to create a selfish, corrupt, and antisocial society. IMHO it's indicative of what I have dubbed Americas "mental illness epidemic"; specifically cluster B personality disorders [0] which are characterised by socially-destructive and self-destructive behaviours.

If that's the world you want for you and your loved ones, congratulations. You've earned it!

[0] https://wikipedia.org/wiki/Classification_of_personality_dis...

No thanks.

Why do you feel grinding insane hours would be beneficial to AI progress?

The government would be far better off figuring out how to take commodity models and applying them to government functions where they can, with deterministic scaffolding and guardrails, to make government more efficient, optionally using RL on traces from their use to improve their performance.

Imagine taking models and fine-tuning them / doing RL rollouts to help automate permit application approvals, as applied specifically to Dutch permit processes. That would be a real help to Dutch businesses!

That type of applied AI is more interesting and effective now than just trying to make another foundational model that isn't going to work well or do anything of economic value.

https://www.siliconcontinent.com/p/nineteen-thoughts-on-ai-a...

> GPT-NL was never built to compete with Claude or ChatGPT. It was trained exclusively on licensed data, and is intended more for governments and companies where privacy and compliance matter more than raw performance.”

That's it? That it didn't aim to compete with SOTA models? Maybe this is something you have to start with something, then ramp up, rather do what only a select few labs been able to do, start with really big models. Especially if you're resource constrained, which since this is a government project, I really hope for the sake of the tax payers it was.

Furthermore, the expertise in designing and training these models is valuable as well. The existing models are good as a starting point in terms of learning from previous mistakes, but we should not just let a handful of American and Chinese people keep the knowledge and expertise.

One problem with this particular project, though, is that copyright has been enforced for Dutch LLM training before, and the AI industry cannot exist without massive scale piracy, the likes of which has never been seen before. A lot of Dutch training material exists in pirated books that AI companies in countries that do not care about copyright have access to, but are exempted from the training set here. The impact of enforcing copyright on an AI model will be quite interesting to see.

They are not neutral technology, they are a direct representation of the training set that has been chosen and how they are aligned.

In many ways, they are ideology made code.

If we leave building them to the US and China, only their way of seeing things will be digitized.

I don't like the idea of that.

Yeah but Europe doesn't build any computer hardware, and EU Green eco-communists and NIMBVYs don't want to have data centers built in their backyard, so the only way left for EU consultancies to milk taxpayer money for the AI bubble, is shipping a sovereign AI model for each country/language.

Watch out US tech sector, we're coming for you. Feel our wrath.

Some would consider that a good thing. There is a lot to be said for VC in recent years not being beneficial for the economy, certainly on an individual level, other than "number go up".

Arguably, staying out of the AI "race" is a good thing

I don't know if it is the right strategy but there's certainly a legitimate strategy in there.

Regulations are not even throughout each of the 27 member states. Each country is relatively small in the world stage.

Until EU progresses towards federalization, discussing this is a moot point.

My ex-neighbor (when I was a teenager, living in Belgium) and very good friend really wanted to make it big. He became a chip engineer, moved to California, raised money for a first startup (it tanked) then raised money for a second startup. He made the world a better place (he created some very specific micro-inverters for solar panels) and made a $$$ exit.

The EU saw exactly zero of the wealth he created and he's never ever coming back to what he considers a failure of a continent.

That's the problem: many of the great minds with the mindset required to do great things already left the EU.

> So yeah, both in economy and warfare, Europe dooms itself to be in the hands of the US, China, or a mix of both.

And in energy (economy is energy and energy is economy, and China really understood that) the EU doomed itself to be in the hands of Russia.

We are a failure of sinking continent.

They're training a model, not funding a startup. €13.5 million is plenty to pre- and post-train a decent model.

I was actually surprised by how little it was.

All these tiny niche models are perhaps fun as an academic exercise or great for the researchers resume but I highly doubt that they'll add any value or will be used for anything serious.

Even if this becomes a somewhat decent model with a fantastic understanding of "gezellig", "kring verjaardag" or "pannenkoeken", how many people will interact with it before the limits of it will drive them back to a frontier model?

Even if the purpose of this is government & other regulated industries, do we really want our government to use a poor model? Either do it right or don't do it at all.