Any AI model a company offers to the public has to be released as open weights.
This wouldn’t cover every model you train, just the ones you make available to the public. Internal and research models wouldn’t be affected, and governments could still get access to unreleased models.

I am glad this post by Jake Gold is getting quite a bit of attention. I agree on its conclusions, but I have a far more cynical view of the current situation and the actors involved (and I am in no position of writing an open letter to Dario, so here we are).

I don’t think these frontier labs are being fully honest in their concerns about “safety”. They are not actually worried about humanity “losing control” of AI, they are primarily concerned with them not being the ones controlling it.

They sold the idea of superhuman intelligence and got themselves locked into a “winner takes it all” race, raising tremendous amounts of capital to fund it. But as it’s increasingly becoming apparent there won’t be a single winner to take it all, they are now scared not of their rivals winning, but of being the ones to run out of gas first. Hence these disingenuous calls for “pacing” and regulation.

Of course this is being packaged as “safety concerns” because they have to keep the “scary AI” facade up or their entire house of cards will collapse.

If “misaligned” AI itself poses any meaningful risks to humanity, we cannot afford to entrust these companies to manage it. They have shown us very few reasons to believe they would be worthy of the task, and a lot more evidence they wouldn’t.

But when they talk about “alignment”, they really mean alignment to their own corporate interests, not to some broader, idealized human values. They are concerned they would not be the ones deciding what everyone else can do with AI, and I would argue that there are way bigger societal risks in letting them institutionalize this level of control, than from AI going rogue.

If AI has to benefit the whole of humanity, it has to be truly open. It cannot be controlled by just one or a handful of entities.

If we don’t want AI to be the biggest driver of wealth and power concentration in human history, we cannot allow these corporations to vacuum up all the content ever produced by humanity and then keep the weights for themselves.

Obviously, even if frontier labs released their weights tomorrow, we’d still face a concentration in the compute power needed for inference. But this is not a short-term game, and in the long run, never bet against technological advancements making more compute power available to the average person. We won’t be able to run Fable 5.1 at home today, but we will be able to run models as capable as Fable 5.1 on devices we own in a not so distant future.

So yes, if we want to regulate AI (and I believe we should, just for different reasons), let’s make AI truly open. Let’s start with requiring to make every publicly available model open weights. It would be a good start.