The basic premise is that, in the near future, a first “seed AI” will be created, with general problem-solving abilities slightly surpassing that of humans. This seed AI would start designing better AIs, initiating a recursive self-improvement loop that would immediately leave human intelligence in the dust, overtaking it by orders of magnitude in a short time. Proponents of this theory also regard intelligence as a kind of superpower, conferring its holders with almost supernatural capabilities to shape their environment (…) This science-fiction narrative contributes to the dangerously misleading public debate that is ongoing about the risks of AI and the need for AI regulation. In this post, I argue that intelligence explosion is impossible — that the notion of intelligence explosion comes from a profound misunderstanding of both the nature of intelligence and the behavior of recursively self-augmenting systems.
Exhaustive post by Françoise Chollet (author of Keras, currently working on Deep Learning at Google) confuting the dangerous misconceptions that fuel the AI debate.
Any AI model a company offers to the public has to be released as open weights. This wouldn’t cover every model you train, just the ones you make available to the public. Internal and research models wouldn’t be affected, and governments could still get access to unreleased models.
I am glad this post by Jake Gold is getting quite a bit of attention. I agree on its conclusions, but I have a far more cynical view of the current situation and the actors involved (and I am in no position of writing an open letter to Dario, so here we are).
I don’t think these frontier labs are being fully honest in their concerns about “safety”. They are not actually worried about humanity “losing control” of AI, they are primarily concerned with them not being the ones controlling it.
They sold the idea of superhuman intelligence and got themselves locked into a “winner takes it all” race, raising tremendous amounts of capital to fund it. But as it’s increasingly becoming apparent there won’t be a single winner to take it all, they are now scared not of their rivals winning, but of being the ones to run out of gas first. Hence these disingenuous calls for “pacing” and regulation.
Of course this is being packaged as “safety concerns” because they have to keep the “scary AI” facade up or their entire house of cards will collapse.
If “misaligned” AI itself poses any meaningful risks to humanity, we cannot afford to entrust these companies to manage it. They have shown us very few reasons to believe they would be worthy of the task, and a lot more evidence they wouldn’t.
But when they talk about “alignment”, they really mean alignment to their own corporate interests, not to some broader, idealized human values. They are concerned they would not be the ones deciding what everyone else can do with AI, and I would argue that there are way bigger societal risks in letting them institutionalize this level of control, than from AI going rogue.
If AI has to benefit the whole of humanity, it has to be truly open. It cannot be controlled by just one or a handful of entities.
If we don’t want AI to be the biggest driver of wealth and power concentration in human history, we cannot allow these corporations to vacuum up all the content ever produced by humanity and then keep the weights for themselves.
Obviously, even if frontier labs released their weights tomorrow, we’d still face a concentration in the compute power needed for inference. But this is not a short-term game, and in the long run, never bet against technological advancements making more compute power available to the average person. We won’t be able to run Fable 5.1 at home today, but we will be able to run models as capable as Fable 5.1 on devices we own in a not so distant future.
So yes, if we want to regulate AI (and I believe we should, just for different reasons), let’s make AI truly open. Let’s start with requiring to make every publicly available model open weights. It would be a good start.
So when people voice fears of artificial intelligence, very often, they invoke images of humanoid robots run amok. You know? Terminator? You know, that might be something to consider, but that’s a distant threat. Or, we fret about digital surveillance with metaphors from the past. “1984,” George Orwell’s “1984,” it’s hitting the bestseller lists again. It’s a great book, but it’s not the correct dystopia for the 21st century. What we need to fear most is not what artificial intelligence will do to us on its own, but how the people in power will use artificial intelligence to control us and to manipulate us in novel, sometimes hidden, subtle and unexpected ways. Much of the technology that threatens our freedom and our dignity in the near-term future is being developed by companies in the business of capturing and selling our data and our attention to advertisers and others: Facebook, Google, Amazon, Alibaba, Tencent.
Essential talk by sociologist Zeynep Tüfekçi on the (very current) risks of machine learning applications in social media and advertising networks.
The most common misconception about artificial intelligence begins with the common misconception about natural intelligence. This misconception is that intelligence is a single dimension. (…) Intelligence is not a single dimension. It is a complex of many types and modes of cognition, each one a continuum.
The main problem with AI still remains finding a suitable definition of the concept of “intelligence”.
