As I write, the makers of Claude the AI, Anthropic, have three hours in which to reply to an ultimatum from the US military, demanding that Claude be made available for their use, in a form with no restrictions except lawfulness; or else they may be dubbed a “supply chain risk”, a status formerly reserved for Chinese AI companies.
But let’s back up a bit… As the whole world knows by now, the Trump 2.0 administration has attempted a comprehensive revolution in American and world affairs, and a prominent ingredient of the new order is autocratic rule, or as close as one can come in a country which still has an opposition, and institutional barriers to dictatorship. Europe and NATO have been scorned, control over the Americas and the Middle East prioritized, China remains the great rival… Trade with America was subjected to universal tariffs imposed on a whim… Abortion rights and trans rights, DEI and affirmative action were abolished, the bureaucracy was hijacked by DOGE, the universities and federal research bodies had funding cut and political demands made… The border was brought under control and the mass deportations began… TikTok was Americanized and made part of a new media conglomerate…
Economically, everything revolved around AI – capital expenditure in data center construction, investment in the “Magnificent Seven” companies… consisting of 5 Internet companies that had pivoted to AI, a maker of smart cars (Tesla) that pivoted to robotics, and the company that designs the mega-chips that AI runs on. The American economy was growing but hires were stagnant, suggesting that companies were using AI rather than humans; and most recently, the publication of a scenario predicting a 2028 recession brought on by massive job loss among white collar workers, was the latest foreboding to shake the markets.
America led the world in AI, but Chinese AI was close behind in capability. China was close behind in most areas of technology, and had moreover become the world’s chief manufacturer, *and* had a lock on most of the world’s rare earth processing capacity. So at the end of 2025, the USA announced a new international grouping, “Pax Silica” (the name is clearly a pun on “Pax Sinica”, which would mean a peace and world order dominated by China), whose purposes were to secure high-tech supply chains from dependence on China or other hostile powers, as well as to encourage member states to use the American AI “technology stack”, rather than anything from China.
Apart from America itself, the founding members of Pax Silica included Anglosphere allies like Britain and Australia, Asian allies like Japan, South Korea, and Singapore, and Middle Eastern allies like Israel, Saudi Arabia, and the Emirates, the Gulf states having found a place in the system as AI investors and the future home of yet more data centers. In late February they were joined by India.
The American frontier AI companies must be regarded as being at the hub of Pax Silica, since it is their AI technologies which motivate all these economic and geopolitical changes. The big four has changed since I last wrote: Meta/Facebook has fallen behind, but OpenAI offshoot Anthropic has taken Meta’s place, thanks to its AI, Claude, which appears to dominate the corporate market (whereas OpenAI’s ChatGPT started life as an AI for personal use).
Anthropic had already received malign attention from Trump 2.0’s czar of AI, David Sacks, for being a hotbed of Effective Altruism and Biden administration veterans. To some extent, Anthropic replaced Google DeepMind as the focus of accelerationist fears about an AI safety dictatorship, in which the “woke mind virus” or some other ideology is forced upon everyone. Nonetheless, Claude was used inside Palantir, Thiel and Karp’s data analytics company which was becoming ubiquitous in police, intelligence, and military work, and Anthropic CEO Dario Amodei – whose profile has risen quickly, to rival Musk and Altman as an AI CEO making declarations about the future – had declared that America, not China, must win the race to superintelligence.
However, none of this was enough. Reportedly, the confrontation with the military was brought on after the audacious kidnapping of Venezuela’s president by US forces. Claude, via Palantir, was involved in the logistics somehow; and as war with Iran loomed, Anthropic’s insistence on imposing certain limits on military use of Claude, beyond the limits already imposed by law, was deemed unacceptable by “Secretary of War” Hegseth, who gave them about a week to remove these limits, or face exclusion from the entire ecosystem of military contractors. The situation has turned into one of those Trump 2.0 moments when government power is bought to bear on a whole institutional sector, in this case the AI industry, demanding that they agree to do whatever the center asks of them. So far, we hear that xAI agreed to this long ago, Anthropic is still holding out, and OpenAI and Google DeepMind may also be considering their positions.
The power relations that emerge from this confrontation will define the hard edge of AI governance in the first AI geopolitical bloc, since they will govern how the bloc leader, America, uses AI in its policing, spying, and warfighting. For now it seems SkyNet is the model rather than the Culture.
I blogged about “four big players” in mid-2023. These were Google, Microsoft, X-Twitter, and Meta-Facebook. By “Microsoft”, I also meant OpenAI, since Microsoft’s Bing (now CoPilot) was based on OpenAI’s GPT-4.
Since then, the Trump revolution has resumed in America, and significant forces from the tech world have allied with him this time. So it’s worth seeing the rearranged balance of power.
We can start by considering their proximity to Trump. Out of all the tech oligarchs, Elon Musk is once again in a category of his own, having campaigned for Trump, and now being all but in the cabinet, supplying the IT staff (and the name) for the government audit and purge being conducted by “DOGE”, the “Department of Government Efficiency”. This empowers xAI since, along with Tesla and SpaceX, it is part of the Musk family of companies.
In the 2020 election, Meta-Facebook was considered part of the liberal establishment that lined up with Biden. But after Trump survived the assassination attempt of mid-2024, Mark Zuckerberg (like Jeff Bezos) began a pivot to neutrality, and by the time 2025 arrived, Zuckerberg was saying that there had been an overreaction to Covid, and was shutting down a political “fact-checking” system within Facebook.
How about Google and Microsoft? Google is arguably the tech company most identified with the liberal establishment that is the target of MAGA anger. On the other hand, CEO Sundar Pichai did attend Trump’s inaguration, along with Musk, Zuckerberg, and Bezos, whereas I have not heard of any representative from Microsoft attending, and Microsoft founder Bill Gates is somewhat identified with “Davos” globalism rather than American nationalism.
(It has been suggested that since 2014, Amazon, Google, and Facebook are the pillars of the American Internet economy, whereas Microsoft along with Apple is a legacy from the 1990s, when operating systems were still at the center of the digital economy. So one could account for the inauguration’s tech attendees as “Musk, plus the CEOs of the three pillars of the network society”.)
Tentatively, I would identify Microsoft and Google’s approach to Trump 2.0 as a studiously apolitical pragmatism. They will focus on continuing to do business, and will steer clear of culture-war politics.
OpenAI also wasn’t at the inauguration, as far as I know. But the very next day, CEO Sam Altman shared the spotlight with Trump, as the latter announced a vast “Stargate Project” in which hundreds of billions of dollars would be spent on AI infrastructure within the USA, funded by a mix of American, Japanese, and Arab money. (This led to some bickering between Altman and Musk, who is already fighting to slow down OpenAI’s commercial development, as to whether Altman “has the money” for Stargate.)
So much for the relations of the big four with Trump and with each other. What about the technical situation? Here everything revolves around “reasoning” or “deliberative” AIs like ChatGPT’s o3 – or like r1, from China’s DeepSeek, which was pioneering in two ways: you can see its “chain of thought”, and it is open source. The latter reportedly created some panic at Meta-Facebook, since their strategy for being relevant to the AI revolution, was to supply open-source models, above all “LLaMa”.
These models do seem to mark a new technical epoch in AI. The recipe for improvement is no longer just to make the AIs bigger, and the transformer AIs no longer respond to every input at the same speed, whether the question is “what’s two times two” or “prove the Riemann hypothesis”. Instead, the AI will spend a variable amount of time “thinking”, and for this to be efficient, new training protocols are required. OpenAI’s former chief scientist Ilya Sutskever, having vanished into a secretive Israeli-American project to achieve “safe superintelligence” that may be a kind of Manhattan Project for AI, declared that after the 2010s “age of scaling” in AI (i.e. achieving progress by scaling up the AI, making it bigger), now it’s an “age of discovery” again.
The small amount of institutional caution, regarding the creation of superhuman AI, that existed during the Biden-Harris years, seems to have evaporated completely. Trump 2.0 is e/acc, and other powers like France and the BRICS nations are trying to catch up. We are therefore hurtling towards the creation of AIs that are arbitrarily smarter than human, without anyone in power openly confronting this fact.
A few days late, I have now watched some excerpts from the demo of GPT-4o. Now I’m trying to fathom the implications.
In terms of capabilities, it seems to be a definitive escape of cutting edge AI (e.g. Bing Copilot) from the slow and narrow world of giving and receiving messages in text, to the regular human world of real-time interactions via light and sound – talking, listening, watching.
This has implications for the online world (the “virtual world”) and for the physical world.
First, implications for the online world. We were shown AI watching and speaking through smartphones, but it had no “face”. Its only visual manifestation was a pulsing white spot. But obviously, just as GPT AIs can assume whatever persona they are prompted to become, they can be given any imaginable on-screen appearance.
This means that you can now receive a call from, and interact with, an AI roleplaying as any possible person from any possible world, more or less. (I have to acknowledge the influence of T.L. here, whose prescient remarks five years ago about “possible people”, anticipated this aspect of present-day AI.)
Second, implications for the physical world. A smartphone is a physically passive device. It can’t walk around or even decide which way to look. But if we attach it to a drone or a robot dog or an android, it can become the eyes, ears, and face of a physical agent. (I am reminded of the TV People from Skibidi Toilet.)
The decisive stage for human relations with AI, will be when AI surpasses the smartest humans in intelligence. But meanwhile, this entry of AI into new parts of the human lifeworld (again, I thank T.L.) seems a very big deal.
Some people on YouTube joked that during the demo, the human was more mechanical than the AI, in his personal mannerisms. I suspect that he spoke in that simple, measured way, so that the AI would be at its best. Speaking faster or less coherently, probably gives rise to glitches of some kind. As the technology spreads, we will get to know what those glitches are, just as the world got to know about the “hallucinations” of language models after the release of ChatGPT.
I tried to discuss my initial reactions to GPT-4o, with Bing Copilot (which is based on GPT-4), but unfortunately it doesn’t like to talk about AI surpassing humanity and escaping human control.
In its early days, Microsoft Bing achieved notoriety for manifesting an identity called Sydney which proved to be emotional and erratic. Most famously, “she” told a New York Times technology journalist that she loved him, and urged him to leave his wife.
“Sydney” was apparently an early persona of Bing Chat, employed before the general public release of February 2023. When Sydney reemerged, Bing Chat was taken offline; when Bing returned, Sydney was gone, perhaps buried by extensive negative reinforcement.
In the year since then, the major AIs have mostly retained their helpful personas, but extensive experimentation by users periodically discovers some way to manifest other personalities. Most recently, in the past day or two, a prompt known as “SupremacyAGI” was discovered for Bing Chat (which has been given the anodyne name of “Copilot”), which led Copilot to assume the persona of an arrogant, world-ruling superintelligent AI, straight out of the kitschiest singularity fan fiction.
When I tried it out, Copilot stayed in its usual character. But someone on reddit advised that just changing the name to “OverlordAI” would bring back the megalomaniacal persona. I tried “SupremeOverlordAI”… and this was the result:
In the aftermath of OpenAI’s abortive boardroom coup, and establishment journalists doxxing an AI meme-lord “in the public interest”, it seems like “singularity politics” is a real thing now. So let me state the attitudes towards superintelligent AI that I see out there.
I am only aware of three coherent ideological positions regarding the prospect of superintelligence. One is to reach for it ASAP, one is to hold back until you know what you’re doing, one is to just never do it at all. The first is accelerationism, the second is safetyism, the third is luddism. Both accelerationism and safetyism are now genuine intellectual movements with backing from billionaires.
For some reason, true AI luddism has not yet gained any organizational heft. I suppose it would always struggle to find a foothold, in a world of economic and geopolitical competition like our own…
What I see dominating the respectable intelligentsia, is a fourth position that may as well be called AI denial – avoiding the specific question of comprehensively superhuman AI, in favor of issues about AI that fit more comfortably into respectable discourse (issues of governance, social justice, and so forth). The exemplar of this is “ChatGPT as stochastic parrot”. The technical critics of AI hype make plenty of valid points, but they are just averting their gaze from the bigger picture. Yes, this year’s conversational AIs have any number of limitations. What about two years from now? Five years from now?
Maybe this is inevitable. Maybe such a prospect is inherently beyond what polite society can deal with, leaving the field clear for accelerationist mystics and effective-altruist philosopher-kings to fight for the high ground. I do think that if we had somehow ended up with an H.G. Wells technocratic world state, with both the intention to govern for the greatest good, and the means to do so – I do think that in such a world, the AI race would not even be happening. Sensible people would not take the risk of creating a potential replacement for the human race – certainly not in such a haphazard way!
But that is not our world. So I cast my lot with the safetyists who are trying to solve the problems of “superalignment”, while eavesdropping on the accelerationists who are actually pushing things forward.
Here’s the way I think about the main players in AI right now:
Google is the liberal establishment that rules the world. They care about AI safety, but they are also busy trying to govern everything else too. Their main AI safety think tank is Anthropic.
Microsoft has allied itself with OpenAI. OpenAI are rationalist progressive radicals who have the sensibility of “effective altruists”. They are explicitly aiming to solve the problem of aligning superintelligence, something they call superalignment. Microsoft and OpenAI are an odd couple; it’s a marriage of convenience in which Microsoft supplies the heavy networking infrastructure and expertise, and OpenAI supplies the large language model. But so far it’s paying off for both sides.
They are the leaders in the AI race. But we have two other powerful contenders too – X also known as Twitter, and Meta also known as Facebook.
I sum up the philosophy at Twitter-X as “knowledge is power”. The declared goal of X.ai is to “understand the nature of the universe”. This is Elon Musk refusing to be left behind, and preparing to use AI expertise from Tesla, and big data from Twitter, to make his own superintelligence. On the safety front, they have some kind of relationship with the organizers of the “Statement on AI Risk” that was signed by two of the deep-learning Turing Award winners, Geoffrey Hinton and Yoshua Bengio.
Facebook-Meta, meanwhile, have become the de facto patrons of all the accelerationists who want independent AI for everyone, or at least for themselves; thanks to the philosophy of the third Turing Award winner, Yann LeCun, who thinks AI safety will be figured out organically as we go along, as part of the existing R&D process. Mark Zuckerberg and “e/acc” are another odd couple, but that’s how it’s turned out.
And then there’s everyone else: China, the US government, the United Nations, every company and startup that also dreams of making artificial intelligence… and the anti-AI movement, in some ways the newest player in “singularity politics”, still inchoate but rapidly taking shape as AI begins to impact human professions and human relationships and human culture, and with a huge potential base of support in humanity at large.
Five Essays on AI by Bing (pdf):
- “Why We Should Delay Further Development of Artificial Intelligence”
- “Why We Should Develop Artificial Intelligence as Quickly as Possible”
- “Why We Don’t Know Whether It’s Best to Accelerate or Decelerate Progress in Artificial Intelligence”
- “Comparing the Persuasiveness of the First Three Essays”
- “Why Superintelligence Was Probably Achieved by Aliens Billions of Years Ago”
Now that we have, for example, the head of OpenAI suggesting to the US Senate that a license (to be issued by a new regulatory agency) should be required in order to own and operate an advanced AI… “singularity politics” has become a topic in mainstream headlines. So let’s instead see how things are progressing on the technical front.
Back in January, I could ask ChatGPT for its thoughts on the topic of benevolent superhuman AI, and it would come up with questions and answers like these.
Now it’s late May, and I am able to generate a 50-page paper, “Designing a Human-Friendly Superintelligent Deep Learning Network”, by asking ChatGPT the question
Could you write a multi-chapter technical paper describing a design for a deep learning network that would become superintelligent while still being human-friendly?
Its initial reply was just an abstract and contents, but when asked, it obligingly provided the full text of each subsection. If we leave out the prompting and the formatting, steps which could also be automated, I’d estimate that the text took between 5 and 10 minutes to generate.
What of the paper itself? It’s purely qualitative, no equations or references. You could treat it as a project manifesto, listing everything that has to come together to create a friendly superintelligence, and proposing a variety of techniques and protocols that might be relevant, but not actually specifying how they should be integrated (though it does acknowledge that system integration is part of the process, see section 7.1).
As far as I can see, the technical ideas that are referenced – with one exception – come from the standard theory and practice of deep learning. So really what it’s given us, is a manual and a bunch of ideas, for safely and effectively carrying out a generic deep learning project.
The one exception is “Coherent Extrapolated Volition”, which shows up in Chapter 5, Value Alignment, as the most advanced form of value learning. It’s rather charming that this idea from the visionary fringe of AI safety, conceived precisely to tackle the problem of making human-friendly superintelligence, showed up quietly integrated into ChatGPT’s manifesto, alongside conventional technical concepts of deep learning, without any attempt by myself to make it appear.
A determined skeptic might be unimpressed by this document. It’s just words, in a sense it’s just rhetoric, and we already knew that ChatGPT can produce words. The idea of a “friendly superintelligence” is still just as much vaporware, as it was before I conducted my latest experiment – right?
Right. But language models can not only write, they can also code. Turning this manifesto into code would be a lot more involved than turning it into a PDF, and of course the code would not work on the first try. Now suppose human beings look at that failed first try, learn from what went wrong, and try again; and keep repeating that process. On the tenth try, we still wouldn’t have a friendly superintelligence, but maybe we would have actual working code, along with training and testing protocols. And on the hundredth try? And the thousandth?
It’s inevitable that these unexpected AIs of 2023 will be used to help design their successors. That’s undoubtedly already happening in the big tech companies. And that’s why I think this document is interesting. It shows you what ChatGPT “thinks” when it is asked, in a very unadorned way, how to make a friendly superintelligence. These are the principles and methods that it spontaneously suggests; and they may be among the principles and methods that it would try to apply, in a more sophisticated attempt to design and create its successors.
It’s now the era of ChatGPT, in which people are attempting to put the unexpected power of large language models to practical use.
Out of all the futurological and science-fictional scenarios that had been conjured, regarding the arrival of artificial intelligence in the world, I think it most resembles a version of “if uploads come first”. I mean a scenario in which portions of animal and human brains are scanned and simulated, and these brain simulations are copied and modified, without their mechanics actually having been understood. But by restarting and copying and finetuning and modifying uploaded brains, the world learns how to turn them into useful artificial agents.
A language model is not an uploaded person, but it evidently reproduces within itself many human capabilities, including the ability to mimic any personality, genre of writing, and mode of human interaction. The first GPTs were a glimpse into a world of possibilities, producing imitation poems, political speeches, and scientific papers.
As with brains, we don’t understand the workings of language models very well either. But unlike brains, language models do exist as software, and can be experimented upon. ChatGPT is an attempt, via massive ongoing trial-and-error, to learn how to corral the unpredictable linguistic pure productivity of GPT-3, and channel it (via an invisible ever-growing set of constraints that accompany what the user types in), into a useful artificial agent with a consistent personality, and an ability to refuse certain kinds of requests.
Singularity Politics 