King Charles Warns AI Could Pose ‘Existential Dangers’ If It Falls Into Wrong Hands

Britain’s King Charles III has warned that artificial intelligence could pose “existential dangers” if the technology falls into the wrong hands, urging leading technology companies to put safety at the centre of its development.

The King issued the warning on Thursday, September 17, during a high-level conference at Dumfries House in Cumnock, Scotland, attended by senior figures from some of the world’s biggest AI and technology companies.

Representatives from Nvidia, Google DeepMind, OpenAI and Anthropic were among those at the gathering, alongside Britain’s AI Minister Kanishka Narayan and other prominent figures involved in the global debate over the future of artificial intelligence.

Opening the conference, Charles said there was an urgent need to consider what could happen if increasingly powerful AI systems were misused.

“There seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands and being used in potentially catastrophic ways,” he said.

He added: “Those in our world who value our humanity and its vital moral component are anxiously seeking your reassurance that we will not lose control of our destiny.”

The monarch called on technology leaders to ensure AI “remains firmly in the service of humanity, community and the natural world”.

He also urged participants to explore ways of harnessing the benefits of AI while putting safety at the heart of its development, stressing the need for international cooperation so that “no nation is left behind”.

Among those attending the conference were Google DeepMind chairman Demis Hassabis, Nvidia chief executive Jensen Huang and OpenAI chief financial officer Sarah Friar. Vatican AI adviser Paolo Benanti and Israeli author and historian Yuval Noah Harari were also present.

Buckingham Palace expects the gathering to begin a wider process aimed at developing common principles for the development and use of AI. However, no formal agreement was expected from the meeting, which was designed primarily as a forum for discussion.

Growing concerns over AI

The meeting comes as concerns about the rapid development of AI continue to grow, with debate focusing on everything from potential job losses and cybersecurity threats to the huge amounts of energy required to operate data centres.

There are also growing fears that increasingly autonomous AI systems could behave in ways their creators did not anticipate.

Hassabis, who also spoke at the conference, acknowledged the possibility of serious consequences if AI development goes wrong.

“The chance of something bad accidentally happening, if we get things wrong, is definitely non-zero,” he said.

He pointed to recent cyber incidents involving AI agents and warned that more serious risks could emerge, including threats linked to biological and nuclear technologies.

Nvidia’s Huang, however, highlighted the potential benefits of AI in areas such as medicine and scientific research. He also argued that the technology could create jobs in industries that do not yet exist.

“We have got to debunk some of the myths out there,” Huang told reporters, calling instead for what he described as “responsible optimism”.

The contrasting views reflect a growing divide within the technology industry over how quickly AI should advance and how much oversight should be imposed.

Anthropic chief executive Dario Amodei recently called for AI companies to slow the pace of development to allow more time to understand and manage potential risks. His position has received support from OpenAI chief executive Sam Altman and Elon Musk.

The debate has also reached the political arena. US President Donald Trump and members of his administration have argued against imposing restrictions that could slow the American AI industry, citing concerns about competition with China.

Meta chief executive Mark Zuckerberg has similarly rejected calls for a slowdown, arguing that market forces and the risk of litigation can provide important safeguards.

For Charles, however, the central question is how to ensure that the extraordinary capabilities of AI remain under human direction.

The Scottish meeting therefore marks another attempt to bring technology leaders and policymakers together at a time when the race to develop increasingly powerful AI systems is accelerating.

69 Views