Dhurjati Mukherjee

For the past few years, analysts have expressed apprehension about the advent of artificial intelligence (AI), particularly because it has been viewed as a threat to employment in labour-intensive countries such as India, where manpower remains crucial. Various arguments have been advanced about the positive impact of the technology, with some suggesting that AI could directly or indirectly compensate for job losses. But in recent times, AI has made headlines for a far more serious reason: growing concerns that it could pose an existential threat to humanity and human life.

The issue of AI and international security was discussed at the United Nations Security Council last week. At a meeting convened by France, industry leaders called for a coordinated slowdown in the development of AI systems. Representatives from OpenAI, Anthropic and Chinese companies such as DeepSeek participated in the discussions. Dario Amodei, head of AI company Anthropic, told the 15-member council that, if not properly managed, AI could become a risk to humanity as a whole. OpenAI CEO Sam Altman echoed similar concerns.

The discussion came at an appropriate time, amid growing calls around the world to slow the development of AI capabilities because of the potential risk of “recursive self-improvement” – the ability of AI systems to continuously improve themselves – which could eventually allow them to escape human control and carry out highly destructive actions. Anthropic has released a 154-page report warning about the risks posed by increasingly capable AI systems, while OpenAI and Amodei have also put forward proposals. These proposals underline the need for comprehensive, mandatory safety standards for the continued development of AI, enforced through a network involving independent auditors, government agencies and international bodies.

In this context, the case of Jack Coxon is relevant. Coxon, who had earlier left OpenAI to join Anthropic, has since left Anthropic as well. He has pointed out that these companies are “racing directly towards self-improving superintelligence and gambling with our lives”. He further said that people building AI genuinely believe that the technology could potentially destroy humanity by the end of this decade. Indeed, some experts have estimated that the risk of AI causing human extinction within the next decade could be more than 10 per cent. According to reports, AI experts from different countries were asked about the risks and reportedly assessed that 18 of 24 categories of AI could carry a more than 10 per cent probability of “catastrophic outcomes”.

These are undoubtedly serious concerns and demand careful introspection. The seriousness of the situation is underscored by the belief among some AI developers that the technology could “cause human extinction within a few decades”. This echoes the warnings issued years ago by a small group of technologists who argued that AI could become catastrophic and that the sensible approach was to proceed cautiously and continuously assess the risks.

Although AI developers use techniques such as human ranking, reinforcement learning and explicit rulebooks such as OpenAI’s “Model Spec” and Anthropic’s “Constitution”, reports suggest that artificial intelligence is “growing beyond its design”, as OpenAI chief scientist Jakub Pachocki has pointed out. He has also noted that developers do not specify every possible rule and that designers build and deploy systems in ways that can produce unexpected outcomes. The concern surrounding superintelligence is that it could potentially eliminate the gap between wanting to achieve something and finding an entirely different way of accomplishing it.

A 2025 review by Cambridge can also be cited in this context. It noted that AI technologies could advance much faster than human understanding and governance. Given the present situation, concerns are growing that control over such systems could be weakened by their increasing autonomy and their capacity for advanced cognitive and strategic planning.

Meanwhile, Dario Amodei has rightly called for slowing the pace of development of increasingly capable AI systems. He has advocated government controls under which the US and other democracies, as well as China, would accept similar limitations. The call came amid reports of warnings from researchers within Anthropic about the possibility of losing control over more advanced models.

There is a strong case for slowing the pace of AI development. The pursuit of ever larger and more capable AI models consumes enormous amounts of capital and resources. Data centres, for instance, require substantial quantities of energy, water and physical infrastructure. These costs also intersect with existential challenges such as global warming, with AI-related infrastructure contributing to greenhouse gas emissions.

A closer examination shows that the warnings now making headlines are not entirely new. AI pioneers Geoffrey Hinton and Yoshua Bengio, whose work helped make modern AI possible, have previously put the risk at roughly 10 to 20 per cent. What is different now is that warnings are increasingly coming from people inside frontier laboratories who are directly involved in designing these systems.

It is time to democratise AI because decisions of such importance cannot be left solely to laboratories in San Francisco, as Altman argued at the Security Council. He said such systems should be shaped “through democratic processes and by governments that are accountable to the people they serve”. The Security Council meeting also highlighted the need for international cooperation, with Altman stressing that “no company, no country can handle this alone”.

Therefore, safety must be paramount in the everyday use of AI in defence, medicine and hospitals, education and law enforcement agencies. In view of the potentially grave risks, the technology needs appropriate regulation. However, as there is broad agreement, merely slowing the pace of development will not resolve the fundamental problem unless companies and governments work together through coordinated efforts.

Meanwhile, safety testing has already begun, with a team working on the issue at the UK AI Safety Institute. However, relatively little is known about the safety techniques being employed and, more importantly, about how artificial intelligence can genuinely benefit humanity.

Most governments should adopt safety standards along the lines of the European Union’s AI Act, which requires the assessment of systemic risks associated with the most powerful AI models and gives regulators powers to ensure appropriate safeguards. But in the long term, this alone may not be sufficient. There is a need for an international panel of experts to analyse emerging risks and develop appropriate frameworks to address them.

As the debate over the advent of AI continues, warnings about extreme technological risks cannot be dismissed. History reminds us that human beings are not necessarily the strongest or most intelligent species. Yet we may have to adopt a cautious “wait and watch” approach while continuing to study the consequences of increasingly powerful AI systems. At the same time, we must learn to accommodate the rise of AI without undermining the fundamental foundations of human existence.

When AI takes over mundane or routine mechanical tasks associated with everyday survival and work, what will we do with our time? This transformation brought about by advanced technology must be aligned with our lives and ways of living. We cannot afford to be indifferent to the social and economic displacement that may lie ahead. Addressing these challenges will require a serious assessment of the entire issue, followed by appropriate measures based on consultations with experts, technology leaders and scientists across the world.