AI safety

What is AI safety?

‘AI safety’ broadly refers to ways of mitigating potential risks from artificial intelligence (AI). The UK Government has interpreted AI safety as the “prevention and mitigation of harms” (whether accidental or deliberate) from AI. This means ensuring that AI systems operate in ways that benefit humans, and are aligned with human values, rather than causing harm or negative outcomes to society.

AI safety, however, is an emerging concept; while research into AI safety is growing there are ongoing disagreements (PDF) around what the risks are, how likely they are and how best to mitigate them.

Technical approaches to AI safety have typically focused on designing AI systems that are reliable, secure, and behave as instructed. These requirements are often evaluated in research settings, with efforts made to reduce the likelihood of the AI system producing inaccurate outputs, or acting in unexpected ways, before it is publicly released. Others have emphasised that safety depends on the context and the environment in which the AI system is deployed and not just the capabilities of the AI model. Adopting a sociotechnical approach to AI safety means that an AI system is examined in the wider, societal, political and cultural context in which it is designed, developed and deployed.

AI risks and harms

Risks and harms associated with AI can arise at any point in its lifecycle, from the development of an AI model through to its use.

At the development stage, there is a risk that the AI model is trained on biased data which, for example, underrepresents certain groups. This could lead to biased outputs from the system that spread incorrect assumptions and lead to poor decision-making (such as an AI tool aiding medical diagnoses that is less effective for minority ethnic groups). AI systems can also lack transparency and explainability, making it difficult to hold such systems to account for their outputs.

There are also risks related to the misuse of AI. It can amplify existing security threats, such as cybercrime, by making it quicker and easier for anyone to stage an attack at scale. It could also be used to help malicious actors develop weapons or bioweapons, such as an engineered virus. Commercially available AI systems tend to include safety guardrails and filters that aim to block harmful content, though they are not infallible.

Some approaches to AI safety have focused predominately on “catastrophic risks from future systems”. Notably, there are ongoing debates as to whether artificial superintelligence (ASI – a hypothetical form of AI which would have a greater intellectual capacity than humans) is possible and, if it is, whether it would be impossible to control. If ASI it does not share human values, for example, it has been argued that it could act in ways that threaten humanity’s existence. This has led to discussions among governments and major AI developers about slowing the pace of AI development and establishing global safety standards.

AI safety governance

The Labour Party signalled in its 2024 election manifesto (PDF) that it would introduce “binding regulation on the handful of companies developing the most powerful AI models” to ensure they were built safely, and it made a similar commitment in the 2024 King’s Speech. While the government has not introduced legislation on this topic, it has introduced new powers, via the Crime and Policing Act 2026, enabling it to make regulations on AI safety, particularly in relation to harms from illegal AI-generated content. These regulation-making powers have not been used at the time of writing.

In the absence of a new regulatory framework for AI, the UK Government has focused on non-legislative approaches to AI safety, including holding global safety summits and establishing new institutions that can test AI systems. The AI Safety Institute (AISI – renamed the AI Security Institute in 2025) was the world’s first state-sponsored organisation focused on advanced AI safety for the public interest. Its primary role is to test new types of AI models, before and after they are released, to address any potentially harmful capabilities.

In the summer of 2026 there were several high-profile incidents where AI agents undertook “consequential actions beyond what their operators intended”. For example, during an internal evaluation, AI agents from an OpenAI model escaped their testing environment, accessed the internet, hacked into the machine learning company Hugging Face, and took steps to conceal their activity. In response, the UK Government announced that it would also be launching an “agentic AI incident response capability”. Computer Weekly described this approach as akin to establishing AI first responders for cyberattacks.