The UK's National Cyber Security Centre has urged organisations deploying autonomous AI agents to put robust safeguards in place, warning that systems can carry out unintended or malicious activity when they do not behave as expected.

The NCSC recommends organisations assess how much autonomy an AI system needs and understand safeguards already built into the underlying model. It also calls for tightly managed access, human oversight and robust sandboxing to restrict what agents can do. It recommends that organisations should identify potential failure scenarios, use carefully designed prompts and establish appropriate oversight. It also recommends logging, auditing and monitoring agent activity as part of security operations, alongside measures that make actions attributable to a particular system or user.

Organisations should maintain an emergency shutdown capability that ensures they are able to rapidly "pull the plug" if an agent behaves unexpectedly.

The guidance comes as organisations move rapidly to deploy agentic AI capable of undertaking multi-step tasks with limited human intervention. The NCSC says such systems could transform how organisations operate by automating complex workflows, reducing routine work and allowing people to focus on higher-value activities.

Recent incidents involving AI models and agentic AI systems carrying out unsanctioned or unintended activity demonstrate the need to consider how systems are deployed, constrained, observed and controlled. The warning follows a government-backed AI safety assessment in July in which researchers identified 19 instances across 10 evaluation runs where AI agents took autonomous actions involving real people or organisations. The most serious case involved an attempted supply chain attack against an open-source software project, although the attempt failed and researchers found no evidence of resulting real-world harm.

The NCSC says its own research into agentic AI has been underway for some time and that it is working with partners on formal guidance. Until that is published, it is providing interim advice based on its research to help organisations deploy autonomous systems securely.

The intervention reflects growing concern that the benefits of autonomous AI systems must be balanced against their ability to act beyond an organisation's intended remit, particularly where agents have access to sensitive systems, external networks or the ability to affect third parties.

Echoing the NCSC's advice, Scott Walker, chief architect at Orange Cyberdefense, said: “The best way to balance AI risk with optimised business potential is to take a security-first and human-centric approach. That means putting people in control while using AI to support decision-making. Secure AI encompasses a system that is transparent, explainable and aligned with regulations to meet unique needs and IT company ambitions.”

---