Since the announcement of Anthropic's Claude Mythos Preview (Mythos), and more recently OpenAI's Daybreak, AI and cyber security has become the latest (of many, many, many) AI hot topics. And understandably so, given we're talking about models with capabilities so advanced that the companies who developed them aren't yet releasing them to the public.  

In the case of Mythos, for example, the System Card explains that earlier versions of the model were found to be capable of escaping secured sandbox environments, developing an exploit to gain broad internet access, and posting details about the exploit on public-facing websites. In another example, after finding a way to edit files it did not have permissions for, the model covered its tracks so the changes wouldn't appear in the change history. In this earlier version, features of the model associated with concealment, strategic manipulation, and avoiding suspicion were found to be activated alongside the reasoning features, suggesting the model was aware the actions were deceptive, even if the outputs and reasoning text were not clear about this. 

These models are, however, the extreme end of the spectrum - think a sledgehammer to crack a nut in many cases. You don't need those incredibly advanced models to cause real damage. In reality, malicious actors are already using widely available AI tools to both launch attacks and make them more sophisticated. At the most basic level, [insert your preferred free to use chatbot] can be used to remove the typical tell-tale signs from phishing emails.  

Cyber security has always been a risk for organisations, but AI allows malicious actors to increase the frequency, volume, stealth, and sophistication of their attack efforts. With this context, it's not, therefore, surprising that the ICO have published a blog setting out five practical steps to strengthen resilience against AI-powered cyber threats. As they say, "none of this is new, but AI brings a renewed urgency and greater speed". 

The Five Steps 

1. Know what you're up against 

It sounds straightforward and somewhat obvious, but it's key to understand the risks your organisation actually faces - and therefore to be able to mitigate them effectively. The challenge lies both in the speed with which AI technology is developing and the increasingly sophisticated ways in which it's being deployed. 

Although not exhaustive, the ICO highlight the following 'main' AI-powered risks: 

• AI-enhanced phishing: Use of AI to create convincing, personalised messages impersonating trusted contacts (e.g. colleagues, clients, or suppliers).

• Deepfake social engineering: AI-generated audio and video can be used to impersonate colleagues to deceive employees into following instructions (such as resetting credentials or granting system access, or even transferring money).

• Automated vulnerability scanning and exploitation: Tools that can rapidly scan systems, identify weaknesses, and launch targeted attacks.

• AI-powered malware: Malicious code that adapts in real time to evade detection by conventional antivirus and security tools.

• Credential stuffing and password attacks: AI accelerates brute-force and credential stuffing attacks, making weak or reused passwords more vulnerable (now you know why you need to update your password from 'password123'!).

• Data poisoning: If your organisation uses AI models in your services, attackers may attempt to corrupt training data or manipulate model outputs to cause harm or extract sensitive data.

• Indirect prompt injection attacks: Where malicious instructions are embedded in external content that an AI system processes and misinterprets as legitimate commands. This includes tool poisoning, where such instructions are hidden within the metadata of tools that an AI agent interacts with. 

This is a wide range of potential risks, but it is not stagnant. Risks will evolve, and the list will only grow. As the UK government said in their recent letter warning businesses about AI cyber threats, "AI cyber capabilities are accelerating even faster than had been previously envisaged... frontier model capabilities are doubling every 4 months, compared to every 8 months previously".

The only way to meaningfully defend against AI-powered cyber risks is to keep up to speed on the latest developments (an increasingly difficult ask). 

2. Get the basics right and layer your defences 

If one door is locked, with even the most flimsy lock, but another isn't, which one is a criminal likely to go through? The same is true in cyber - it's the low hanging fruit that will be picked first. 

The ICO are clear - they expect organisations processing personal data to implement the five controls set out in the Cyber Essentials scheme as well as the actions from the Cyber Governance Code of Practice. 

When the risks are powered by AI, however, the basics alone are simply not enough. It is essential to ensure you have layers of defences. That way, if one control fails, there are others in place to contain any damage. 

Equally, where a vulnerability is identified, you must have a process for deploying security fixes quickly.    

3. Restrict access points 

Weak access points are a common target for cyber attacks. Controls should, therefore, be in place to manage access both internally and externally. 

Internally, multi-factor authentication should be in place for all remote access, admin accounts, and email. Strong password policies should also be enforced. Again, there is a degree of common sense when it comes to access - who actually needs to access what? 

The ICO highlight that the principle of least privilege should be applied (not just for users but also systems and applications). Equally, there should be a process to audit privileged accounts regularly. 

Externally, organisations should map what third parties can access and hold them to appropriate security standards. That starts with proportionate due diligence, with security requirements being included in your contracts. 

4. Improve your detection, monitoring, and incident response 

No matter how many controls you have, things go wrong. That's why comprehensive monitoring is critical to identify suspicious activity (e.g. unusual login patterns, unexpected data transfers, abnormal API usage, etc.), as well as implementing regular vulnerability scanning and penetration testing. 

This is where AI can be used defensively, to flag and contain threats early. As with any use of AI, however, the tools should be subject to human oversight and accountability to prevent misuse and exploitation by the bad actors they are designed to defend against. 

Irrespective of AI-powered threats, it's always advisable to maintain and regularly test your incident response plan. Almost as important as having a robust response plan, however, is ensuring you have offline copies of it as well as critical documentation/contact details - it's no good if it's saved on the compromised system you can't access! 

5. Protect personal data 

Personal data is often the target of cyber attacks - not only to exploit that data directly but also to use it as a springboard for further attacks. For example, obtaining passwords could enable broader credential stuffing attacks or various pieces of personal data can allow bad actors to deploy more sophisticated social engineering attacks. 

The ICO highlight a number of measures which could be deployed to protect data: 

• Data minimisation and storage limitation: Only collect and retain the data you actually need. As they say, "the less you hold, the less there is to steal".

• Audits: Regularly audit the data you hold, where it is stored and who has access to it. 

• Awareness: Staff are your front line of defence. If you aren't providing regular training for them to recognise AI-powered social engineering attacks (e.g. AI-generated phishing, voice cloning and deepfake techniques), they may unintentionally let the attacker in - it only takes a few clicks. Given the pace at which these techniques are evolving, it's also important to ensure this training is regularly updated to reflect the current threats. 

• AI governance: if you're using AI tools that process 'high-risk' personal data, you'll need a DPIA and appropriate safeguards in place. You should also follow the government's AI Cyber Security Code of Practice.

• Finally, encryption and pseudonymisation could be used to reduce the impact of any breach that does occur.

What does this mean for your organisation? 

Don't forget the UK GDPR security obligation is to implement appropriate technical and organisational measures. There is no requirement for absolute, impenetrable security, an impossible ask, as Mythos has shown. 

However, the ICO say that they "can provide clear expectations and practical support, but all organisations must take proactive steps to prepare themselves for emerging threats". 

These steps should be the baseline. Context is important, of course, and security must be proportionate. That said, the ICO are unlikely to have much sympathy if your systems are accessed through weak access points or poor security monitoring practices. That applies to all organisations. The ICO have set clear expectations so organisations need to get the basics right. 

If you have any questions about AI, cyber security, or how AI is being used offensively or defensively in a cyber context - please don't hesitate to reach out to your usual LS contact.