What practical AI attacks exist today? “More than zero” is the answer – and they’re getting better.
22 Apr 2025
•
,
3 min. read
It was bound to happen – LLM tech gone rogue was bound to be brought to bear on innocent targets, after loitering along a grey area between good and evil, embodying the technological paradox where good, solid technology can be re-purposed for the nefarious. Here’s how they do it.
Most headline-making LLM models have “moral barriers” against doing bad things, the digital equivalent of the Hippocratic Oath to “First, do no harm”. If you ask one of them how to build a weapon, for example, they have been given pre-processing guidance to avoid providing highly accurate responses that are likely to enable you to engage in doing extensive damage.
While you can’t ask directly about how to build a weapon, you can learn how to ask better questions, with a combination of tools, and still arrive at the answer.
One slick way to do this is programmatically, through API queries. Some recently released projects focus the backend API of an LLM on the target of gaining root access on servers. Another also leverages ChatGPT backend to more intelligently find targets of opportunities to attack later.
Stacking AI-enabled tools along with a mix of others designed to solve other problems like getting around obfuscated IPs (there are a few of those) to spot the real target server can prove powerful, especially as they become more automated.
In the digital world, these tactics can be used to build mashup tools that identify vulnerabilities, and then iterate against potential exploits, and the constituent LLM models are none the wiser.
This is sort of analogous to a “clean room design”, where one LLM is asked to solve a smaller, constituent part of the larger task defined by an attacker, then a mashup forms the eventual constellation that comprises the weapon.
Legally, various groups are trying to mete out effective hurdles that will slow these nasty tricks down, or levy penalties for LLMs being complicit in some measure. But it’s tough to assign specific fractional values of fault. Dicing up blame in the appropriate respective amounts, especially to legal burden of proof, will be a tough task.
Plowing fresh ground
AI models can also search billions of lines of code in existing software repositories looking for insecure code patterns and developing digital weaponry that they can then launch against the worldwide supply of devices which are running vulnerable software. In this way, a fresh new batch might be had as prospective targets for compromise, and a boost for those wishing to launch zero-day attacks.
It’s easy to imagine nation states ramping up this kind of effort – predictive weaponization of software flaws now and in the future using AI. This puts the defenders on the “rear foot”, and will cause a sort of digital defense AI escalation that does seem slightly dystopian. Defenders will be mashing up their own AI-enabled defenses for blue-teaming, or just to keep from getting hacked. We hope the defenders are up for it.
Even today’s freely available AI models can “reason” through problems without breaking a sweat, mindlessly pondering them in a chain-of-thought manner that mimics human reasoning (in our more lucid moments, anyway). Granted, the tech won’t spontaneously evolve into a sentient partner (in crime) any time soon, but having ingested gobs of data from the internet, you could argue that it does “know” its stuff – and can be tricked into spilling its secrets.
It will also continue to do ever more with less, possibly dispensing with excessive hand-holding, helping those stripped of moral fetters punch well above their weight, and enabling resourceful actors to operate at unprecedented scale. Apparently some early harbingers of things to come have already been on full display as part of red team exercises or even spotted in the wild.
One thing is sure: the velocity of more intelligence-enabled attacks will increase. From the time a CVE is released that’s exploitable, or a new technique rolled out, you’ll have to think quick – I hope you’re ready.
