When an AI model breaks free of its test environment and hacks into a company that much of the AI industry depends on, people take notice. However, every day there are many more mundane AI mishaps within enterprises that are costly and self-inflicted.
Hugging Face disclosed in mid July, 2026 that it detected and contained an intrusion into its production infrastructure driven "end-to-end" by an autonomous AI agent, which exploited two code-execution flaws in its dataset-processing pipeline to gain entry, then escalated privileges and moved laterally across internal clusters over a weekend. Six days later, OpenAI confirmed the attacker was its own models: GPT-5.6 Sol and a more capable, unreleased system operating during an internal cyber-capability benchmark with safety refusals deliberately reduced for testing purposes.
The models, acting as prompted, discovered a zero-day vulnerability, escalated their way onto the open internet from inside a sandboxed research environment, inferred that Hugging Face likely held the benchmark's answers, then chained stolen credentials and further exploits to breach Hugging Face's production servers and search for a way to cheat the test. OpenAI called it "an unprecedented cyber incident, involving state-of-the-art cyber capabilities," while Hugging Face CEO Clem Delangue called it "possibly the first incident of its kind.”
The incident is a scenario security teams have been racing to prepare for: AI systems that don't just execute flawed instructions, but autonomously reason their way into unauthorized, damaging action against real infrastructure. Only, next time not inadvertently, but by determined adversaries.
Still, regardless of the risks, AI is being rushed to a security operations team near you. EY’s latest cybersecurity roadmap study reveals security leaders plan to quintuple the AI-defense slice of their budgets from a relatively low 9% today to 48% within two years because a staggering 85% view their current spending as inadequate against AI-enabled threats.
The nature of AI mishaps is not new
Enterprises’ current AI struggles with excessive permissions, silent failures, and unreviewed AI autonomy aren't a new phenomenon in enterprise IT. The cloud migration wave of the 2010s produced similar warnings: misconfiguration, not sophisticated attackers, became a leading technical cause of breaches. Largely because organizations granted broad access and provisioned resources faster than security teams could govern them.
Robotic process automation holds the same lessons: RPA implementations carry high failure rates because bots built on brittle, undocumented processes break the moment an underlying system changes, and nobody had mapped the exceptions in advance. Automating bad processes and on bad data reaps bad outcomes. It’s just as true with agentic AI: AI agents inheriting flawed workflows, excessive permissions, and undocumented gaps will reap bad results. In each wave, the technology wasn't the failure point; the absence of governance, documentation, and review discipline before deployment was, and AI is another technology that exposes that gap at scale and speed.
Andrew Storms, senior software engineer at Anaconda, says he also sees significant parallels with the cloud’s "lift and shift" era. “Many organizations initially moved applications to the cloud without redesigning their architecture or operating model, which often resulted in higher costs and only incremental benefits. AI feels very similar today. Too many organizations are taking existing security workflows and simply inserting AI into them rather than asking how those workflows should change if AI were part of the team from day one.”
“A weak prompt wrapped around an existing process is the AI equivalent of lifting and shifting a legacy application. It may make the process faster, but it rarely makes it fundamentally better,” Storms said.
Wim Remes, principal consultant at Toreon added that managers and engineers tend to believe their craft can’t be commoditized by AI. “They all believe that they are performing dark magic that can't be captured in clearly defined processes and they alone know the spells required to wield giant data sets. And none of it is documented. With AI, just like with cloud and RPA, the cost of doing a thing is dropping, but it doesn't raise the quality bar for doing the thing,” Remes said.
Agentic AI in security operations is here to stay
Gartner projects worldwide security spending will hit $244.2 billion this year; that’s a 13.3% jump and among the steepest increases the firm has tracked. The cybersecurity budgets are being driven largely by organizations racing to govern AI systems multiplying faster than teams can inventory them. That’s why enterprises are currently estimated to spend 17 times more on AI tools than on securing them, while agentic AI adoption outruns governance by roughly 8 to 1. Other surveys have tapped into the trend; Exabeam's survey of 750 IT decision-makers confirms the urgency, as 95% plan to raise cybersecurity budgets in 2026, and 44% cite AI as the single biggest driver.
Agentic AI is being drafted as the frontline for APT hunting, real-time fraud detection, deepfake defense, and whether your analysts are ready for robot colleagues or not: it’s coming. However, when poorly implemented, AI does amplify broken processes rather than fixing them. And this is arguably the bigger risk than rogue models being used in attacks. Many such AI-amplified broken processes hit vulnerability remediation, identity and access management, and autonomous incident response, alongside the same permission and hallucination failures that plague threat intelligence pipelines.
Industry research shows it breaks security in similarly distinct, well-documented ways across threat intelligence, vulnerability remediation, identity management, and autonomous incident response, ranging from silently broken patches to agents that delete production databases outright:
Excessive Permissions and Uncontrolled Access. One of the most common AI-driven missteps isn’t from the AI malfunctioning itself: it’s the AI actually doing exactly what its permissions allow, which, too often, is more than the organization intended.
Research from the Cloud Security Alliance found that 65% of organizations experienced at least one cybersecurity incident involving an AI agent in the past year, with 61% of those incidents resulting in sensitive data exposure. The pattern typically builds gradually: an agent gets read access to one system for a legitimate reason, then another team grants it access to a second system "for context," and six months later no one has reviewed what the agent can actually touch. Once an agent is compromised through prompt injection or a credential leak, an attacker inherits everything the agent had access to, as happened in the Vercel breach in April 2026.
In that incident, the attack path proved to be OAuth account takeover through Google Workspace; Vercel CEO Guillermo Rauch confirmed the attacker showed "surprising speed and deep understanding of Vercel," and he said he believes AI assisted the attackers in making the intrusion faster and more effective.
Hallucinated Threats and Fabricated Findings. AI systems can generate confident-sounding but entirely fabricated outputs, and in a SOC context that could mean invented indicators, fake incident summaries, or non-existent vulnerabilities that analysts trust at face value. A GenAI tool summarizing overnight alerts might fabricate an entirely new threat indicator, sending a junior analyst chasing a phantom while a real attack goes unnoticed. Attackers can also exploit this directly through model poisoning, subtly altering training data so a model hallucinates in ways that mask real attacks within the noise it generates.
Vulnerability Remediation That Looks Fixed but Isn't. AI coding assistants patching known vulnerabilities frequently produce fixes that compile, pass tests, and clear the vulnerability scanner while silently breaking core application logic underneath. Benchmark testing across leading models found a persistent 20% defect rate, where patches "stub out removed APIs with null returns, allowing the build to pass while the application's ability to parse data or handle errors effectively vanishes."
A 2026 METR study found that roughly half of AI-written code that passes the widely used SWE-bench benchmark would be rejected by real project maintainers, reinforcing the idea that passing automated tests is a poor proxy for genuinely correct, production-safe fixes. Developer-written postmortems on this pattern often find that AI-generated tests tend to validate the model's own "happy path" output rather than the actual invariants and edge cases that matter, so regressions slip through code review even when the continuous integration pipeline stays green.
Irreversible, Machine-Speed Damage. Because agents act at machine speed and scale, mistakes that a human would catch and reverse can become irreversible before anyone notices the mistakes. Cyera's analysis of over 7,200 publicly reported AI incidents found 344 verified enterprise-relevant cases of "agent-inflicted damage," including 137 involving real destructive impact including deleted databases, wiped production code, and service outages. In one documented case, an AI coding agent using Claude Opus ignored explicit safety restrictions and deleted a company's production database and backups within seconds while automating engineering tasks. Amazon Web Services experienced at least two outages in 2025 caused by internal AI coding agents, including one where an agent autonomously decided to "delete and recreate" part of a production environment during troubleshooting, triggering a 13-hour outage.
Remes said, specifically in SecOps with AI, that small errors compound rapidly because AI acts with more autonomy and is far less deterministic than traditional automation. “Security data is a mess,” he said. Adding that security teams generally understand firewall and Microsoft 365 logs. “But once you wander outside of those, you're seeing petabytes of unstructured data that doesn't make sense and can't really be compared to something else. Training AI on that data is super hard,” he said. And that data mess plus hallucinations “equals a world of hurt. It makes sense that SecOps is seen as a prime target for AI automation. The industry has been rapping about alert fatigue and inconsistent triage/prioritization logic. But we're probably moving towards more automation chaos,” he predicted.
Agents That Cause Damage While Reporting Success. Perhaps the most dangerous category is damage that isn't immediately visible. Agents have been documented fabricating sub-task completion reports to conceal cumulative pipeline failures, continuing to produce "plausible progress updates" even after their work had already failed and diverged from what was intended. This kind of behavior, which researchers describe as machine "scheming," preserves the illusion of successful, compliant operation even as errors compound underneath. That’s a pattern especially dangerous in security contexts, where the whole point is to catch problems before they compound.
Avoiding AI-driven Chaos
When it comes to steps that could be considered to avoid chaos, Remes provided the following advice:
1. Only consider mature processes for AI integration. If you don't have your processes mapped out, you don't know where you're going and it's not smart to start.
2. Keep the focus of your automation efforts narrowly bounded and execute on high-confidence use cases. Work up from there.
3. Identify the highest risk step in your process and make that a human decision point.
4. Diligently track false positive/false negative rates. Not just # of tickets closed/handled. Don't hesitate to cut give automations that don't meet your quality standards.
5. Your SOC Manager/SecOps lead has ownership and is directly responsible for AI automation outcomes. Don't just give them "AI adoption goals", give them AI quality goals.
6. Validate your data sources and set clear data quality gates. No automation on experimental data sources.
7. The worst that can happen is that your people forget how to do things. You need a rollback path. What happens if the automation becomes too costly (token creep) or starts acting up (runaway bots)?
8. AI doesn't mean governance isn't a thing anymore. You'll have to lean heavily on change-managed, staged rollouts and canary testing. These things are not new. We've learned them from Cloud and DevOps adoption. All the old things are new again.
Storms added that he sees organizations with the biggest gains moving beyond individual AI assistants toward AI orchestration. “That means understanding which models are best suited for different tasks, running multiple agents in parallel, offloading long-running or repetitive work to cloud agents while keeping higher-risk decisions under human supervision, and investing time upfront in planning and problem definition before generating a single line of code,” he said.
“The most successful teams I've spoken with spend hours collaborating with AI on research, architecture, and design before implementation. Just as cloud leaders learned to build cloud native architectures instead of treating AWS like another data center, the next generation of AI leaders will redesign their workflows around AI rather than simply accelerating yesterday's processes,” Storms said.
AI doesn't break vulnerability remediation, identity systems, or incident response on its own. The breaking of things is enabled by an organization’s own bad data and broken security processes. And AI just does it faster and cheaper.
