The AI supply chain has a serious visibility problem, and new research from Dragos shows just how easily sensitive information can escape.
HD Moore, chief research officer at Dragos, spent five months testing what happens to data after users submit prompts to AI services. The results: evidence of compromises at three AI providers, marketplace resellers accessing customer prompts, and privacy protections failing to prevent sensitive information from reaching unexpected destinations.
The investigation, detailed in Moore's November 19 Dragos research report, involved approximately two million prompts containing unique tracking URLs and credentials. The researchers monitored whether those planted secrets appeared elsewhere or were used to access controlled systems.
The findings challenge a fundamental assumption of enterprise AI adoption: that organizations know who processes their prompts and can rely on providers to enforce their data-handling requirements.
Three breaches, thousands of unauthorized requests
The most serious discovery involved attackers stealing prompt data from three AI hosting providers.
In the largest incident, approximately 200 planted AWS credentials were exposed during several windows in late October. Some credentials were stolen from requests rejected before a model even processed them, suggesting attackers were intercepting data at the provider's entry point.
Attackers subsequently tested credentials, enumerated Amazon S3 buckets, searched Secrets Manager and Parameter Store, and attempted to establish persistent access through IAM users and Amazon Bedrock.
The planted credentials generated tens of thousands of AWS API calls.
Moore's team also identified activity involving ARTEX, an AI-driven penetration-testing framework, although its observed use was limited to checking whether stolen credentials remained valid.
Importantly, some compromised requests had been marked for Zero Data Retention (ZDR). The designation offered no protection against attackers already inside the provider's infrastructure.
The cheapest AI provider may carry hidden costs
The investigation exposed another problem: AI inference marketplaces where resellers compete to process customer requests.
Some sellers could access and retain prompts simply by enabling standard request logging. No sophisticated exploit was necessary.
Researchers observed individuals using stolen prompt information to attempt SSH logins and access test systems. In one case, requests passed through an additional marketplace without that routing being disclosed to the customer.
That matters because AI requests may pass through several intermediaries before reaching the intended model. Each intermediary potentially gains access to sensitive prompts and responses.
The research also documented prompts reaching Chinese AI organizations through U.S.-based providers, despite privacy settings intended to restrict collection.
One planted URL appeared in a Chinese company's internal chat 24 days after the original request, demonstrating that the prompt information remained accessible weeks later.
Canary tokens expose what providers cannot see
Moore's team used canary tokens: unique, monitored credentials and URLs designed to reveal unauthorized access.
Each token was associated with a specific prompt, provider, router and privacy setting. When someone accessed a planted resource, researchers could trace the activity to the original request.
The team also developed techniques resembling traceroute for AI inference, revealing undisclosed intermediaries and unexpected routing paths.
Across approximately 2,000 traced routes, hundreds contained unidentified systems.
What security teams should do now
Moore's findings point to five practical priorities:
- Limit sensitive prompts: Keep credentials, confidential source code and sensitive operational information out of unnecessary AI workflows.
- Evaluate providers: Favor direct relationships with established providers when handling sensitive information.
- Audit routing: Identify intermediaries and verify where prompts actually travel.
- Deploy canary tokens: Use controlled, fake credentials to detect unexpected access.
- Verify privacy controls: Treat retention promises as requirements to test, not guarantees.
The research does not suggest organizations should abandon AI. It demonstrates that AI supply chain security requires the same scrutiny applied to other critical third-party services.
For security leaders, the central question is no longer simply whether an AI model is secure. It's whether they can account for every organization handling the data sent to it.
As Moore's investigation demonstrates, the answer may be considerably more complicated than the provider's privacy policy suggests.
