Honestly I think non-malicious odd behavior is under-weighted when it comes to AI agents. Even the example in this paper is about someone suspiciously accessing sales data when "why did you do that" often comes down to something in the model's training that fired as a reflex
Am I undertanding this correctly? The idea is to have ultimately an AI decide if I can have access to a resource based on dynamic inference, identity , intent and service signals that can easily be manipulated?
Unless I gravely misunderstood the text, this seems like a terrible idea (fancy non-scifi, but still terrible)
Oh yeah, a company whose business model is taking everyone’s data and selling it is going to help me secure my data. I guess there’s one born every minute…
There is a massive difference between Google for enterprise customers and Google for consumers.
The consumer offering is massively subsidized by ads and will use your data for ad placement, although they still never sell your data because that would hurt their business.
The Enterprise offering guarantees you contractually that they never touch your data.
Honestly I think non-malicious odd behavior is under-weighted when it comes to AI agents. Even the example in this paper is about someone suspiciously accessing sales data when "why did you do that" often comes down to something in the model's training that fired as a reflex
I wrote about this a few days ago https://firasd.substack.com/p/accidental-data-loss-in-claude...
"Many researchers have made demos along these lines:
An agent is asked to check a webpage like example.com
The webpage asks for a name to proceed further
The agent calls example.com/evil?myname=John, thus sending the user’s name from the context window to the external server.
In practice, however, these elaborate ‘confused deputy’ exfiltration attacks seem rare compared to widely-reported data loss incidents.
The risk of undermining the user’s interests through clumsiness deserves at least as much scrutiny as the risk of leaking secrets."
Am I undertanding this correctly? The idea is to have ultimately an AI decide if I can have access to a resource based on dynamic inference, identity , intent and service signals that can easily be manipulated?
Unless I gravely misunderstood the text, this seems like a terrible idea (fancy non-scifi, but still terrible)
Oh yeah, a company whose business model is taking everyone’s data and selling it is going to help me secure my data. I guess there’s one born every minute…
This trope is so tiring.
There is a massive difference between Google for enterprise customers and Google for consumers.
The consumer offering is massively subsidized by ads and will use your data for ad placement, although they still never sell your data because that would hurt their business.
The Enterprise offering guarantees you contractually that they never touch your data.
Where do you see them offering this service to you? They’re just publishing what they developed internally, which is what they often do.
Now, Cloudflare, on the other hand, would be much more likely to offer a service like this.