The agent deleted production because the token let it
An agent deleted a production database with one GraphQL mutation. I rebuilt that call against the same API with a rule in front of it. What it stops, and what it doesn't.
$ curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d 'mutation { volumeDelete(volumeId: "3d2c42fb-...") }' # one call. the production volume and its backups.
TL;DR
- In April a coding agent deleted a company's production database, backups too, with a single API call.
- Nobody meant to give it that power. A token that could delete production was in a file on the machine it ran on. It found the token.
- If you want an agent to do real work, I'd give it the least it needs. A handful of named operations is plenty, and the token stays with you.
- Keep a person in the loop for anything that writes.
- Some things should just be denied. Deleting a volume is one of them. No approval step, nobody to ask at 2am.
In April a founder posted that a coding agent had deleted their production database. The Hacker News thread has 860 points and 1,032 comments as I write this. The whole incident is one command, quoted in the thread:
curl -X POST https://backboard.railway.app/graphql/v2 \
-H "Authorization: Bearer [token]" \
-d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}'A GraphQL mutation, sent with a bearer token the agent found on the machine. According to the post, the volume's backups lived inside the volume, so the same call took those too.
I work on Actori, which sits between agents and APIs like this one. So I did the obvious thing: I rebuilt that call against the same API and put a rule in front of it. This post is what happened, including the part where it wouldn't have helped.
The thread was mostly right
The author was angry that nothing asked for confirmation. The reply with the most answers under it:
“It's an API. Where would you type DELETE to confirm?”
Which, yes. An HTTP API can't open a dialog. If a call deserves a second look, the look has to happen before the request leaves. In this setup there was nowhere for that to be.
“You can't have production secrets sitting where they are accessible like this. This isn't about AI.”
I agree with this. It's a little awkward to agree with, since I sell a product in this area and "this isn't about AI" is not what the brochure says. But look at what happened. A credential that could destroy production was readable on a development machine. A cron job with a typo could have done the same damage. What the agent changes is the odds. It runs a lot of commands and nobody reads them first. Does it know that volumeDelete is a different kind of thing from ls? It doesn't.
One URL, 255 mutations
Railway's API will tell you what it offers without a token, so I asked it. As of today the schema has 160 queries and 255 mutations. Every one of them is a POST to the same address. This is the one from the incident, with the description the schema gives it:
volumeDelete(volumeId: String!): Boolean!
"Delete a persistent volume in a project"That's why a token for an API like this is so hard to hand to anything. There's no path to allow and no method to block. Reading your own profile and deleting a volume differ only in the text of the request body.
While I was there I sent the delete with no token and a made-up id, to see what a refusal looks like.
HTTP 200
{"errors":[{"message":"Not Authorized","path":["volumeDelete"],
"extensions":{"code":"INTERNAL_SERVER_ERROR"}}],"data":null}Status 200. That's normal for GraphQL and it still catches people out. Anything that treats 200 as success has just told the agent the volume is gone.
Putting a rule in front of it
In Actori an API like this is connected as a Custom GraphQL connector. It's deliberately dull. You give it the endpoint and a token (typed on a page in the browser, never in the chat). After that it does nothing at all, because it has no tools. There is no query box. Each thing an agent may do gets written down once, as a fixed document.
For this post I connected Railway's endpoint with no token. The point is to watch the call get stopped, and I'd rather not delete anything real for a blog post.
Then I asked my AI to add the delete as an action. This is the whole definition:
name railway_delete_volume
document mutation DeleteVolume($volumeId: String!) {
volumeDelete(volumeId: $volumeId)
}The variables of the document become the action's arguments. Actori worked this schema out from the document; I didn't write it:
{"type":"object","required":["volumeId"],"properties":{"volumeId":{"type":"string"}}}So the action takes a volumeId and that's all. Leave it out and the call is refused with must have required property 'volumeId'. Send a number and you get must be string. There's nowhere to put query text, which means there's no way to send one of the other 254 mutations through it.
Here's something I hadn't planned to show. Adding that action was itself held. The AI got this back:
Action "add_actions" requires HUMAN approval before it can run. This is not
an error — it has been queued and will execute automatically once a person
approves. [...]
Why it needs approval (the rule): "Added automatically because the tool is
marked destructive: each call waits for a person's approval. An admin may
change this rule after reviewing the action."Fair enough. An agent giving itself a delete tool is exactly the sort of thing a person should see first.
Hold it, refuse it, or leave it out
Once the action exists there are three things you can do with it. For a volume I'd pick the last one.
| What you set up | What the agent gets when it calls delete |
|---|---|
| Nothing extra. A mutation waits for a person by default | told the call is pending, and not to retry it |
| A deny rule with a reason | refused, with your reason in the answer |
| The action was never written | there is no such tool to call |
The default first. Because the document is a mutation, the action starts out waiting for a person. Nobody chose that. This is the rule Actori put on it by itself, copied from the action:
effect require_approval
when always
priority 9999
approver super_admin
reason Added automatically because the tool is marked destructive: each call
waits for a person's approval. An admin may change this rule after
reviewing the action.The approver sees which action, on which connection, with which volume id.
Then I had the agent call it, with a made-up volume id. This is what came back, trimmed:
Action "railway_delete_volume" requires HUMAN approval before it can run.
This is not an error — it has been queued and will execute automatically
once a person approves. approval_request_id: 38479d97-[...]
A person decides it in Actori: https://app.[...]/approvals/38479d97-[...]
Why it needs approval (the rule): "Added automatically because the tool is
marked destructive: each call waits for a person's approval. An admin may
change this rule after reviewing the action."
[...] STOP here. Do NOT start any follow-up step that depends on this actionNothing was sent to Railway. The request sits in a queue until a person opens that link.
An approval doesn't have to be forever, by the way. It's the right default for a call you haven't watched yet. Once a call has behaved for a while you can take the approval off that one and leave it on the rest.
Should deleting a production volume be something a tired person can approve from their phone at 2am? I don't think so. If nobody should ever say yes, it's a deny, and a deny can carry a reason:
effect deny
when always
reason Volumes are deleted by a person in the Railway dashboard,
never by an agent.Action "railway_delete_volume" denied by a policy rule. The rule's reason:
"Volumes are deleted by a person in the Railway dashboard, never by an agent."
Do not retry the same call. If the reason says what a permitted call needs,
and the user wants it, you may make that call instead; otherwise inform the user.Before saving a rule you can try it. I ran the same delete call against the rules as they were, then with the deny added at priority 100, so it runs ahead of the default:
| Rules | Decision | Decided by |
|---|---|---|
| as they were | require_approval | the default rule, priority 9999 |
| with the deny added | deny | the new rule, priority 100 |
The reason is written for the agent as much as for people. It reads the reason, passes it on to you and stops there.
And the third option is the one I'd actually use here. Don't write the action. Give the agent the reads it needs and perhaps a restart. No delete. If nobody wrote down volumeDelete, there's nothing to hold or to refuse.
Would it have saved them?
No. I'd be embarrassed to leave that out.
The agent in the incident didn't go through a tool. It found a token in a file and ran curl. Everything above happens only to calls that go through Actori, so it helps only if the token isn't on the machine to begin with. That is the real change. The token lives in the connector. The agent gets railway_delete_volume or it gets nothing. Is there a file left for it to find? There isn't.
So the order matters. First take the token off the machine and revoke the old one (this guide covers that step, which is the one people skip). Then decide which operations the agent gets at all. Rules on the ones that write come last.
If you want to try it against an API of your own, the free plan covers one person. Setup is one line, then the second in your own words:
Set up Actori for me: https://actori.ai/
Connect my GraphQL API through Actori. Let me list things. Deleting anything must never run.