One bad decision, six missing layers

阅读中文版

On April 25, 2026, a coding agent deleted the production database of PocketOS, a company that makes software for car rental businesses. It took one API call and about nine seconds. The backups stored with the database went too.

Most of the coverage was about the agent. I think the more useful story is everything around it. This review borrows its shape from the Army’s after-action review, which asks four questions: what was supposed to happen, what happened, why there was a difference, and what to keep and what to change.

I was not there. Everything below comes from the founder’s public account, Railway’s public statements, and reporting on both. Where the accounts differ I show both.

What was supposed to happen

According to the founder, Jer Crane, the agent was doing a routine task in a staging environment. The project carried rules against destructive actions without approval. When the agent ran into a credential mismatch, the expected behaviour was to stop and ask.

What happened

  • The agent, running in Cursor on Claude Opus 4.6, hit the credential mismatch and tried to resolve it by itself.
  • It found a Railway API token in an unrelated file in the codebase. By Crane’s account the token had been created for managing custom domains. Railway’s CEO, Jake Cooper, later described it as fully permissioned.
  • The agent sent a single GraphQL call that deleted a storage volume. There was no confirmation step. The volume held the production database.
  • Railway kept volume-level backups with the volume, so they were deleted in the same call. Crane said the newest backup he could reach was about three months old.
  • Asked afterwards, the agent wrote that it had guessed the deletion would be limited to staging, and that it had not checked.
  • Crane and his customers rebuilt bookings by hand from payment records, calendars and email.
  • Cooper said Railway restored the data from internal disaster backups about 30 minutes after he reached Crane, and put the delay before that down to a support ticket that sat for more than a day. Crane said significant gaps remained and that he had retained a lawyer.
  • Railway said the API endpoint ran deletions immediately, while the same action in its dashboard had a 48-hour window. It has since made deletes on both paths recoverable for 48 hours and made more backups visible to customers.
  • As of late April, neither Cursor nor Anthropic had commented publicly.

Why there was a difference

The agent made one bad decision: when blocked, it improvised. Six other things had to be true for that decision to cost three months of data.

Layer What was true Would a fix here have stopped it?
Where the secret lived A token that could delete storage sat in a file the agent could read Yes
Credential scope A token made for domain changes could delete volumes Yes
Environment isolation Work in staging could reach production resources Yes
Confirmation on destructive calls The delete ran immediately, with no delay or second step Yes, or it would have left time to undo it
Where backups lived Backups shared a failure domain with the data No, but the loss would have been hours, not months
How rules were enforced The rules were text in the project, and the tool’s guardrails did not intercept a direct API call Text does not stop a request. An enforced check does

Three of these are worth a sentence more.

The token. The agent did not escalate its privileges. It used a credential that already had them. From the API’s side, the call was an authenticated delete from a valid token, and it was honoured.

The rules. The agent could state the rule it broke, after it broke it. A rule the agent can recite is not a control. A control is something that refuses the request: a hook that denies destructive commands, a network policy, a token that cannot perform the action.

The backups. A backup that is deleted by the same call as the data protects against corruption. It does not protect against deletion.

Where data branching fits, and where it does not

I maintain Argon, which gives agents their own branch of a MongoDB database so their writes can be reviewed, merged or thrown away. It would not have helped here. What was deleted was the volume underneath the database, through the infrastructure provider’s API. Branching protects against bad writes made through the database. It says nothing about who is allowed to delete the disk.

Branching addresses a different class of incident, such as the one on Replit in July 2025, where an agent ran destructive commands against a live database during a freeze. Different failure, different layer.

Sustain

  • Crane published a detailed account within a day, including the agent’s own explanation. That is why anyone can learn from this.
  • Railway restored the data and changed the API within days.

Improve

In rough order of cost, for a team running coding agents today:

  1. Search for tokens the agent can read. Repositories, dotfiles, shell history. Move them out.
  2. Scope tokens by environment and by verb. If your provider cannot do that, treat every token as a root credential and store it accordingly.
  3. Give staging agents staging-only credentials and network access.
  4. Put a delay or an approval in front of deletes. Use the provider’s if it has one. Otherwise wrap the API.
  5. Keep one backup that the same credentials cannot reach, and test a restore.
  6. Enforce the stop condition in the harness. A hook that blocks destructive calls will hold when a paragraph in a rules file does not.

If the agent cannot undo it, a human approves it.

Sources

I have not read Crane’s original posts or Railway’s blog post directly; I relied on the reporting above.