The Day Axia's Safety Systems Caught Something We'd Been Missing For Three Weeks

Autonomous systems are only as good as what happens when they fail.

Last week, Axia — the AI business operating system V8 is building commercially — had an autonomous deploy bot quietly destroy 17 commits of documentation. Architectural decisions, methodology writing, implementation briefs. Roughly 2,600 lines of proprietary work that sits at the core of an active partnership negotiation.

The bug had been there for three weeks.

The recovery took 30 minutes.

I want to write about why, because the gap between those two numbers is the whole point.

What happened

Axia’s staging environment promotes new code to production through a bot that waits for approval in Discord. I tap a button, the bot does the work, the system moves on.

Under the hood, the bot was using a destructive git pattern — overwriting repository history rather than merging into it. For weeks that worked fine, because the state of the history happened to be safe for destruction. Nothing of value got erased because nothing of value was in the path.

Then the state changed. Work had been committed from a different machine. The next promotion would have erased it.

It didn’t — but only because three separate defences caught it. The local development machine still held the full correct state. A sandbox server had the dangling commit recoverable through git’s garbage-collection system. And the Scaffold methodology that V8 runs on explicitly mandates diagnosis before recovery — resist the urge to “just fix it” until you understand what broke.

That third one is the interesting part. The engineering reflex when you see something broken is to undo the last thing that changed. That reflex would have destroyed the work permanently. Pausing to diagnose is what saved it.

What was almost lost, and why I’m writing this anyway

Weeks of methodology writing, architectural records, and implementation briefs — material that’s directly commercially relevant right now.

The reason to write about it publicly isn’t that it’s dramatic. It’s that it’s normal.

This was the fourth incident of the same class. The rule against the destructive git pattern had been declared explicit on April 8. I’d written it down. I’d reviewed it. I’d reminded myself. And the failure happened anyway, because the rule lived in a document, not in the system.

Documentation-only rules drift. Structural rules don’t.

Willpower and memory are variable inputs. Servers aren’t. If you want a rule to hold, the rule needs to live somewhere the system will refuse to compromise on.

The fix

Two layers.

The first is where the destructive git pattern now gets rejected — not by the bot, not by me, but by the server that hosts the repository itself. The rule is no longer something anyone has to remember. The system will reject the command regardless of who or what issues it.

The second is that this protection required a small upgrade in repository plan, which is a trivial monthly cost for permanent structural defence. The equivalent of installing a proper lock instead of a sticky note saying “please remember to close the door.”

I’m making a point of not fixing the client-side bot code. The server-side enforcement makes the bot’s behaviour irrelevant — if the bot tries the bad pattern, it just fails safely. That’s the point of layering defence closer to the data you’re protecting.

Why this matters beyond the incident

Most of the operational incidents a solo founder or small team deals with aren’t dramatic. They’re small latent risks that sit in the system for weeks, unnoticed, until the state of the world happens to align against them. By the time the pattern is visible, the damage is already done — or nearly done.

Operational rigour isn’t paranoia. It’s knowing which failure modes are worth designing against before they happen.

For this one, the cost of a structural defence was roughly the price of a monthly coffee. The cost of what it protects against is work I can’t afford to reproduce. That’s an easy trade once you see it framed that way. The harder discipline is doing that calculus before the near-miss, not after.

How this connects to what V8 builds

Axia is built using what we call Scaffold — an internal methodology for running AI-native operations where a small team (or one person) moves at the velocity of a larger one. A core Scaffold principle is: design for the recovery, not for the best case. Assume things will go wrong. Make “wrong” cheap to come back from.

This post is an example of the methodology applied to itself. The bug existed. The system caught it. The lesson converted into structural defence that no future version of me can forget to apply.

If you run a business that depends on your own work output — client deliverables, IP, proprietary methodology — the same question is worth sitting with: what are the failure modes I’m currently defending against with memory and vigilance, rather than with structure?

Those are the ones that will eventually catch you out.

---

Alan Law is founder of V8 Global and architect of Axia. He writes Operator’s Log posts on how AI-native systems get built in practice. For the community-facing side of V8, follow Gina Cheng.