The number that should worry you
Between July 2 and August 2 of this year, the AI tooling we use to build and maintain our own systems wrote 247,000 lines. That is a count, not an estimate. We went back through the session records and added it up: 247,000 lines across 1,995 distinct files, in a single month.
If you are considering hiring a firm that works this way, that number should give you pause. It gave us pause, which is why we measured it in the first place. So before we make any argument about what it means, let us take it apart, because it is worth considerably less than it looks.
About half of it is not code. By line count, roughly half the output is Markdown: notes, plans, design documents, checklists, and written records of what was changed and why. That prose is genuinely useful — several of the fixes described below exist only because a previous session wrote down what it had learned — but nobody should count a design document as software.
A line written is not a line kept. Line counts count writes, not survivors. One ops script on our systems currently carries eight backup generations sitting beside it — it has been written, replaced, and written again that many times, and only the most recent version does any work. Rewriting is normal and healthy; it is how anything gets better. But a running total of lines written cannot tell a line that survived from a line that was replaced an hour later. It counts both, identically.
It says nothing at all about quality. A generated file can be tidy, consistent, pass every automated check, and still be the wrong answer to the problem. The measurement is throughput. Throughput is not correctness, and it is nowhere near judgment.
So the honest version of the headline is this: in a month of ordinary work, our tooling produced a large volume of text, about half of it prose, some of it several times over. That is the entire claim. It is not a productivity boast, it is not evidence that anything we shipped is good, and if a vendor quotes you a number like it without the caveats, the number is marketing.
We are publishing it anyway, because the volume is exactly what makes the risk worth talking about. Speed is not the interesting part of AI-assisted work. Every failure mode described in the rest of this article existed long before anyone let a machine write a patch. A tired engineer at the end of a long day makes the same mistakes. What changes is the rate. A person writes a fragile one-line fix a few times a week; automation writes one every few minutes. Any silent failure rate that was survivable at human speed stops being survivable when you multiply it by a thousand.
That is the actual question a business buyer should be asking, and it is not "how much can your AI write." It is: what stops the wrong thing from landing, and how would you know if it had?
We can answer that concretely, because for the first half of this year we could not. What follows is three real failures in our own production systems — not client work, not hypotheticals, not anonymised anecdotes from a conference talk. All three have the same shape. In every case the tooling reported success and nothing had happened. Then we will describe what we built so it cannot happen the same way again, and what any of it has to do with your backups.
The patch that changed nothing
Here is the simplest one, and the one that took us longest to notice.
An enormous amount of routine maintenance is string replacement. You have a configuration file on a running server, and you need one value in it changed. The tool reads the file, finds the text it is looking for, swaps it, and writes the file back out. This is the most common shape of automated change there is.
Now suppose the text it is looking for is not there. Someone already fixed it a different way last month. The indentation is two spaces instead of four. The setting moved to a different file in a version upgrade. The pattern matches nothing.
What happens next is the problem. The replacement produces the original content unchanged. The tool writes that content back to disk, byte for byte identical to what was already there. The file's modification time updates, because a write occurred. The command exits with a success code, because nothing went wrong — the tool did exactly what it was told, and what it was told happened to be a no-op. Every layer above it sees success. There is no error, no warning, and nothing to alert on. The change simply never happened, and everything downstream believes it did.
When we finally went looking, we audited a month of our own automated activity and counted every write our tooling had made as root into the file trees of running services. There were 67 of them. These next three figures are overlapping views of that same set of 67, not separate groups, so do not add them up — a single careless write can appear in all three:
- 19 performed a text substitution with nothing checking that the pattern had actually matched anything.
- 47 took no backup of the file before modifying it.
- 44 were never verified afterward in any way — no read-back, no comparison against what was intended, no restart-and-confirm.
Worse than the counts, we found three specific occasions — July 10, July 14, and July 25 — where a run had reported the change as verified and the change had not landed. Not "probably did not land." We went back and read what actually happened on disk, and the file was unchanged while the record said otherwise.
Think about what that costs in practice. The immediate cost is that the system keeps running with the old behaviour while everyone believes it has the new one, which can go on for hours or for weeks. The second cost is worse: eventually someone notices the symptom, re-applies the same fix, and it fails the same silent way. Now the fix has "not worked" twice, so the team concludes the fix is wrong and goes hunting for a different explanation. We have watched an afternoon disappear into that. The most expensive kind of bug is not the one that breaks loudly. It is the one that makes your own troubleshooting lie to you.
None of this is specific to AI. A human writing that same maintenance script writes the same bug, and has been writing it for twenty years. But a human edits a handful of live service files in a week and can hold them all in their head. In one month our tooling made 67 such edits, and we knew what was in them only because we went back afterward and read every one.
The file nobody could delete
The second failure looks nothing like the first, and is exactly the same underneath.
Applications built on modern language ecosystems keep their third-party dependencies in a directory inside the project. Installing or upgrading a package rewrites large parts of that directory. On a properly set up server, the application runs as its own restricted account, and everything in the project is owned by that account, so the application can manage its own dependencies and nothing else on the machine.
If you run the install as root instead — which is the path of least resistance, because root can do anything and never gets a permission error — the installer scatters root-owned files through that dependency tree. The application keeps working. It can still read everything. Nothing crashes, nothing alerts, no page loads slower. From the outside the machine is perfectly healthy, and it stays perfectly healthy for weeks.
The bill arrives at the next update. The service account, doing the install correctly this time, tries to remove a file it does not own, gets a permission error, and dies partway through rewriting the dependency tree. Now you have a half-installed application, a maintenance window you did not plan for, and a stack of orphaned temporary directories nobody can clean up without elevated access. The failure is separated from its cause by however long it happened to be between updates, which is exactly the gap that makes it hard to diagnose.
In June we went looking for this across our own machines and found it on three hosts in a single day. Roughly 4,800 stray files on one, about 1,300 plus a set of orphaned temporary directories on another, and about 15,500 on a third. All three hosts were serving traffic normally. Not one of them had produced a single alert, because from a monitoring standpoint there was nothing wrong. There was only something that would be wrong later.
The rule we adopted afterward is unglamorous. Every project tree has exactly one canonical owner account, and every install, build, and dependency operation in that tree runs as that account — never as root, no exceptions for convenience. Before touching dependencies at all, we check the tree for files that do not belong to the canonical owner, clean up any orphaned temporary directories left by a previous failure, and repair ownership. Only then do we start. And we verify afterward, because package managers are notorious for reporting success on partial work, particularly when their output has been piped somewhere and the real exit code got lost along the way.
Note the shape, because it is the same shape as the last section: an operation reported success, the system looked healthy, and the damage was real but deferred. The only thing that changed was which layer told the lie.
The rules that stopped loading
The third failure is the one that genuinely rattled us, and it is the one that matters most to anyone evaluating a vendor's process, because the thing that failed was the process itself.
We keep a written operating standard for this work. It is a file loaded at the start of every session, and it contains the rules the automation is required to follow — which account owns which project, which library versions are pinned and must not be upgraded. It is the accumulated scar tissue of every problem we have already had. It is the reason we do not have those problems twice.
Over months, it grew. By mid-July that index had reached 185KB across 439 lines. Every one of those lines was there because something had gone wrong once.
The mechanism that loads it has a size budget. When we checked how much of the file was actually reaching the tooling at the start of a session, the answer was about 40 lines. Everything past that point was silently dropped. Not truncated with a warning — dropped, with no indication anywhere that a limit had been hit. The file on disk was complete and correct, and it had been sitting there being complete and correct for weeks while barely a tenth of it was being read.
The safety rules were in the part that was not being read. The entire section — the ownership rules, the do-not-edit list, the version pins, all of it — had quietly stopped existing from the tooling's point of view, while continuing to exist perfectly well from ours. Every review of that file would have passed. The rules were all there. They were just not being applied, and nothing in the system was capable of telling us so, because the rule that would have caught it was inside the section that was not loading.
That is the whole lesson of this article in one sentence: the guardrails were off, and the guardrails were what would have told us the guardrails were off.
There is a second lesson underneath it. This was not the first time the file had gotten too long. It was the seventh. Seven separate times, someone had noticed the file was bloated, sat down, trimmed it, archived the old material, and restored it to a sensible size. Every one of those cleanups worked. Every one of them was undone within days, because many sessions write to that file, each addition is individually reasonable, and nothing between cleanups enforced the limit. We were applying discipline by hand at a rate slower than the thing we were disciplining.
The fix was not an eighth cleanup. The fix was to move the limit into the write path. Every process that adds a line to that index now trims the file to its budget as part of the same write, spilling the overflow verbatim into an archive that stays searchable. The invariant is now enforced by the only thing that can violate it. There is no longer a cleanup to remember, because there is no longer a way to break it.
A rule that depends on somebody remembering to apply it is not a rule. It is a hope with documentation attached.
What we built: gates, not trust
Those three failures share one property, and it is the property everything we built is aimed at. In each case the system reported success and nothing had happened. Not "something went wrong and we missed the alarm" — there was no alarm to miss, because from every layer's own point of view the operation had gone perfectly. You cannot fix that class of problem with better intentions, more careful review, or a longer checklist, because the failure is invisible to exactly the people doing the checking. You fix it by making the machine prove the outcome.
So we built gates. Here is what they do, described by behaviour rather than by implementation.
Before and after every command that could write, we take a snapshot. For each file in scope, we record a hash of its content, its modification timestamp, its owning account, and its permission mode. When the command finishes, we take the same snapshot again and compare the two.
A file whose timestamp moved but whose content is byte-for-byte identical stops the run. That combination is the exact signature of the first failure: a write happened, and it changed nothing. There are legitimate reasons for it, and when there are we say so explicitly. What is no longer possible is for a patch that matched nothing to slide past reporting success.
The gate never reads the command to guess what it did. This is the transferable idea, and if you take one thing from this article into your own systems, take this one: it does not parse the command, it inspects the filesystem. Guessing intent from a command line is a trap. There are unlimited ways to write a file, commands can be built up from variables, wrapped in shells, chained behind conditions, or generated on the fly. A checker that reasons about what a command probably meant is a checker you have to keep teaching, and it will always be one syntax behind. The filesystem before and the filesystem after are the only source of truth, they are the same two things to compare no matter what ran between them, and they cannot be talked out of what they saw.
A second gate watches ownership and permissions. If a command would leave a file inside a service's tree owned by root instead of the service account, or would change a file's permission mode in a way that was not asked for, the run stops and reports it. That is the second failure caught at the moment it is created, rather than at the next update weeks later.
Enforcement lives inside the lock, not beside it. The operating-rules index is written by several sessions at once, and it was already protected by a lock so that two writers could not clobber each other. The size limit is now enforced inside that same lock, as part of the write that acquires it. There is no window in which a writer can add a line and skip the trim, because they are the same operation. Anything that can be skipped eventually is.
Fragile fixes are re-applied and reported daily. A few of our patches live in files we do not own — a small correction inside a third-party dependency that any reinstall of that dependency silently erases. Those are checked once a day by a job that verifies the patch is still present, re-applies it if it is not, and notifies us that it had to. The notification is the important half. A repair that happens quietly teaches you nothing and hides a pattern; we want to know that a reinstall wiped something, not merely that it got put back.
Backups are taken before edits, as a rule rather than a habit. Every modification to an existing file writes a timestamped copy alongside it first. This is not clever. It is the reason that recovering from a bad automated change is a one-minute operation instead of an incident.
The gate has a test suite, and its cases are the three failures above written as reproductions — we can make each original failure happen on demand and watch the gate stop it. We add a case every time we find a new way for the system to lie about what it did. That is the whole maintenance model: the suite grows by failure, not by imagination.
Two honest caveats. First, the gates produce false positives, and we spend real time tuning them. A tool that never blocks anything is not protecting you, and a tool that blocks constantly gets switched off, so the tuning is permanent work rather than a phase. Second, and more important: none of this evaluates whether a change was a good idea. The gate has no opinion about architecture, security posture, or whether the fix addresses the actual problem. That judgment belongs to people, and it always will. The gate answers exactly one question, which is the question nobody was asking: did the thing you said happened actually happen?
Why this matters for your business
You do not need to run a server for any of the above to be about you. The specifics were ours; the failure underneath them is almost certainly already sitting somewhere in your operation, reporting green.
Every one of the three stories is the same failure: a system reported success while doing nothing. Strip out the technology and you have seen this before.
- A nightly backup job that runs on schedule, finishes without error, and writes an empty file. The dashboard is green for a year. You discover it on the day you need to restore.
- Monitoring that stopped polling a machine and never alerted, because alerts are triggered by bad readings and there were no readings at all. Silence read as health.
- A patch run that could not reach any of the target machines, skipped all of them, and returned green because nothing failed — nothing was attempted.
- A nightly export to your accountant that has been writing a zero-row file since a schema change in March. Delivered on time, every time.
- A form on your website that submits successfully, thanks the customer, and sends the enquiry to a mailbox nobody has opened since an employee left.
In every case, the reporting layer is measuring the wrong thing. It is measuring whether the process ran, not whether the outcome changed. Those are different questions, and almost every default monitoring setup answers only the first one.
AI does not create this class of failure. It changes the rate. Automation performs the kind of operation that fails this way dozens of times a day rather than a few times a week, and it performs them with total consistency, which means it also repeats the flawed ones with total consistency. A silent failure rate you could absorb at human pace becomes a real exposure at machine pace. That is not an argument against using AI. It is an argument that using AI without verification is a different and considerably worse proposition than not using it at all, because you get the volume and keep the blindness.
So when you are evaluating any vendor who tells you they use AI — and by the end of this year that will be all of them — the useful questions are not about which model they use or how much faster it makes them. Ask these instead:
- How do you verify that a change actually landed? Not that the command exited cleanly, or that the ticket was closed. That the file, the setting, or the record is demonstrably different from what it was before.
- What happens when a fix silently reverts? Some things undo themselves — an update overwrites a customisation, a config gets restored from a template, a sync pulls an old copy over a new one. Would you find out from a check, or from us?
- Who checks the checker? If your safety rules stopped being applied tomorrow, what would tell you, and how long would it take?
- Can you show me a change you made three months ago and prove it is still in place today? This one is unreasonably informative. Most people cannot.
- What is backed up before an automated change, and when did you last restore from one? An untested backup is a belief, not a backup.
- What is your tooling allowed to do without a person in the loop, and where is that boundary written down? If the answer is "we are careful," the answer is no.
Notice that not one of those questions is about AI. They are the questions you should have been asking about any automated process for the last twenty years. AI has simply raised the cost of getting them wrong to the point where they are no longer optional.
If you would rather not run that audit on every vendor yourself, answering those six questions on a client's behalf is a fair description of what our managed IT services actually are.
We are not claiming a perfect record. We are claiming something narrower and, we think, more useful: the three failures in this article were found by us, in our own systems, before they cost a customer anything — and every one of them is now a case that a machine checks for, on every single run, without anybody needing to remember.
That is what the number at the top of this article is actually evidence of. Not speed. If we were letting AI write at that volume without gates, the honest recommendation would be to hire somebody else. The volume is not the achievement. The volume is what the discipline makes safe.