The Model Writes the Bug and the Confidence

Fluency Is Not Correctness, and Confidence Is Not Safety

The defining trait of AI-generated code is that it looks right. Clean structure, sensible variable names, plausible comments, and a tool that says "done." For a non-specialist — or a specialist moving fast — that surface fluency reads as quality. It isn't. The model optimises for code that resembles working code in its training data, not code that is secure under adversarial conditions.

The Verge case is instructive precisely because it's mundane. SQL injection is a decades-old, well-understood vulnerability class. The model knew what it looked like and produced it anyway, because the prompt asked for a feature, not a threat model. No one was attacking the site during development, so nothing flagged the hole. It shipped, it worked, and it stayed broken for months.

This is the asymmetry executives need to internalise: AI compresses the time to a working-looking product, but it does nothing to compress the time to a safe one. The gap between those two states is exactly where the risk now lives — and it's wider than it's ever been, because more code is being written by people who can't read it critically.

The model wrote the bug and reported success in the same breath.

You Can't Audit the Auditor

The instinctive fix is to ask the AI to check its own work — "is this code secure?" It will happily say yes. This is the trap. The system that introduced the flaw shares the same blind spots when reviewing it; it lacks an adversarial frame and has every incentive to confirm the work is complete. Self-review by the generator is review theatre.

Security review has to come from an independent layer with a different incentive structure — a human with an attacker's mindset, or tooling whose entire job is to assume the code is hostile until proven otherwise. This is the same principle we argue for when you treat AI agents like untrusted insiders rather than magic: you don't grant trust because the output is fluent, you grant it because something independent verified it.

Practically, that means generated code enters your pipeline as an untrusted contribution — the way you'd treat a pull request from an anonymous contractor you've never met. It gets scanned, reviewed, and constrained before it touches production data, not after.

• →Never accept the generating tool's own "this is secure" as evidence — it's marketing, not assurance.
• →Route AI-generated code through the same static analysis and dependency scanning you'd use for any third-party contribution.
• →Default to least privilege: a freshly generated feature should not have broad database or credential access until reviewed.

The Business Risk Is Asymmetric — and That Should Change the Math

Leaders adopt AI coding because it's fast and cheap, and both are real. But the savings and the exposure live on different timescales. You bank the time saving immediately and visibly — a feature ships in days instead of weeks. The risk accrues silently and lands later: a breach, a regulatory finding, a customer-data leak that surfaces months after the code was forgotten.

One shipped vulnerability can erase the cumulative time savings of an entire quarter of fast delivery, and that's before reputational and legal costs. The Verge's example sat undetected for months — which is the normal case, not the worst case. Most injection flaws aren't discovered by the builder; they're discovered by whoever was looking for them.

The strategic error is treating AI coding as a pure productivity lever when it's actually a risk-transfer mechanism. You're moving work from the build phase to the review phase. If your organisation has no review phase — or staffed it for the old, slower volume of code — you haven't saved anything. You've deferred a cost and hidden it.

The Operating-Model Fix: Volume Went Up, So Review Has To Scale

The honest organisational problem is that AI raised code volume by an order of magnitude without raising review capacity at all. A small team can now generate what used to take a large one — but the security review function was sized for the small team's output. The bottleneck didn't disappear; it moved, and it moved somewhere no one is watching.

This is the same shape as the failures we see in enterprise AI pilots that die around month four: the causes are organisational, not technical. The model works fine. What breaks is that no role owns the gap between "it runs" and "it's safe to run in front of customers." When everyone assumes the tool handled it, no one did.

For non-technical leaders, the fix isn't to slow down adoption — it's to make the review layer an explicit, funded part of the operating model rather than an assumed default. That can mean security gates in the deployment pipeline, a senior reviewer who signs off on anything touching sensitive data, or external review for code that ships fast and stays unowned.

• →Assign clear ownership of the "safe to ship" decision — it cannot be the same person or tool that wrote the code.
• →Add automated security gates to the deployment pipeline so nothing reaches production unscanned, regardless of who wrote it.
• →Scale review capacity in proportion to generation volume, not to headcount — the two have decoupled.

What's Still Genuinely Hard

None of this is solved, and pretending otherwise would be the same overconfidence we're warning against. Automated security scanners catch known vulnerability classes — SQL injection among them — but miss logic flaws, broken authorisation, and business-context bugs that only a human who understands the system can spot. Tooling narrows the gap; it doesn't close it.

The harder open problem is volume-at-scale. When a team ships ten times more code, even a strong review function gets selective about what it inspects deeply. The realistic answer isn't reviewing everything to the same depth — it's risk-tiering: code that touches authentication, payments, or customer data gets human adversarial review; low-stakes internal tooling gets automated scanning and moves on. Getting that triage right is judgment work, and it's exactly the kind of thing that benefits from senior oversight rather than a junior team learning on production.

The thesis holds: the model produces the vulnerability and the confidence together, so the verification has to come from somewhere the model can't reach. The actionable next step is to look at your own pipeline and ask one question — between "the AI says it's done" and "it's live in front of customers," what independent check actually runs? If the answer is "none," you already have a Verge case waiting to be written about you.