Quick Answer
Across api-doctor's first five weeks of external usage — 129 CLI runs on 61 real codebases — 67.9% of provider scans surfaced at least one warning- or error-level finding, averaging 2.3 distinct rules per flagged scan. The most common finding wasn't a leaked secret. It was a correctness bug: a Supabase query that selects a tenant column like user_id but never filters by it, firing in one in four scans.
This is proprietary first-party data from real codebases, not another citation of outside research — for the broader security statistics, see the vibe coding security crisis. And read from the other side of the API, it's a measurement providers have never had: every rule encodes a specific documented requirement, so every fire marks a place where a provider's docs — public, often agent-optimized — still didn't make it into the code an agent wrote.
What this data is (and isn't)
Everything here comes from the CLI's anonymous telemetry: which rules fired, on which provider integration, in what run context. No code, no file paths, no project names. All usage by our own team is excluded before any number is computed.
Two honesty notes. First, this is early data — dozens of codebases, not thousands. We report percentages, and we'd rather underclaim than decorate a small sample with false precision. Second, the yardstick differs from lab benchmarks, which grade freshly generated samples. We measure integration code that's actually sitting in a repo — much of it AI-assisted, some human-written — against provider-specific rules. The numbers aren't directly comparable, and we won't pretend they are.
Docs are upstream of all of this
Agents don't invent integration code. They write it from context: training data, docs, MCP servers, llms.txt. A model is only as smart as the context it's given, so the quality of AI-written integrations is largely downstream of how well each provider's context reaches agents.
Here's the strange part: API companies know this and can't measure it. Ask API teams how they evaluate their docs for agents and you hear three answers. Internal evals — prompt a model against the docs and grade the output. Vibe-coded sample projects to see what breaks. And watching support tickets, Discord, and Reddit for complaints. The first two are synthetic; teams keep finding that what passes an eval still fails inside a real repo, where the docs compete with the codebase for the agent's attention. The third is real but arrives weeks late, filtered through whoever was annoyed enough to post.
The industry has invested heavily in the presentation of context — docs platforms, MCP, llms.txt are all bets on format. There's been no way to measure context quality: whether any of it changes what agents actually write. Each rule in api-doctor links to the doc section it encodes, so each fire rate below is a per-requirement failure rate, measured on real code. This is what the missing feedback loop looks like.
Two-thirds of scans find something real
When api-doctor detects a supported provider and scans that integration, 67.9% of the time it finds at least one issue at warning or error severity. These aren't style nits — every rule encodes a documented requirement or a known production failure mode, the kind manual review misses.
Findings also cluster: flagged scans averaged 2.3 distinct rules. Code that gets one invisible requirement wrong usually gets several wrong — the agent had no signal to catch any of them.
Wrong beats insecure
| Category | Scans with ≥1 finding | What it means | |---|---|---| | Correctness | 43.6% | Runs fine, does the wrong thing | | Reliability | 39.7% | Breaks under retries, failures, scale | | Security | 9.0% | Exposes secrets, data, or unverified inputs |
By raw fire volume the skew is stronger: 55% correctness, 39% reliability, 6% security. The public conversation is dominated by security; our scans say the everyday tax is code that is quietly wrong in ways neither a demo nor an agent-written test will catch.
One caveat: our engine ships more correctness and reliability rules than security rules (roughly 53/52/34), so security has fewer chances to fire. But the 5–7x gap in the data is far larger than the gap in rule counts. The skew is real.
The most common bug is a data-isolation bug
supabase-scope-queries-by-tenant-column fired in 25.6% of scans. It catches a query that selects a tenant column — user_id, org_id — but never filters by it. Without row-level security on that table (and in AI-scaffolded projects there often isn't), the query returns every tenant's rows. With one test user, it's invisible: the demo shows the right data because there's only one tenant's data to show.
Now the docs lens. The correct pattern is thoroughly documented — scoped queries and RLS are core sections of Supabase's docs, and the rule links straight to them. The docs are good, and agents still miss the requirement one time in four. An internal eval won't surface that (the eval prompt has no half-built multi-tenant schema in context); Reddit surfaces it after data leaks. A provider seeing this number learns which documented requirement isn't landing — the difference between rewriting docs on instinct and fixing the one section with a measured failure rate.
Rest of the top five: .single() without an error check (17.9%), unchecked mutation errors (16.7%), missing fail-fast env validation (16.7%), non-idempotent mutations (15.4%). All invisible in a happy-path demo.
Where the scans happen
Provider mix: Supabase 53.8% of scans, Resend 19.2%, Tiptap 16.7%, with OpenAI computer-use, Auth0, and Lovable in the tail. That's the vibe-coding stack, almost exactly — these scans skew toward fast-moving, agent-built codebases, which is worth remembering when generalizing.
Run context: 83.7% local, 9.3% CI, 6.2% agent-invoked. Most people run the check by hand, after the fact. Only one run in eleven happens in CI — the one place it covers every PR automatically. Given the two-thirds hit rate, that gap is the one we most want to close; it's why CI is now the primary way we recommend running api-doctor.
Where we could attribute a scanned integration to a specific coding agent (~18% of scans; attribution is conservative), Claude models accounted for the large majority, with OpenAI's Codex making up the rest. Too small to rank agents — but the dimension matters: a requirement that lands with one model and not another is a context problem with a specific fix, and no support channel will ever show you that split.
Not yet in the data: unknown_sdk_calls, our measure of which SDK surface agents actually touch — including endpoints they never discover. The instrumentation is newer than most of these scans; it'll be in a future refresh.
If you're the API provider
Flip every number around and this report is about your documentation:
- Doc sections have failure rates, like endpoints. "This requirement fails to land 25% of the time" is actionable the way an error-rate dashboard is. No eval or Reddit-scrape produces it.
- Measure impact, not publication. Ship an MCP server or rewrite a guide, then watch whether fire rates on the affected requirements move. Context work without measurement is publishing into the void.
- Close the loop where the code is written. This data exists because the check runs in developers' CI, the moment the agent's code is born — the same place the fix can be applied automatically. Feedback that ends in a fix compounds; feedback that ends in a report gets triaged.
Documentation stops being content that gets published and occasionally complained about, and becomes an interface with agents whose performance is continuously measurable.
A living report
The numbers reflect external scans through July 28, 2026. We'll refresh them at this URL as the sample grows, and if a conclusion stops being supported, we'll change it rather than defend it.
Want your codebase in the next edition (anonymously — rule IDs and counts only, --no-telemetry always works)? npx @api-doctor/cli . — or better, put it in CI, where the bugs actually get caught.