← All posts
·3 min read·Reuben Santoso

AST Rules vs LLM Prompts: Why You Can't Ask Another AI to Review AI Code

Testing AI code with AI is like asking a hallucination to detect hallucinations. Here's the deterministic alternative that actually works.

Quick Answer

AST rules vs LLM prompts comes down to determinism. AST rules parse your code into a syntax tree and check it against fixed patterns encoded from provider documentation, so the same input produces the same verdict every time. LLM prompts ask a probabilistic model for an opinion — useful for architecture and logic, but if the original code was hallucinated, the reviewing model shares the same blind spots that produced the bug.

The market for AI code verification is quietly splitting into two camps. One uses another LLM to review your code. The other uses static analysis. People treat them as competitors, but they're really solving neighboring problems, and they are not interchangeable.

LLM-based review

Tools in this camp hand your code to a model and ask it to spot the issues. What you get back is natural language: "this might fail if the API returns a 429." That's genuinely handy for architecture, style, and logic.

Here's the catch. The model is probabilistic. Run it twice on the same code and you can get two different answers. And if the original code was hallucinated in the first place, the reviewing model carries the exact same training priors, so it's liable to miss the one hallucination you most needed it to catch. You're asking a hallucination to fact-check a hallucination.

AST-based rules

Tools in this camp parse your code into an abstract syntax tree and check it against fixed rules. No model, no inference, no vibes. The rule either fires or it doesn't.

This is what api-doctor does. Every rule comes out of provider documentation and gets encoded as a tree pattern. The moment your code uses an API method wrong, whether that's the wrong argument shape, a deprecated parameter, or a missing required field, the matching rule trips.

The tradeoff is honest: AST rules only catch what someone wrote them to catch. They don't reason and they don't generalize. A rule for Resend won't do a thing for your Stripe code.

So which should you use?

Both. Lean on LLM review for the high-level reasoning, and lean on AST rules for API contract correctness. They live at different layers and catch genuinely different things, so you're not really choosing between them.

api-doctor takes care of the AST layer so you don't have to hand-write the rules yourself. And the rules are open source, so you can read exactly what's being checked instead of taking my word for it.

Frequently asked questions

Q: Can an LLM reliably review AI-generated code? Not reliably. The reviewing model is probabilistic and carries the same training priors as the model that wrote the code, so it's liable to miss the exact hallucination you most needed it to catch.

Q: What do AST rules catch that LLM review misses? API contract violations: wrong argument shapes, deprecated parameters, and missing required fields. Each rule is encoded from provider documentation and fires deterministically — same input, same output, every time.

Q: Do I have to choose between AST rules and LLM review? No. They live at different layers. Use LLM review for architecture, style, and logic, and AST rules for API contract correctness.

Try api-doctor

Deterministic AST rules for AI-generated API integrations. Not a prompt.

npx @api-doctor/cli .