Research project

Claude Code Security

Research: AI vs. rule-based bug detection

Hybrid pipeline: 39% → 52% vulnerability coverage

See how it’s builtSource (opens in a new tab)
Illustration: a code scan passing through two lenses, a rules grid and an AI lens
  • Semgrep
  • Claude Code
  • Python
  • Flask
  • OWASP

The story

A research paper asking whether AI can find security bugs better than traditional scanners. The answer: they catch different bugs, so the best result comes from using both, plus a human.

The hard part
Keeping the comparison honest: the ground truth was defined before any tool ran, and Claude Code was non-deterministic (3, then 6, then 11 findings on the same code and prompt), so the conclusion had to rest on bug classes, not a single score.

How it’s built

Inside Claude Code Security

System architecture

  1. STAGE 01

    • Every commit · Client
      • → SAST scan (Semgrep, CodeQL) (syntactic bugs)

  2. STAGE 02

    • SAST scan (Semgrep, CodeQL) · Service
      • → LLM post-filter at PR gate (false positives 68% to 6%)

  3. STAGE 03

    • LLM post-filter at PR gate · AI
      • → Claude Code semantic audit (pre-release)

  4. STAGE 04

    • Claude Code semantic audit · AI
      • → Human review (all patches)

  5. STAGE 05

    • Human review · External

Engineering decisions

  1. Decision 01

    Ground truth first

    The expected vulnerabilities in vulpy were fixed before any tool ran, so no result could be read to fit the story.

  2. Decision 02

    Measure by bug class

    SAST caught 75% of syntactic bugs and 0% of semantic ones. Splitting the results this way shows each tool's blind spot instead of one misleading average.

  3. Decision 03

    Layer the tools

    SAST on every commit, an LLM post-filter at the PR gate (false positives 68% to 6%), Claude Code before release. Coverage rises from 39% to 52% without drowning developers in noise.

  4. Decision 04

    Audit the auditor

    Claude Code's own risks were reviewed through the CIA triad with real CVEs, and its non-determinism (3, then 6, then 11 findings on the same code) is why human review stays mandatory.