CodeCrucible: Open blueprint for LLM-based code vulnerability scanning

CodeCrucible is an LLM-driven SAST tool built to discover code vulnerabilities, but the team's primary intent is to share the design blueprint behind it—the architectural choices, tradeoffs, and implementation patterns that let other teams adapt the approach to their own pipelines.

The tool's core innovation is whole-repository concatenation: packing the entire codebase into a single LLM call, adding directional prompts, and letting the model reason across the whole thing. Most shipping LLM-driven SAST systems put a traditional analyzer (such as CodeQL or Semgrep) in front of the model; the analyzer nominates candidate findings, and the LLM validates them. CodeCrucible flips that relationship: the LLM does the primary analysis, and the team pays in tokens instead of static-analysis engineering. This matters because snippet-anchored systems only show the model what the upstream engine already knows to look at; if no rule fires or the engine fails to trace a path, the LLM never sees the code. Whole-repo analysis removes that limitation.

The article identifies four core design questions any LLM-driven SAST tool must answer: (1) How do you compress the codebase into the model's context window (options include AST summarization, embedding-based retrieval, agentic exploration, and snippet analysis)? (2) How do you ask the model to find vulnerabilities (open-ended prompts, CWE-specific passes, self-critique loops)? (3) How do you separate real findings from confidently stated but unsupported ones (second-pass LLM review, deterministic filters, taint-style verification, human triage)? (4) How do you make nondeterministic LLM output usable in a pipeline designed for consistent outputs (temperature-zero generation, multi-sample voting, structured output constrained by schema, deterministic post-processing)? The article emphasizes that these choices are not independent; selecting one shape forces different choices elsewhere, and laying them out together makes the tradeoffs visible before implementation.

Whole-repo concatenation worked better than expected for three reasons. First, modern reasoning models hold long code contexts better than the field assumed when snippet-first conventions took hold; a 200K-token window evaluating a 50K-token repository operates well within its comfort range. Second, code compresses much more aggressively than prose once you strip low-value content such as binaries, lockfiles, vendored dependencies, and generated artifacts—the opposite of most retrieval-augmented generation (RAG) systems tuned for text. Third, cross-file reasoning is one of the places where LLMs are genuinely better than deterministic SAST, so each additional file in context can add real value. Prior work, notably Snyk's CodeReduce paper, had ruled the idea out by compressing code 20–23× with hierarchical delta debugging before sending it to an LLM, on the premise that the model should see only the reported defect and required context. The article counters that token cost, hallucination on long contexts, and irrelevant code are engineering problems—filtering, chunking, two-pass review, cost controls—not reasons to reject the architecture.

To handle scale, CodeCrucible treats whole-repo analysis as the default and makes compromises explicit when necessary. It estimates worst-case prompt overhead, skips feature detection when a single chunk fits, reruns overhead calculation when feature detection trims the prompt, and only then computes chunk budget. If chunking becomes necessary, the strategy is to chunk as late and as semantically as possible: keep file boundaries intact, group related files by import graph and directory proximity, duplicate small high-priority shared files for clarity, and merge undersized chunks to avoid repeating prompt overhead. Critically, the tool never slices files to fit the budget; if a file is too large, it is skipped with an explicit warning.

Filtering precedes architectural decisions. CodeCraucible filters in layers: filesystem walk (honoring .gitignore, skipping obvious non-source), heuristic filtering (tests, documentation, vendor code, low-value operational files), and large-file exclusion. Importantly, filtering preserves structure: small files like .proto, .sql, and GraphQL files are always included because they define trust boundaries, and local import graphs are maintained so related code stays together once whole-repo analysis stops fitting. Prompts receive the same discipline, split by phase (feature detection, main analysis, audit, CWE-specific deep review, optional context compression) and tailored by codebase. On smaller repositories the analysis prompt includes every section; on larger ones a cheap feature-detection pass over manifests and entrypoints determines which prompt sections to include. The team discovered that elaborate prompt scaffolding mattered less than expected; a short Carlini-style CTF prompt framing the task as a direct "find the bugs" security challenge performed as well as heavily steered versions while being cheaper and faster. The repository includes three prompt variants for reference: default detailed prompts (prompts/default/), the shorter Carlini-style set (prompts/carlini/), and an exploit-focused variant (prompts/exploit-proof/), explicitly preserved so other teams can experiment rather than treating prompts as temporary tuning files.

The release of CodeCrucible reflects a broader shift in how teams approach code analysis at scale. The article identifies a key insight: older advice about aggressively restricting inputs to language models was premised on tight context windows—a constraint that no longer holds in the same way. Modern reasoning models operate comfortably within 200K-token windows when analyzing 50K-token repositories. This opened a design space that prior work (notably Snyk's CodeReduce paper) had closed off, compressing code by 20–23× before sending it to an LLM on the assumption that models should see only the reported defect and its immediate context.

The article frames its contribution not as a finished tool but as a blueprint—a reusable set of design choices and implementation patterns that other teams can adapt. This distinction matters because SAST integration is environment-specific: how a scanner is invoked, where it runs, and how findings surface depend heavily on the pipeline. A black-box tool often requires significant rework to fit a new environment. By releasing the design reasoning alongside the code, the authors aim to help others make similar architectural decisions in their own contexts.

The whole-repository approach works for three concrete reasons the article identifies: modern context windows are large enough, code compresses much better than prose (because low-value content like binaries and lockfiles can be stripped), and cross-file reasoning is a genuine LLM strength where rule-based static analyzers struggle. The chunking strategy—keeping files whole, preserving import relationships, and never slicing code to fit a budget—reflects a practical philosophy: token budget is the scarce resource, so filtering before chunking matters more than clever compression.

No comments yet. Be the first to share your thoughts!