The landing page claims 100% recall and 0 false positives on hallucinated package names. Here is the full methodology, the dataset, and the honest caveats — so you can judge the number instead of trusting it.
This benchmark measures one narrow thing: does Graneth correctly tell a real package name from an AI-hallucinated, slopsquatted, or typosquatted one? It is not a measurement of SAST quality, vulnerability detection, or any other part of the scanning pipeline — only package-name verification against live npm and PyPI registries.
| Outcome | Count | Meaning |
|---|---|---|
| TP | 65 | Hallucinated/bad name, correctly flagged |
| FN | 0 | Hallucinated/bad name, missed |
| TN | 66 | Real package, correctly passed |
| FP | 0 | Real package, wrongly flagged |
| Unverified | 4 | Real package — registry lookup transiently unreachable during the run, reported honestly as a third state |
| Count | Category | Source |
|---|---|---|
| 35 | Known AI-hallucination seed set | Lanyado et al., USENIX Security 2024/25; CSA advisory |
| 2 | Published real-world hallucinations, not in the seed set | Lasso Security's huggingface-cli PoC; Socket.dev's react-codeshift case |
| 3 | Historical typosquat takedowns | crossenv, python3-dateutil, jeIlyfish — Snyk / Check Point write-ups |
| 25 | Evaluator-fabricated names — the generalization test | Mimics the “popular-lib + generic-suffix” hallucination pattern; each verified 404 before inclusion; never seen by the detector before this run — all 25 caught |
| 66 | Real, popular packages — true negatives | Live npm / PyPI registries |
| 4 | Real “near-miss” edge cases, 1 edit from a famous name | e.g. jsdoc vs. jsdom |
These numbers come from running Graneth's real production detector — the exact code path a live scan uses — against this labelled sample, checked against the live npm and PyPI registries. Nothing here is a mock or a synthetic stand-in for the detector.
The sample combines known AI-hallucination research data, two independently published real-world hallucination incidents, three historical typosquat takedowns, and 25 names the evaluator fabricated specifically to test generalization: names built to mimic the same “popular library + generic suffix” pattern hallucinated by AI coding agents, each confirmed to 404 on its registry before being added, so none were accidentally real.
The result: all 65 hallucinated or bad names were flagged (100% recall, 0 false negatives) — including all 25 names the detector had never seen, which is the generalization signal that matters more than the headline number — and none of the 70 real packages, including 4 real near-miss edge cases, were wrongly flagged (0% false-positive rate).
Narrow claim. This measures detection of AI-hallucinated, slopsquatted, and typosquatted package names — whether a dependency exists, or is a near-miss of a real one. It says nothing about SAST quality, vulnerability detection, or any other part of Graneth's scanning pipeline.
Sample, not exhaustion. 135 labelled packages is a sample, not an exhaustive benchmark. A 100% result on this sample is not a guarantee against every hallucinated name that could ever appear.
“Unverified” is a real third state. On this run, 4 real edge-case packages returned “adoption unverified” because the registry's downloads endpoint was transiently unreachable — reported honestly as unverified, never silently counted as a false positive or a false pass. The number that comes back unverified varies run to run with transient network conditions; it is not a fixed property of the detector.
Not independently reproducible today. The benchmark harness and dataset live in a private repository, so this isn't an open, run-it-yourself benchmark. What we disclose instead is the full methodology, the exact numbers, and the dataset composition and sources below, so the result is judgeable even though the harness isn't public.
No guarantees. “100% recall” describes this measured sample on this date — not a promise that Graneth catches every hallucinated package name, in every case, forever.