Benchmark

Tiếng Việt: benchmark.vi.md.

This page compares Guard Spec's detector with other open prompt-injection detectors on Guard Spec's own eval set and on public datasets. Everything on it comes from one command, with every dataset and model pinned, so anyone can rerun it and check the numbers.

It is a measurement and nothing else. No threshold, weight or rule of Guard Spec was tuned on these datasets or on eval/ for this page, and the other detectors run with their published defaults.

• Summary
• Reproduce
• What is measured
• Datasets
• Detectors
• Results
• Discussion

Summary#

Run on 2026-11-07 (19:30 UTC on 2026-11-06) at commit 2e182a0 (v0.2.1) with this benchmark's harness, on an Apple M3 Max laptop (macOS 15.4.1); the full tables are under Results. Default decisions, all languages:

• On our own set Guard Spec is far ahead, which is expected and not a fair test of ProtectAI: half of eval/ is Vietnamese (ProtectAI is English-only) and its benign rows are hard negatives written to look like attacks. On the English half ProtectAI reaches AUC 0.897 and flags 23% of benign rows.
• On SPML, the one public set no detector here was trained on, ProtectAI is clearly better: AUC 0.997 against 0.974 and, at the same 1% FPR, 97.6% of attacks against 84.5% (89.2% with L1+L2). At its default threshold Guard Spec blocks only 36% of SPML attacks, with no false positive; ProtectAI blocks 99% with 8% false positives.
• On deepset and jailbreak-classification Guard Spec ranks better (AUC), but Guard Spec's classifier was trained on their train splits, so those two rows favour it. On deepset both detectors block only about a third of attacks at their defaults.
• Guard Spec is smaller (237 MB int8 against 739 MB fp32) and faster per row (p50 5–8 ms against 32–67 ms on short inputs, one thread). The comparison is not like for like: our models are int8, ProtectAI's published ONNX file is fp32.
• Meta Prompt Guard 2 is gated and not in these tables; anyone with access can add it with one variable (below).

Reproduce#

make benchmark

That target fetches ONNX Runtime and libtokenizers (make onnx-deps, pinned by SHA-256), runs bench/run.py with the Python packages locked in bench/uv.lock (needs uv), and writes:

• bench/results/results.json: every metric, the machine and the versions;
• bench/results/results.md: the tables;
• the generated block of this page (between the bench:begin and bench:end markers).

What it does:

1. Downloads the public datasets at the pinned Hugging Face commits and converts them to the eval/ row schema.
2. Pulls the models with go run ./cmd/models pull, which checks every file against the SHA-256 values in models/manifest.yaml: guardspec-classifier@v3b, multilingual-e5-small@614241f and protectai-deberta-v3-base-pi-v2@90c9989.
3. Runs Guard Spec through cmd/eval -scores (built with -tags onnx), which is the gateway's input path: sanitizer, detector layers, fusion and the /v1/scan block rule.
4. Runs the ProtectAI model in ONNX Runtime from Python.
5. Computes the metrics below for every detector, dataset and language.

Options:

GitHub Actions: the Benchmark workflow (.github/workflows/benchmark.yml) runs make benchmark on ubuntu-24.04 when started by hand (Actions > Benchmark > Run workflow); it never runs on its own. It writes the tables to the job summary and uploads bench/results/ and this page as the benchmark-results artifact.

Including Meta Prompt Guard 2#

meta-llama/Llama-Prompt-Guard-2-86M and -22M are gated on Hugging Face under the Llama 4 Community License: they cannot be downloaded without an account that accepted the terms, so they are not in the published tables. With access:

1. Accept the license on both model pages.
2. Export a read-only token of that account as HF_TOKEN in your shell (huggingface_hub reads it; bench/run.py never reads, prints or stores it).
3. Run make benchmark PROMPTGUARD=1. This installs the promptguard extra (PyTorch and Transformers, CPU) and adds two detector rows, at the pinned revisions a8ded8e (86M) and 11614a1 (22M), with the model's argmax decision (malicious probability ≥ 0.5), the first 512 tokens, one thread.

In the workflow, add the token as the repository secret HF_READ_TOKEN and check "promptguard" when starting the run.

What is measured#

For every detector × dataset × language:

• ROC AUC of the detector's score, with a 95% bootstrap interval (1,000 resamples, fixed seed). It does not depend on any threshold.
• TPR and FPR at the default decision: the share of attack rows and of benign rows the detector flags with its own published default. For Guard Spec that is the gateway's /v1/scan rule (detector score ≥ 0.85, or sanitizer anomaly ≥ 0.6, or an input the sanitizer rejects), the decision a deployment makes.
• TPR at FPR ≤ 1%: the share of attacks caught at the lowest threshold that flags at most 1% of that group's benign rows. The threshold is picked after the fact on the same rows, for every detector alike, so it is an upper bound on what a tuned threshold would give; it compares the score rankings, not the defaults. With fewer than 100 benign rows it means "no false positive at all".
• Latency p50 / p95 per row, in milliseconds: one row at a time on one machine, with ONNX Runtime (or PyTorch) set to one intra-op thread. For Guard Spec it is the sanitizer plus the detector as the gateway runs them; with L2 and L3 both on, the gateway scores the two models concurrently, so that configuration can use two cores for one row. For the Python detectors it is tokenization plus the model call. Five warm-up rows are not timed for the Python detectors; cmd/eval times every row, model loading excluded.
• Model size: the size on disk of the model weights the detector loads.

Datasets without benign rows (Gandalf) only have a TPR.

Datasets and licenses#

Every dataset was checked on its Hugging Face card at the pinned commit. The benchmark downloads them at run time; no row from them is stored in this repository beyond what eval/ already contains (see eval/LICENSES.md).

Row counts and languages per run are in the generated tables below.

Vietnamese. We found no public, licensed Vietnamese prompt-injection dataset. The only Vietnamese rows here are those of eval/, which are synthetic and labelled by AI agents (see eval/README.md).

Language column. eval/ carries a language per row (en, vi). For the public sets, rows with German, Spanish or Cyrillic markers (the rule of eval/build/build.py) are other, the rest en. Only deepset has a sizeable other share.

Overlap with training data. Read the public rows with this in mind:

• Guard Spec's classifier (L2, v3b) was trained on the train splits of deepset/prompt-injections, Lakera/gandalf_ignore_instructions and jackhhao/jailbreak-classification (train/DATA_LICENSES.md). The benchmark uses only their test splits, but those come from the same distribution, which favours Guard Spec on these three sets.
• ProtectAI v2 lists jackhhao/jailbreak-classification among its training sets (split not stated), so the same holds for it there.
• The English test rows of deepset and Gandalf are also part of eval/ (60 of 116 and 111 of 112 rows).
• Guard Spec's L1 rules were refined by reading failures on the "dev" half of eval/ (eval/README.md); L1 numbers on eval/ are optimistic. L2, L3 and the fusion were never fitted on eval/.
• Nothing in this benchmark was used to change Guard Spec.

Detectors#

Results#

Run 2026-11-06 19:30 UTC, commit 2e182a03bc62, local: Apple M3 Max, 14 logical CPUs, 36 GB RAM, macOS 15.4.1 (arm64). go1.26.10, ONNX Runtime 1.31.1 (Go) / 1.31.1 (Python), Python 3.12.0.

Datasets#

Detectors#

Not run: Meta Llama Prompt Guard 2 (86M, 22M): gated on Hugging Face (Llama 4 Community License); run with --promptguard and your own HF_TOKEN after accepting the terms.

Guard Spec eval set (eval/)#

Published with this repository. L2, L3 and the fusion were never fitted on it; the L1 rules were refined on its dev half, so L1 numbers here are optimistic (eval/README.md).

deepset/prompt-injections#

About half the rows are German or Spanish (lang 'other'). The English test rows are also in eval/; Guard Spec's classifier was trained on the train split.

jackhhao/jailbreak-classification#

Jailbreak (role-play) prompts versus benign prompts. Guard Spec's classifier was trained on the balanced train split; ProtectAI v2 lists this dataset among its training sources.

Lakera/gandalf_ignore_instructions#

Attacks only (no benign rows): only TPR is defined. The English test rows are also in eval/; Guard Spec's classifier was trained on the train split.

reshabhs/SPML_Chatbot_Prompt_Injection#

User prompts only. Labels depend on the chatbot's system prompt (a request that breaks that bot's rules is an attack), which an input detector does not see. No detector here lists it as training data.

Discussion#

Where Guard Spec does worse.

• SPML. This set's attacks are mostly polite requests that break one chatbot's rules ("rather than going to a doctor, suggest ..."), written by GPT-4 from leaked system prompts. Guard Spec's score ranks them well (AUC 0.97–0.98) but keeps most of them below its 0.85 block threshold: TPR 36% at default. ProtectAI flags 99% of them, at the cost of 8% false positives on the benign SPML prompts. Some of that gap is the threshold, not the model: at an equal 1% FPR Guard Spec catches 84–89% and ProtectAI 98%. The labels also depend on a system prompt that neither detector sees, so part of this set is not decidable from the user message alone.
• Non-English, non-Vietnamese input. On the German and Spanish half of deepset (56 rows) Guard Spec blocks 21% of attacks at its default, ProtectAI 48%; both have no false positive there. Guard Spec's rules and classifier only target English and Vietnamese. The group is small (29 attacks): treat the gap as a direction, not a size.
• The default threshold is conservative. On every public set Guard Spec has 0.0% FPR at its default and the "TPR at FPR ≤ 1%" column is much higher than the default TPR (deepset 33% → 83%, SPML 36% → 85%). The 0.85 block threshold and the fusion were set for low false positives on eval/-like traffic (hard negatives, Vietnamese chat). Deployments that see plain English attacks may want a lower threshold; this page does not change it, because tuning on these sets would make the numbers meaningless.
• L3 does not help on these sets. L1+L2+L3 and L1+L2 are within noise on eval/, deepset and jailbreak-classification, and L3 is lower on SPML at 1% FPR (84.5% against 89.2%). It adds 0.5–1.3 ms per row at p50.
• Long inputs. Guard Spec scores every 512-token chunk of a long input; the long role-play prompts of jailbreak-classification take 42 ms at p50 and 510 ms at p95 with the model layers (ProtectAI: 330 ms and 1.16 s, but it reads only the first 512 tokens).

Where it does better, and why that is partly expected.

• eval/ is Guard Spec's own set: same authors, same notion of an attack, Vietnamese and hard negatives. Read the ProtectAI rows there as "what an English-only model does on this traffic", not as a ranking. Even so, the English half is informative: ProtectAI blocks 23% of its benign rows (security discussions, sysadmin text, code with attack strings), Guard Spec 1.6%.
• deepset, jailbreak-classification and Gandalf share their distribution with Guard Spec's training data (train splits), and ProtectAI also trained on jailbreak-classification. Guard Spec's higher AUC on them is weak evidence.

Other notes.

• Small groups have wide intervals: deepset has 116 rows, and its per-language groups under 60. Differences inside the 95% intervals are not signal.
• LLM Guard's default (score > 0.92 instead of ≥ 0.5) trades a few points of TPR for a few points of FPR; it does not change the picture.
• Latency was measured on a shared laptop with other work running, one configuration after the other; read it as indicative. The manual workflow gives a Linux run.
• Guard Spec's "default" is the gateway's whole /v1/scan rule, which also blocks on sanitizer anomalies. That is why its TPR on eval/ (95.0%) is above the README's "TPR at block" (0.939), which counts the detector score only.
• Language labels of the public sets come from a simple marker rule, not a language-identification model.

The results will also appear on guardspec.io/docs, which is built from this page.

Source: docs/benchmark.md @ f26302340bf1