Benchmark

Same model, same database, different context strategy.

Qwen3-32B (vLLM)Sets S21 · Q333 · ABS · TND

Tool sufficiency

354/354

Engine answers the graph tools reproduce

Conflict F1, JSON

—

– planted events

Conflict F1, PDF

—

– rendered PDFs, end to end

Entity resolution

—

PDF subjects with varied titles

Loading results

Limitations

Read these before quoting a number

S21 is the only independent gold, and it has 21 questions: a one-question swing moves accuracy by about 5 points.
Q333 gold comes from the same deterministic engine whose logic the BidBrain graph tools share, so it favours tool-based systems. It is near-gold, not ground truth.
ABS and TND are synthesized by us from the graph; they test coverage of absence and tender reasoning, not generalisation.
All systems use the same model and database; latency is wall time on one shared vLLM server and varies with load.
Exact match is within 0.5%; proportional credit is the dataset's official partial score.