Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0.
ProofPaper ↗
Key points
- We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench's published executions.
- In SemBench's published results, 21% of text results change quality between executions, including on engines whose plans are deterministic.
- Changing rows are rows on which models disagree with each other; identical requests within one execution disagree as often as across executions; and one execution's decision margin or reasoning length predicts which rows change.
- We derive reporting rules for benchmarks and design rules for engines.
Sources (1)
- [1]Same Query, Different Answer: Run-to-Run Variance in Semantic Query ProcessingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 04:34 PM
Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0.
We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench's published executions.
Extractive summary: sentences quoted from the sources.