Research
Papers and datasets worth knowing, ranked by significance and community attention.
PaperBeyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review
Here, we introduce a verification-centric perspective on LLM-assisted peer review, emphasizing error detection as a critical and resource-intensive task.
Reasoning-Informed Visual Editing
To study this gap, we introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE), and extend it to RISEBench++, a more comprehensive and fine-grained benchmark for this emerging task.
SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models
Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input.
CARE: Certifying Acceleration for Vision-Language-Action Inference
Prior work accelerates VLA inference using techniques like action chunking and visual-token pruning, typically evaluating based on latency and average task success.
Agent Plasticity: Measuring Self-Improvement Through Experience
We introduce agent plasticity, the efficiency with which an agent converts experience into gains in future held-out performance.
MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent
To bridge this gap, we formulate text-to-music intent alignment as satisfying a per-request rubric of independently verifiable items covering both a request's explicit requirements and its implied musical intent.
TestPrism: Rethinking Test Evaluation Beyond a Single Reference
Large language model (LLM) coding agents have advanced test generation across diverse programming tasks.
Incidental information contaminates patient notes and disrupts clinical reasoning in large language models
We propose a dual encoding hypothesis of clinical reasoning and distraction in LLMs, with preliminary evidence that LLM components associated with disruption by incidental information also support clinical reasoning.
A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers
In addition, we considered multiple evaluation metrics in order to investigate their differing behaviour and applicability on a dataset from the Financial Narrative Summarisation (FNS 2020) shared task, which is composed of annual reports published by firms listed on the London Stock Exchange and their corresponding summaries.
Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair
We propose a Weighted Quality Index (QI), inspired by the ISO/IEC 25010 software quality model, that combines functional correctness, maintainability, security, and generation efficiency under configurable weighting schemes.
Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation
An answer judge instructed to grade the final commitment should reject an explicitly wrong final value even when an earlier value matches the reference.
Cross-Provider Review as a Runtime Contract for Coding Agents: A Controlled Pilot and Fault-Injection Study
We describe an advisory cross-provider review contract: distinct resource pools, bounded execution, restricted reviewer capabilities, complete input delivery, usable semantic output, explicit failure states and durable per-attempt evidence.
PatchBench: Measuring Collateral Damage in Activation Patching
To address this gap, we introduce PatchBench, a benchmark of empirically observed model-specific jailbreak failures inducing actionable harmful answers.
Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge
We evaluate what probabilistic evidence verification adds beyond number matching using Jev as a source-support verifier for GPT-4.1-mini calculation traces.
The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules
We treat this keep-if-better step as selection under measurement noise, model the correlated errors of the candidates in a single decision, and study empirically what happens when the evaluation set is reused.
Coding-Agent Benchmarks Should Match Their Users' Task Flows
We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories.
Leveraging a four-quadrant approach for evaluating Redpine Science
This report evaluates Redpine Science on two levels: the relevance of the retrieved chunks, and a model's answer when it has access to Redpine Science compared to web search.
Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging
Public leaderboards for AI models are read continuously, and attackers can see every published standing.
Holdout Best-of-N: Unbiased Evaluation and Its Cost
We study evaluation from a fixed matrix of $K$ independent scores per candidate for a policy that selects using $J$ fresh scores.
Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver
MemoryAgentBench's Conflict Resolution split is read as measuring "selective forgetting".
All Verdicts are Not Equal: Rethinking LLM Judge Reliability
LLM-as-a-Judge is the standard paradigm for NLP evaluation, yet its systemic reliability remains poorly understood despite being widely treated as a deterministic ground truth.
InterviewPlayground: A Simulation Environment for Evaluating AI Interviewers
To address this need, we develop InterviewPlayground, a simulation environment for evaluating AI interviewers using simulated study participants whose behaviors are grounded in social theory.
Evaluating Exact Output and Checkpoint-State Prediction in Real Programs
We present a benchmark for predicting final output and checkpoint state from source and input alone.
How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices
Automated research systems support literature synthesis, ideation, experiments, writing, and peer review, but their evaluation is dispersed across tasks, benchmarks, and studies that are difficult to compare directly.
In-Ride Alcohol-Impairment Detection in E-Scooterists with False-Alarm Control
This paper proposes a new approach in which onboard sensors evaluate the rider as the trip unfolds, raising an alarm as soon as enough evidence of impairment has accumulated.
Design Creativity Bench: Measuring creativity in LLM-Generated UI
Our work introduces Design Creativity Bench, a benchmark that evaluates diversity and appropriateness in UI designs.
RideBench: A Large-Scale Exogenous-Aware Benchmark for Ride-Hailing Time Series Forecasting
We release Ride-Hailing, a large-scale ride-hailing time series dataset synthesized from DiDi's marketplace data across 200 spatial areas.
How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys
Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists.
Stale, Misattributed, or Late: Where Personal Memory Fails Before Generation
Using Personal Fact Memory (PFM) as a reference layer, we find that temporal validity is primarily a property of memory construction in our setting.
LLM Persuasion Is in the Eye of the Evaluation
Large language models (LLMs) have already been shown to match or exceed human experts in persuasion.
Robust Decentralized Fairness Auditing
We propose Auditopus, a novel approach for robust decentralized fairness auditing.
Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?
Screening in systematic reviews (SRs) is manual and time-consuming.
InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain
We introduce InsClaimBench, an end-to-end benchmark for evaluating insurance claim adjudication across the decision chain.
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
We introduce TestJack, a scalable framework for evaluating patches beyond fixed tests.
Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth
We introduce a calibration-first framework that produces supervision scores without access to ground-truth labels or a shared annotation space.
Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions
We introduce Conditional Accuracy Profiling (CAP), a post-hoc diagnostic framework that decomposes pairwise LLM-judge accuracy into eight conditions organized into content sensitivity, robustness, and rationale quality.
Finding Blind Spots in AppWorld and WorkArena Task Verifiers
Execution-based task verifiers decide whether an agent succeeded.
Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance
We study this effect through a preprocessing-aware benchmark that evaluates 11 forecasting models across 16 reversible preprocessing pipelines on 29,000 M4 time series.
Epistemic Uncertainty-Aware Defect Detection for Quality Control in Medical Device Manufacturing
Objective: We investigate whether accounting for epistemic uncertainty can improve the reliability of automated defect detection in medical device manufacturing.
Graph-monotone entrywise guarantees for MLE and Rank Centrality on general comparison graphs
We prove that both the standard maximum likelihood estimator and Rank Centrality achieve a high-probability entrywise error rate of order $1/\sqrt{λ{\mathcal{D}}}$ up to logarithmic and dynamic-range factors, where $λ{\mathcal{D}}$ is the algebraic connectivity of the count-weighted observation graph.
Contextualization of Third-Party Cloud Security Findings
Finding severity is the main driver of how security teams prioritize remediation.
Stable Scores, Unstable Answers: Frame Phase and Option Order in Video Multiple-Choice Evaluation
Video-language models are ranked by multiple-choice accuracy on frames from a uniform grid.
When does conformal calibration need censoring weights? Cause-of-failure prediction sets under competing risks
Split conformal prediction sets for competing-risks labels at a fixed horizon require calibration labels that right censoring can leave unobserved.
Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations
Concept drift threatens production machine learning, yet the empirical behavior of multivariate two-sample drift detectors at scale remains under-characterized.
Vector Map Quality Metrics for Contextual Autonomous Driving Systems
Ensuring safety in autonomous driving requires continuous map maintenance supported by reliable quality indicators.
CHARTER: Auditing Reference Substitution in Hierarchical Compact-Evidence Evaluation for Computational Pathology
In hierarchical compact-evidence pipelines, candidate filtering introduces a strategy-specific candidate-conditioned prediction alongside the original full-bag prediction.