Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?

Key points

• Prior work has explored large language models (LLMs) for automating this step, but LLMs are evolving rapidly, so earlier performance claims may no longer accurately reflect their screening performance.1
• We used an existing software engineering SR screening benchmark (SESR-Eval) as our data.1
• Using this data, we evaluated eight new LLMs for screening performance.1
• Differences between secondary studies are still bigger than between LLMs. Computing the overall screening decision from criterion-level decisions degraded screening performance only slightly.1

Sources (1)

• [1]Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 01:25 PM

  Screening in systematic reviews (SRs) is manual and time-consuming.

  Prior work has explored large language models (LLMs) for automating this step, but LLMs are evolving rapidly, so earlier performance claims may no longer accurately reflect their screening performance.

Extractive summary: sentences quoted from the sources.