SciExam for ENSO: Can AI Agents Build Climate Models?
The AI Science Exam for El Nino-Southern Oscillation (SciExam for ENSO) is a benchmark in which agents build low-order stochastic models of ENSO, the dominant mode of interannual climate variability, from real observations.
Key points
- Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid.
- Hidden graders then test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, and score a published model in the same way.
- Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting.
- SciExam for ENSO can thus evaluate agent research where no answer is known, and the results suggest that agents can already build competitive models whose structures bear on questions that scientists still debate.
Sources (1)
- [1]SciExam for ENSO: Can AI Agents Build Climate Models?arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 05:52 PM
The AI Science Exam for El Nino-Southern Oscillation (SciExam for ENSO) is a benchmark in which agents build low-order stochastic models of ENSO, the dominant mode of interannual climate variability, from real observations.
Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid.
Extractive summary: sentences quoted from the sources.