Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation

Key points

• A crucial component of this pipeline is specification generation (SpecGen), which produces a formal contract against which an agent can prove implementation correctness.1
• Proof generation can obtain deterministic feedback from a theorem prover, but SpecGen lacks a definitive check that a generated specification captures the user's intent.1
• We take a step towards holistic SpecGen evaluation with a unified dataset assembled from $350$ existing Lean tasks, including $189$ from VERINA and $161$ from CLEVER, and a framework covering formal validity, reference similarity and equivalence, and behavioural adequacy.1
• Across four SpecGen configurations, restricting the generalized tree edit distance (GTED) comparison, a reference similarity measure, to $32$ jointly measurable VERINA tasks changes the VERINA configuration's position from second to fourth in mean similarity, showing the importance of measurement coverage.1

Sources (1)

• [1]Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 02:11 AM

  When generating verifiable code, natural language requirements are mapped to machine checked code using LLMs and agentic workflows.

  A crucial component of this pipeline is specification generation (SpecGen), which produces a formal contract against which an agent can prove implementation correctness.

Extractive summary: sentences quoted from the sources.