Learning from Failures: A Failure-Driven Prompt Refinement for LLM-Based Vulnerability Analysis

Key points

• Large Language Models have emerged as promising tools for software vulnerability analysis, but their effectiveness depends heavily on prompt design.1
• Using the Damn Vulnerable Java Application (DVJA), we identify recurring failure modes, including false positives, false negatives, unsupported reasoning, and CWE misclassification, and translate them into targeted prompt refinements.1
• The results show that failure-driven refinement improves the reliability of LLM-based vulnerability analysis while yielding reusable prompt design principles.1
• More broadly, this work demonstrates that recurring model failures provide a principled foundation for prompt engineering, enabling the systematic development of more reliable LLM-based vulnerability analysis systems.1

Sources (1)

• [1]Learning from Failures: A Failure-Driven Prompt Refinement for LLM-Based Vulnerability Analysis
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 02:15 PM

  We propose Failure-Driven Prompt Refinement (FDPR), a methodology that analyzes recurring model failures to guide evidence-based prompt refinement.

  Large Language Models have emerged as promising tools for software vulnerability analysis, but their effectiveness depends heavily on prompt design.

Extractive summary: sentences quoted from the sources.