Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness
We present Goldsmith, an agentic pipeline that turns a small gold set---expert-annotated calibration examples representing the intended task boundaries---into a reusable structured annotation definition.
ProofPaper ↗
Key points
- Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model.
- Goldsmith treats this definition as a trainable textual object.
- In prompt-optimization comparisons, Goldsmith improves over direct rewriting, OPRO, APE, and PromptBreeder under matched evaluation protocols.
- These results show that scarce expert supervision can support both task-definition learning and scalable annotation.
Sources (1)
- [1]Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation HarnessarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 05:45 AM
We present Goldsmith, an agentic pipeline that turns a small gold set---expert-annotated calibration examples representing the intended task boundaries---into a reusable structured annotation definition.
Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model.
Extractive summary: sentences quoted from the sources.