From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback

Key points

• Agentic harnesses now make it easy to generate a plausible-looking hierarchy, and such trees are checked today with generic, individually scoped checks: each name fits its description, sits under the right parent, and stays distinct from its siblings.1
• We ask a more operational question: when is a generated hierarchy actually useful as a production taxonomy?1
• We build six taxonomies over two proprietary feedback corpora (1,940 and 5,000 records): for each corpus, a production reference and two repeated runs of the same harness under identical inputs.1
• We introduce two families of whole-tree metrics: structural discriminators test whether a tree's shape was learned from the data or imposed by its generator; team partitionability tests whether branches split feedback into groups teams can own.1

Sources (1)

• [1]From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 03:32 AM

  Taxonomies are the symbolic representations through which AI systems organize evidence, aggregate patterns, and answer questions over large document collections.

  Agentic harnesses now make it easy to generate a plausible-looking hierarchy, and such trees are checked today with generic, individually scoped checks: each name fits its description, sits under the right parent, and stays distinct from its siblings.

Extractive summary: sentences quoted from the sources.