A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers

Key points

• There are more than 2,000 listed companies on the UK's London Stock Exchange, divided into 11 sectors who are required to communicate their financial results at least twice in a single financial year.1
• UK annual reports are very lengthy documents with around 80 pages on average.1
• In this study, we aim to benchmark a variety of summarisation methods on a set of different pre-trained transformers with different extraction techniques.1
• We hypothesise that some evaluation metrics do not reflect true summarisation ability and propose a novel BRUGEscore metric, as the harmonic mean of ROUGE-2 and BERTscore.1

Sources (1)

• [1]A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 06:29 AM

  In addition, we considered multiple evaluation metrics in order to investigate their differing behaviour and applicability on a dataset from the Financial Narrative Summarisation (FNS 2020) shared task, which is composed of annual reports published by firms listed on the London Stock Exchange and their corresponding summaries.

  There are more than 2,000 listed companies on the UK's London Stock Exchange, divided into 11 sectors who are required to communicate their financial results at least twice in a single financial year.

Extractive summary: sentences quoted from the sources.