Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts

Key points

• Social media is a rich source of real-time public sentiment, but widely used sentiment analysis tools are often applied without understanding their limitations.1
• We measured agreement using two statistical measures: Cohen's kappa for pairwise comparisons and Fleiss' kappa for multiple raters.1
• The Twitter-roBERTa-base model showed the strongest alignment with human ratings, outperforming both bespoke sentiment tools and LLMs, particularly in distinguishing negative versus non-negative sentiment.1
• Our findings underscore that domain-specific fine-tuning remains crucial for reliable social media sentiment analysis, and human-centered evaluation remains essential for establishing gold-standard labels.1

Sources (1)

• [1]Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 04:11 PM

  In this study, we evaluate the inter-rater reliability of three bespoke sentiment analysis tools (TextBlob, VADER, and Twitter-roBERTa-base) and three large language models (LLMs: Qwen3-32B, GPT-OSS-120B, Llama-4-Maverick-17B) against six human raters across 100 tweets.

  Social media is a rich source of real-time public sentiment, but widely used sentiment analysis tools are often applied without understanding their limitations.

Extractive summary: sentences quoted from the sources.