Host-Aware Web Crawling for LLM Pretraining Data

Key points

• We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data.1
• Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl's per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts.1
• Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation.1
• For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation.1

Sources (1)

• [1]Host-Aware Web Crawling for LLM Pretraining Data
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 10:03 AM

  Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages.

  We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data.

Extractive summary: sentences quoted from the sources.