RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts
In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100$\times$ longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions.
ProofPaper ↗
Key points
- The genome holds the blueprint that governs the biological properties of the cell.
- The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data.
- However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks.
- Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants.
Sources (1)
- [1]RAGenome: Scaling Retrieval-Based Genomic Language Models to Long ContextsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 11:44 AM
In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100$\times$ longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions.
The genome holds the blueprint that governs the biological properties of the cell.
Extractive summary: sentences quoted from the sources.