Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes
Large language models (LLMs) can expose memorized training sequences under prefix-based extraction: given a prefix from a training example, the model may assign high probability to the original continuation.
Key points
- This motivates examining whether contextual conditioning mitigates memorization or merely changes the set of memorized samples that become extractable.
- We investigate this issue through paired item-level measurements of probabilistic suffix extraction.
- For each prefix-suffix pair, we score the target suffix under an empty prompt and under retrieved contexts of varying relevance, across three open-weight instruction-tuned models.
- We find that context does not simply erase memorization.
Sources (1)
- [1]Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated PrefixesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 02:57 PM
Large language models (LLMs) can expose memorized training sequences under prefix-based extraction: given a prefix from a training example, the model may assign high probability to the original continuation.
This motivates examining whether contextual conditioning mitigates memorization or merely changes the set of memorized samples that become extractable.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 8, 2026Forms of LLM-Integrated Applications from LLM-Chats to Autonomous AI Agent System
- Oct 7, 2026RFChipAgent: Multi-Agentic AI Flow for Analog/RF Chip Design
- Oct 7, 2026Rethinking access control for RAG with Amazon Quick and Amazon Bedrock
- Oct 7, 2026unslothai/unsloth v0.1.904-beta: Train your own Decision model
- Oct 7, 2026EntroPrefill: Renyi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generation
- Oct 6, 2026EmbeddingGemma 2: an open, lightweight multimodal embedding model