TraceRelay: Attention-Aligned Recurrence over Rolling Traces
We present TraceRelay, an attention-aligned recurrent architecture that distributes persistent representations over a rolling sequence of low-dimensional traces.
ProofPaper ↗
Key points
- Local right looking attention forms increments from lower-layer representations; delivery is delayed until all attended inputs are in the causal past.
- A fixed additive phase recurrence accumulates the delayed increments, and left-looking attention reads the resulting residual augmented stream.
- A stride-wise prefix sum supports parallel prefill and bounded-buffer continuation.
- We study 36 small-model runs on Equal Repeats, bounded Dyck closing-type prediction, and causal Most-Freq generation, using three seeds per setting.
Sources (1)
- [1]TraceRelay: Attention-Aligned Recurrence over Rolling TracesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 11:34 AM
We present TraceRelay, an attention-aligned recurrent architecture that distributes persistent representations over a rolling sequence of low-dimensional traces.
Local right looking attention forms increments from lower-layer representations; delivery is delayed until all attended inputs are in the causal past.
Extractive summary: sentences quoted from the sources.