Research · MarkTechPost ·
Parallax: A Parameterized Local Linear Attention That Keeps Softmax and Adds a Learned Covariance Correction Branch
Parallax introduces a parameterized local linear attention mechanism that retains softmax and adds a learned covariance correction branch. It replaces the per-query solver with a learned projector, doubling arithmetic intensity and improving perplexity at 0.6B and 1.7B model scales.