Modular Gated Attention
I built a PyTorch sequence layer that learns how to mix local attention, global latent context, and recurrent memory. Controlled tasks let me inspect when those mechanisms help—and when they don’t.
- Token-wise gates over three complementary information paths.
- Parameter-matched Transformer comparisons on copy and palindrome tasks.
- Length extrapolation, distractor noise, gate diagnostics, and module ablations.




