Monodratic: learned product-hash routing for sparse causal attention [R]

Hi everyone, I'm an independent researcher sharing Monodratic, a sparse causal-attention architecture with learned product-hash routing. The idea is that after RoPE, source blocks are assigned to bounded causal posting lists, while each query probes product addresses, reranks the returned candidates, selects a fixed number of remote source blocks, adds guaranteed local blocks, and then runs exact causal softmax over just those tokens. I implemented it as a stateless [batch, sequence, width] -> attention-delta mixer, so normalization, residual updates, feed-forward layers, and inference scheduling are left to the host model. What I found is that -learned routing with 2 selected remote blocks out of 5 eligible: 763/768 correct associative-recall answers across three seeds (99.35% mean, 98.05% minimum). -an equally wide untrained router: 425/768. Local-only attention: 151/768. -forcing the labelled target block while keeping the same maximum R2 attention budget recovered all five remaining errors, reaching 768/768. -sparse selected-set attention agreed with an independent dense selected-mask oracle to a maximum absolute error of 1.43e-6. -the packed CPU routing implementation showed a fitted timing exponent of 0.993 from 4,096 to 32,768 tokens under the fixed, balanced configuration. -all reported learned-route and scaling runs recorded zero posting overflow. The limitations are that the experiments are synthetic, the implementation is portable PyTorch rather than a fused kernel, and the report does not claim natural-language quality, asymptotic linear construction, or deployment speed. Paper: https://github.com/Misul-Computing/Monodratic/blob/main/output/pdf/monodratic_proof.pdf Code and reproduction: https://github.com/Misul-Computing/Monodratic I would particularly appreciate technical feedback on the routing construction, the controls, and what the strongest next evaluation should be. submitted by /u/dttdrv [link] [留言]