In the Shazeer 2019 paper "Fast Transformer Decoding: One Write-Head is All You Need", section 2.4 there's the MultiheadSelfAttentionIncremental function
The calculation of new_K (and new_V) mentions M, which isn't defined. This is a typo, s/M/x/, correct?
As a follow-up on my blog post about reverse mode automatic differentiation from earlier this week (eli.thegreenplace.net/2025/r…), here's a new project:
github.com/eliben/radgrad - an educational implementation of Autograd-like reverse mode AD via tracing
Part 5 of my Raft blog series is out. This one is about fixing linearizability issues that arise from client retries and achieving exactly-once delivery
eli.thegreenplace.net/2024/i…
This concludes the series! Thanks for reading
A post I wanted to write for a while now:
How to do ML inference in Go through a Python sidecar server
eli.thegreenplace.net/2024/m…
After a brief hiatus of 4.5 years, I've finally published Part 4 of my "Implementing the Raft consensus algorithm in Go" series. In this part, we build a simple but strict serializable key/value database on top of Raft.
eli.thegreenplace.net/2024/i…
Cute weekend toy: eliben.github.io/lsystem/
Simulation for L-systems in JS: generating arbitrary recursive fractals from a formal context-free grammar
TIL that pycparser [github.com/eliben/pycparser] gets ~20M downloads / month from PyPI. According to Github it's depended upon by 87K packages. 😳😳
