So the main recent additions are efficiency tweaks. Like Mixture of Experts (but that's also a few years old) and Multi-head Latent Attention. There are a few ideas like Mamba, Samba etc. but they have not been scaled yet, so it's not yet clear if they can compete with SOTA LLMs

Thanks Sebastian, love your work! Link for others to your full talk LLMs Then and Now: sebastianraschka.com/pdf/sli…