My calculator is a transformer

I got interested in seeing whether I could "compile" a program into transformer weights, instead of training. I've been working on it for a couple of months now but finally decided to just stop and write it up, so this is a bit of a long post but maybe some of you will find it interesting. Basically I define the residual stream as a set of "registers" and generate the attention weights and MLP functions that execute an RPN interpreter (e.g. 2 3 + 2 * should produce 10.) For now I settled on di