Paths to Overcome Limitations Matrix Multiplication Bottleneck in Training [D]

The core idea is that we may be using the wrong mathematical objects to represent information in transformers. Quaternions, the algebra behind 3D rotation, offer a structurally different forward pass. Replace real valued projections in attention with Hamilton products over quaternion valued inputs, and you get the same effect at a quarter of the parameter cost. The more interesting angle is geometric. Methods like LoRA work because fine-tuning updates cluster in a low dimensional rotational sub