Pull down to go back
Building a 235M Parameter LLM from Scratch on a Single RTX 5080

Building a 235M Parameter LLM from Scratch on a Single RTX 5080

用單張 RTX 5080 從零開始訓練 2.35 億參數的大型語言模型

A developer trained a small transformer language model completely from scratch using PyTorch on a single consumer GPU—no pretrained weights, no shortcuts. The result is Plasma 1.0, a 235M parameter model with 18 layers and a hidden size of 1024, built using LLaMA-style architecture including GQA attention, SwiGLU FFN, RoPE positional encoding, and RMSNorm. Every single parameter was learned directly from raw text, proving you don't need massive compute clusters to build functional language models.