Language-model-based compression for Python source using n-grams + arithmetic coding (~33% better than zlib on Flask) [P]

I’ve been experimenting with language-model-based compression for source code, using a simple n-gram model combined with arithmetic coding. ill make a repo soo... The setup is straightforward: tokenize Python source, estimate P(xt∣xt−n+1:t−1)P(x_t \mid x_{t-n+1:t-1})P(xt​∣xt−n+1:t−1​) using an order-4 n-gram model, and feed those probabilities into an arithmetic coder. The coder converts the predicted distribution into a bitstream, so compression performance is directly tied to how well the mo