Low accuracy (~50%) with SSL (BYOL/MAE/VICReg) on hyperspectral crop stress data — what am I missing? [R]

I’m working on a hyperspectral dataset of cabbage crops for nitrogen deficiency detection. The dataset has 3 classes: Healthy Mild nitrogen stress Severe nitrogen stress I’m trying to use self-supervised learning (SSL) for representation learning and then fine-tune for classification. What I’ve done: Tried multiple SSL methods: BYOL, MAE, VICReg Used data augmentation (spectral noise, masking, scaling, etc.) Fine-tuned with a classifier head Evaluated using accuracy and F1-score Proble