Pull down to go back
Run CUDA + ROCm simultaneously with -DGGML_BACKEND_DL=ON!

Run CUDA + ROCm simultaneously with -DGGML_BACKEND_DL=ON!

用 -DGGML_BACKEND_DL=ON 同時運行 CUDA + ROCm!

After significant effort, I've finally cracked running large language models like Minimax 2.7 Q4 using both CUDA and ROCm at the same time, completely bypassing Vulkan. The setup offloads all 63 layers to GPU with CUDA handling 83.6GB, ROCm handling 40.3GB, and CPU handling the rest. The real win here is dramatically faster prefill performance. If you're running on Windows, the build process involves clearing your build directory and using CMake with Ninja generator pointing to your AMD ROC installation.