아래로 당겨서 돌아가기
Best RTX Pro 6000 vllm settings?

Best RTX Pro 6000 vllm settings?

Best RTX Pro 6000 vllm settings?

Just got myself (for my company) a RTX Pro 6000 Blackwell Workstation card. Managed to get really good TPS on qwen3 27b fp8. Using it for many agents that specialize on one specific task at a time. Trying to get the best possible Speed + Concurrency running on vllm 0.20.1 nightly cuda 13.1. Engine 000: Avg prompt throughput: 763.5 tokens/s, Avg generation throughput: 1320.2 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.4%, Prefix cache hit rate: 1.3%, MM cache hit rate: 0.