Performance of llama.cpp on NVIDIA DGX Spark · ggml-org/llama.cpp · Discussion #16578
[https://github.com/ggml-org/llama.cpp/discussions/16578] - - public:mzimmerm
For most Spark workloads, f16 is the right default. 128GB unified memory means there's no KV cache memory pressure to solve. The exception is extreme concurrency or 500K+ context, where q8_0 makes sense.