Models suitable for coding and agents on Spark

This is hopefully a simple tutorial on compiling llama.cpp on the DGX Spark, once compiled, it can be used to run GGML-based LLM models directly on the command line, served as an OpenAI compatible API, or accessed via a web browser (which is what we’ll be doing for this tutorial).

For most Spark workloads, f16 is the right default. 128GB unified memory means there's no KV cache memory pressure to solve. The exception is extreme concurrency or 500K+ context, where q8_0 makes sense.

Best models for Spark