See section : Starting with llama.cpp
See section : Starting with llama.cpp
UI for chats - as advanced as any proffessional chat apps. Talks to model servers, http on the user chat end, openAI API on the model end.
Proxy to servers serving models and exposing them as OpenAI API
Unsloth Dynamic V3.1 (Nov 2026) is our latest quantization algorithm for Qwen3.8-27B, DeepSeek-V4.1-Flash-1017 and Qwen3.8-Flash-Next, resulting in 10-15% increased accuracy and 50% less token usage vs other quants of the same size. We're working on an even better version of Dynamic quants which will be applied to Qwen4 and all future models.
Site compare AI models sizes, quantization, requirements etc
Also A) DeepSeekFlash-4.1-550B-FP16-1TB is better at coding than Qwen-3.8-27B-Q??-S??. B) If using Strata: 1) SSD NVMe PCIe 4.0 2) Faster CPU matters!! C) 17:10 DeepSeek: Typical coding agent step: send/reads ~ 3,000 new tokens (needs 25s) , writes ~ 750 (needs 150s). The rest of tokens, ~120,000 tokens, is already cached. D) NOTE Qwen is 5 times FASTER at the above E) on 2026-10-07, llama.cpp added their own GPU cache for MoE models.
NOT much gain. Try looping model Dory, but only 2 loops. The rest is superficial. Probably will not fit 4GB card, but maybe some sibling would
Betelguese is Fork of llama.cpp uses NVM SSD instead of VRAM
Describes cace on llama.cpp for MoE models
Full-stack local AI, all on one harness/agent (uses Hermess agent) is defined as : 1) LLM with images 2) context 100K tokens 3) image generation 4) embedding - huge memory of AI 5) speech to text to understand speech 6) text to speech for answer as speech 7) video generation is NOT possible on 32GB
They claim to run on-device with 120B/145B params. They also have server (like ollama.cpp or strata)
GPU price history. RTX should be about CDN $400. 2026-11-11 it is $631. GeForce RTX 3060 12GB price history in Canada: current low $631 CAD, 12-month low $367 CAD, 12-month median $549 CAD.
Models suitable for coding and agents on Spark
This is hopefully a simple tutorial on compiling llama.cpp on the DGX Spark, once compiled, it can be used to run GGML-based LLM models directly on the command line, served as an OpenAI compatible API, or accessed via a web browser (which is what we’ll be doing for this tutorial).
For most Spark workloads, f16 is the right default. 128GB unified memory means there's no KV cache memory pressure to solve. The exception is extreme concurrency or 500K+ context, where q8_0 makes sense.
Best models for Spark
Max is slower than a DGX Spark for diffusion models. Those are compute bound and a Mac has like 60TFlops Fp16 while a Spark had 140. And it's x2 and x4 for Fp8 and Fp4 on the spark. For Mac it's unsure given that Fp32 and Fp16 go at the same speed.
instruct the AI coding agent to only use modern frameworks on the web
Buy 2TB NVM PCIe SSD. There are some listed at just $130-170 (???) but all sold out (2026-11-10) Only those costing >350 are available.
Buy gard disk: 12TB is sweet spot, avl for $414: WD Purple Pro 12TB 3.5 SATA 256MB Hard Drive $414.99 Online - Sold Out In Store - Sold Out
Guide to Inference - includes sizes of models, terms, etc
5 things to do in first 30 mins: 1) no phone 2) sunlight, ideally short walk 3) water with lemon 4) protein 5) regular wake up time
Emacs document reader, using dynamic modules in emacs.
k-means / e-means is a method of vector quantization. I assume this method is used in ai models quantization?
Trying to use whiteboard plugin : https://workspace.google.com/marketplace/app/whiteboard_for_google_meet/1005191159113