Local LLMs

Cloud models are fine for Rivet day to day. I wanted local capacity for embeddings, batch evals, and experiments that do not meter every token. pve3, GERTY, has two Tesla V100 32GB cards. That is enough VRAM for a 40B if you pick the right quant.

vLLM, 1Cat-vLLM fork. KV-cache config, compressed-tensors weirdness on Volta, some FP8/MTP, community Deckard-40B checkpoints. The BF16 base on disk and the quantized artifact are not the same animal. Full FP8 on a V100 is usually the wrong bet versus W4A16 plus speculative decoding.

The point is agents. Tool-calling harnesses write the train and eval data, so the scaffold has to be honest before SFT. Rivet Local, CT114, runs those sessions on local models and smaller Qwen graders. Results go under /rivet-shared/deckard-eval/ so we can compare later.

Local embedder offload into RivetOS memory. MTP speculative paths on 4-bit weights. Tighter eval loops before any fine-tune. Better numbers on the cards I already own beats downloading another 70B.