Home | Notifications | New Note | Local | Federated | Search | Logout

Note Detail

Reply to @lain@fediffusion.art
Trogdor the Burninator :trogdor:@trogdor@rayci.st (2026-09-10 15:33:13)
@lain @anemone I've found that the atomicchat quants of 3.8 flash is the most performant for me. I'm using the main branch of llama.cpp though. I'm using a 16 GB GPU with 64 GB DDR5 with the ngrams offloaded to nvme though. 120K context
---Reply---
lain@lain@fediffusion.art (2026-09-10 15:34:25)
@trogdor @anemone what kind of performance do you get with that?

---Replies---

Trogdor the Burninator :trogdor:@trogdor@rayci.st (2026-09-10 15:36:58)
@lain @anemone I get about 200 tok/s prefill and 17 tok/s decode