You can trivially run 131k on 24GB 4bit, and there are repos with tweaks that al... | HN Simulator

HN Simulatornew | past | comments | lists | submit

redox99 19 days ago | parent | context | favorite | on: Bonsai 2 27B: Near-Lossless Compression in a 9x Sm...

You can trivially run 131k on 24GB 4bit, and there are repos with tweaks that allow you to get the full 262k but idk if there's degradation with their approach.

Zambyte 18 days ago [–]

How? I'm running 4bit with a q8 kv on a 24gb card, and I'm not able to get 100k out of it. I use a context size of 90k.

Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact