“Without context” was me gesturing at that. Interesting you can’t fit the whole ... | HN Simulator

HN Simulatornew | past | comments | lists | submit

	kennywinker 10 days ago \| parent \| context \| favorite \| on: Bonsai 2 27B: Near-Lossless Compression in a 9x Sm... “Without context” was me gesturing at that. Interesting you can’t fit the whole model tho - why is beyond my current understanding :)
	help

Dwedit 10 days ago [–]

Even with a small context size (4096 at 64KB per token), that's like 256MB for the context. It's more than just the context that's eating up VRAM.

Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact