I'm developing an inference engine and need access to the latest Neural Accelerator features, and therefore need to keep up to date with the Apple releases. M1 through M5 Max 128GB. As a broke bootstrapped un-funded startup founder this is extremely tough to afford given the price increases and all.
> "There is nothing happening between requests to the inference engine."
This is true for vLLM/SGLang/etc but LayerScale is making use of idle GPUs. A new novel inference engine for agentic workloads. Our advanced scheduler enables double the workload on the same number of GPUs (e.g. prioritization of on-demand queries and background processing of batch workloads simultaneously). Let me know if you want to try it out!
Damn. I just bought a maxed out MacBook Pro M5 Max 128GB 8TB, still waiting for it to be delivered. I could get 256GB RAM M5 Ultra 1TB for roughly the same price, and it's double the memory bandwidth. Which one would you recommend? I do plan to run local LLMs.
M5 Ultra if the portable form factor is not a requirement. (I have a M5 Max 128GB 2TB for context.)
That being said: you might want to consider at least 2TB storage. Having 256GB RAM to fit models is less useful when you can only store a 3 or 4 large models on your disk. The M4 -> M5 transition doubled the drive bandwidth (at least for the MacBook Pro, I'd assume the Studio is no different) making them extra nice for loading large models. Then again, you can always add an external NVMe drive over thunderbolt, so the tradeoff is less straightforward than with a portable MBP.
Contact Apple care and see what they can do. They have been known to do upgrades if you buy around the same time. But with the ram situation that courtesy might be gone now.
