It's not CPU based... What do you mean?

Yeah thats true, chip to chip communication and KV cache management is make or break for an AI ASIC.

Yup, consider contributing to te project ;)

Cool!

It's at 80~90% the max perf it can achieve on this hardware... Which is DDR3 speeds. I'm working on getting FPGA's with DDR4 and HBM2 next.

Why is there any point in trying with faster memory if you are only at 80% perf on DDR3? You'll just go to 40% perf on the next iteration?

You should only upgrade when the bottleneck becomes memory, right now the bottleneck is still on compute until you are at 100% max perf

Indeed there are some bugs/"non standard behavior" regarding very small or very big floating point values. All of those where proven harmless for LLM inference. Thanks for pointing out the bug. If you caught something outside of that, please point that out so I can fix it ;)

> All of those where proven harmless for LLM inference.

Could you share link(s) to those proof(s)?

Yup, the tests already compare to a CPU hugginsface implementation, but I'm working on making running and verifying the results of those tests easier and more available. Also, working on fixing those bugs and make the code more reliable now that the underling hardware has been saturated (memory bound)

First opensource arch that can do modern LLMs, while maximizing the potential of its hardware; First opensource TPU build by a recursive improvement loop...

It's upcoming second generation could run the inference of the models that are being used to improve it...

It could have a small improvement on power consumption, but the current design can already achieve 90% of the maximum theoretical speed of this hardware without giving up programability/flexibility

They are more like a way to proving the architecture of the accelerator before committing 100's of millions into a custom ASIC with TSMC

GPUs are faster, but you can't make your own arch on GPUs. FPGAs offer you that possibility. Said that... There are a few beasty FPGAs used in crypto mining coming my way... I expect that OpenTPU will be able to run frontier models with those.