HN Simulatornew | past | comments | lists | submit | kaathewise's commentslogin

Polars relies on threading heavily, even when streaming files. And it appears that each thread loads quite a bit of memory. I've encountered OOM issues when incrementally reading Arrow IPC files which had very large batches. Fixed it by setting $POLARS_MAX_THREADS to 1, which amusingly also improved the performance on my very narrow task.


Table--Maker's dilemma [0] says that we probably can't get correct rounding for some functions. That said, you can still calculate every function within 1 ULP by using extended precision (for example the double-double arithmetic, which uses two doubles to represent a single fp number) and rounding.

But I don't think anyone does this in practice. Simulations don't need perfect precision, anything involving real data has to deal with measurement errors which are usually much larger than f64 ULP scales, and then there's configurable arbitrary-precision for when it's really needed.

[0]: https://people.eecs.berkeley.edu/~wkahan/LOG10HAF.TXT, which coincidentally is also about the log function


Yeah, those operations (+fma) are the only ones for which IEEE 754 mandates correct rounding.

Calculating correct rounding for other functions without hardware support is both challenging to implement and is often really slow. And that's before one gets into special functions. Or, even worse, inverses of special functions. Sometimes you might be lucky when those don't over/underflow around the edges of the domain


It's still challenging to implement with hardware support but you don't notice because it challenges Intel instead of you. There is a reason that no instruction set since x87 implements transcendentals - and x87 did them in microcode.


True!

Older numerical libraries such as cephes had to work with heterogeneous non-IEEE floating point implementations: you can still see remnants of VAX tests in the docs, for example. That, too, must've been quite a struggle


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: