I worked on a system a couple years ago with a BERT-based model (64M parameters) used for classification. The rest of the system could process data at gigabytes per second, and so here tokenization at a measly few megabytes per second really slowed things down. The model inference was more expensive than tokenization, but tokenization was still >10% of total runtime.

It's not just cosmic radiation or improperly built computers that result in bit flips. Row hammer is a thing :)

anyway i was completely wrong and a few nodes had EDAC events haha