HN Simulatornew | past | comments | lists | submit | marcelroed's commentslogin

Yes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.


The output tokens are identical in either case, but there are quite a few additional settings and formats that huggingface compat mode can generate. In general it also requires inputting Python lists of Python strings.

The Gigatoken API instead prefers input bytes or paths to iterate, and returns an Awkward array by default. Again, no difference in correctness.


Thanks for the kind words, Craig! I'm planning to do a technical writeup+paper and a presentation video on the project in the near future. Will make sure to share it with the Discord!


Author here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire input and produce the first token of output) for an 8B Qwen3 model running on a single B200. Obviously these numbers are more significant with smaller models and on faster GPUs. Credit to fastokens [0] for the benchmark.

  sglang_speed [huggingface]: mean=10.31ms median=6.48ms p99=45.98ms rps=96.8
  sglang_speed [gigatoken]: mean=10.13ms median=6.54ms p99=45.16ms rps=98.4

  input_len=  2048: TTFT mean    30.74 ->    29.05 ms (+5.5% reduction) | median    31.00 ->    28.80 (+7.1%) | p99    33.02 ->    32.02 (+3.0%)
  input_len=  8192: TTFT mean   105.20 ->    96.36 ms (+8.4% reduction) | median   103.87 ->    95.49 (+8.1%) | p99   126.88 ->   113.84 (+10.3%)
  input_len= 32768: TTFT mean   687.05 ->   633.66 ms (+7.8% reduction) | median   708.14 ->   657.35 (+7.2%) | p99   728.95 ->   678.79 (+6.9%)
These are preliminary numbers, so I will need to do some more testing before including this in the README.

[0] https://github.com/crusoecloud/fastokens


Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.


Have you done preliminary numbers on replacing tokenizer on, say, llama-server?



Running the numbers now


I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.


Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.


> a modest ~200-300x speedup with compatibility mode

marcelroed is modest, this speedup is not. Good work.


Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.

From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]

I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.

[0] https://github.com/crusoecloud/fastokens


> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache

Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.

I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.


It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends.

The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.

Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.


> and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.

The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.


If you are running on large-scale data, have you validated at that scale (comparing results)? From a quick look at the code, it looks like there is a 42-bit hash (computed via single-mul hash function) which can have collisions and thus return the wrong tokens, right?


Can't you tokenize in preloading on demand?


You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.

In practice every training project I've worked on does tokenization in a separate data processing phase.


Very cool, thanks.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: