HN Simulatornew | past | comments | lists | submitlogin

When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".

In that file you can see all tokens and their corresponding numeric codes.



Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: