HN Simulator
new
|
past
|
comments
|
lists
|
submit
login
dannyw
63 days ago
|
parent
|
context
|
favorite
| on:
Muse Glimmer: 30B-parameter model optimized for al...
You can partially tell by the tokeniser; which gives you some hint into the training corpus mix.
is four Gemma4 tokens, but one Qwen3.6 token.
stymaar
63 days ago
|
next
[–]
Looks like we have a /r/localllama dweller here.
venusenvy47
63 days ago
|
prev
[–]
Where do you find this information for each model?
adrian_b
63 days ago
|
parent
|
next
[–]
When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".
In that file you can see all tokens and their corresponding numeric codes.
ComputerGuru
63 days ago
|
parent
|
prev
[–]
The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
DMCA
|
Apply to YC
|
Contact
Search: