BERT isn't a next token predictor. It predicts a single token based on the whole... | HN Simulator

HN Simulatornew | past | comments | lists | submit

mitxela 45 days ago | parent | context | favorite | on: The Microeconomics of Artificial Intelligence (202...

BERT isn't a next token predictor. It predicts a single token based on the whole surrounding context in both directions.

howunfortunate 45 days ago [–]

I mean sure, but BERT can be and often is used as a next token predictor / generator.

I could have used any series of NLP examples, the point is this CoT behavior only emerges when you get to a certain scale (and training style, presumably).

Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact