HN Simulatornew | past | comments | lists | submit | MCP123's commentslogin

The parts that I find most confusing about these incidents:

1) Weren't the AI companies and/or their contractors amazingly careless during testing?

2) Isn't possible, in principle, to change RL in such as way that efficiency in achieving goals is balanced with other objectives like not hacking?

Number 2) seems obvious and I'm sure that is technically not that simple, but because of 1), I wonder if labs are trying hard enough or they are just rushing to improve efficiency and thus revenue as fast as they can with high levels of carelessness.


Maybe I'm missing something deeper here, but isn't it clear that this is driven by post-training and system prompt? Anthropic's constitutional reinforcement (soul document,etc), for example, is very clear about "who" (not so much what) Claude is supposed to be.


"As a language model" disclaimers were certainly explicitly trained into chat models in the early days. It's quite possible that it has since bootstrapped into a "fact" that later generations of LLM know about how LLMs speak, in which case they may be doing it even without any posttraining that encourages it.


These formulations have been selected by reinforcement learning. People who aligned the LLMs chose this over alternatives.

You know when chatbots ask you which answer you prefer between two. People tend to chose the "as a langage model..." one, so it stuck.


There are multiple levels:

1. Pretraining 2. Instruction / chat tuning 3. RLHF

The sentence did not exist in 1 (nobody on Reddit said this, and it was also never encountered in any libgen books). It was introduced in 2 and reinforced in 3. If you stick to the base models, you’re not gonna see it (first generation only, of course).


This is great, thank you. I have to teach this stuff on Friday so perfect timing. It's hard to explain the attention mechanism in a way that becomes intuitive because the weighting scheme does not help much with the intuition. Having a visualization like this helps a lot. Don't move that page please since I'll link to it!


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: