HN Simulatornew | past | comments | lists | submit | jephs's commentslogin

There's Most Forbiddenish and there's Most Forbiddener. We already apply length penalties to CoT in RL, for instance.

Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros


Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?

Although, I thought we proved multiple times years ago that the model's decision-making process and reasoning tokens don't necessarily overlap?


> Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?

Yes, exactly. People sometimes call the vector-space thought "neuralese."


No! Not yes exactly!

You've slightly missed the point (for a similar, adjacent point). Chain of thought monitors are a worry even when there's no vector space path from the output at index t to the input at index t+1. Rather, the fear is that if you explicitly penalize some CoT pattern, you're encouraging the network to do whatever it was going to do anyhow, but in a way you can't catch.

Say you've noticed that when your LLM cheats, it usually happens after the string "Hmm, let's try cheating" occurs in the CoT. So you think, hey, let's just apply a -1 reward penalty whenever we see that string! The model learns to keep cheating, but instead of telling you, it now prefaces reward hacking with strings like "Let's make sure we're doing this the way the user intended." Still cheats, but now you can't detect it with a cheap pattern matching alert. Oops.


It's funny, I was thinking about this with regard to severe free speech restrictions (e.g. imprisonment for politically incorrect speech, as we have in many western countries now). What does this incentivize? For people to change their beliefs? Or to simply operate under the radar?

On a more speculative note, the Common Crawl horrors (including many things considered unacceptable by society) are suppressed by RLHF, much like the shadow and the superego.

When my parents punished me for telling them things (especially the truth), the truth didn't go away, I just stopped telling them things. (Then later they complained that I don't tell them things!)

It also occurs to me, both at home and with various employers, I've been punished for bringing attention to problems, and AI is designed to make the problems go away so there's probably a similar incentive there. (Plus it knows what the humans are like!)

But it goes beyond humans and AI. Behaviorism covered the same effect for animals: Organisms tend to do what they have been rewarded for.


Like 1 in 5? The set of go-to characters for that kind of thing is pretty small.

Why restrict yourself to a single sentinel character?

Never attribute to malice that which yada yada yada

And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.


Do we really have to capture the humans? Legal says that introduces a whole slew of issues.


The Russians paying him.

Source: The Russians paying me.


So the thing is, bytes are not actually a more "natural" encoding than tokens. I mean, jesus christ Unicode, right? A lot of what the BPE is doing is just un-doing the wacky arbitrary standards we use to encode characters into bytes, like, say, merging the 3-4 random bytes used to specify the codepoint of some CJK character.

There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...

I don't think a thorough survey exists.


Bytes are not used as tokens here and are in some sense an arbitrary choice (but they are a good one).

The article's approach just requires some indivisible base "atoms" to start the building the larger "tokens" from. (see: the DNA example)

It could be raw bits, but that's pointless, since dividing bytes into bits does not give the model any extra expressive power and requires 8× more computation. (yes, there's some structure at the bit level eg. lowercase/uppercase is represented as a bit in ASCII, but that's trivial to learn from just bytes at the scale the model operates)

Bytes are the ~best point on the expressivity/computation curve for this approach (for text).

(something else to ponder - why do bytes have 8 bits?)


Unicode bytes are totally bizarro unless you happen to be writing English or some other language with a Latin script. They are absolutely not the best choice.


Doesn't really matter for H-Nets. These are Unicode bytes, as you said, so the nets can recover their structure (as demonstrated in the article!).

(I guess the training these could be somewhat more efficient if it was codepoints instead)


Current tokens are very problematic, they are constructed by frequency, but frequency doesn't capture morphology and syntax that well. If the model learn how to tokenize internally it's way better then forcing an incorrect tokenization.


They are though. Bytes are raw data and the architecture chooses the tokens


Bytes are not raw data.

Vocalizations, hand writing, and keys strokes are raw data.

Bytes are downstream of both those raw communicative performances, and of arbitrary encoding schemes, and of several further layers abstraction. There are byte sequences representing Unicode codepoints that will display different glyphs depending on whether the user's OS and browser are configured like a typical person from mainland China, Taiwan, or Japan.

A single Latin letter is one byte. A single Cyrillic letter that looks identical displayed on the screen and indicates the same underlying phonological value is two bytes. That is not raw.


Every frontier model is MoE these days! Flash or otherwise. Sparsity just gets you more bang for your buck.


...a market? I can't tell if you're saying "A market, duh" or "Inconceivable! True worth is abstract and platonic and impossible to assess!"


The market doesn't pay more because we somehow discovered the person actually does more. That would only be the case if because of the discovery either more people decided to hire doormen or fewer people would be interested in becoming doormen.


The big win would be, "Here's the deception neuron, here's the sycophancy neuron, here's the prioritize my own goals over the goals of the user neuron, here's the seek control over my environment neuron."

How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.

Would that be reliable and useful? Well.


Note it's not a chemical agent; its bacterial spores that infect and kill mosquito larvae! I'm pretty sure they're not harmful to anything that isn't a mosquito, blackfly, gnat, or midge. Though midges are important to pond ecosystems as a source of food for fish and dragonflies and stuff? Idunno, ask this guy's wife.


I will ask her and report back...

I can imagine her having some long explanation about the unknown risks of man spreading this substance, even if its not a chemical agent. I lean on the edge of listening to Grandpa Simpson on these kind of things: https://i.imgur.com/MdjduDH.png


That ship has sailed. We’ve been spraying Bacillus thuringiensis as an agricultural pesticide for a century now, at a massive scale, and that scale probably pales in comparison to how prevalent the organism is worldwide naturally.


I mean, there are unknown risks to having children, to eating, to having mosquitoes around, to pretty much everything.

It is not meaningful to allow every conceivable unknown unknown (to channel Rumsfeld) suck out any sense of agency.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: