This happens a lot, which is why agile-style methods kept popping up to replace waterfall techniques, even before anyone tagged them with the ‘agile’ label.
Ironically, that led to a Agile™ consultant-industrial complex that tried to make it a cookie-cutter process again, but we still haven’t escaped the need to break out the IDE and actually put idea into something concrete, and I doubt we ever will.
Well, OpenSSH contributes to this by mixing hashes from the shadow file. The musl code would be correct if OpenSSH passed in the hash from the user account that is being logged-in.
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
OK, I think I see what’s going on with this, in this configuration.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
I think the conclusion that can be drawn from this is that the behavior of crypt() is howling-at-the-moon insane, and it should no longer be considered for any system built in this century, at least not without extensive wrappers that correct and normalize all these edge cases.
In this particular case, it is type safe. NULL, by the C standard does not refer to an object. It means that every pointer in C is effectively an option type, which mean that a function that returns a pointer can return NULL for "no result", which, in the case of crypt() is perfectly adequate.
It is the opposite that is unsafe, that is, when the returned value can't be NULL, but it stills returns a pointer. C++ has references in an attempt to fix this problem.
Because C tends to prioritize performance over safety, a NULL pointer dereference is considered undefined behavior and the compiler can invoke nasal demons if it makes the code go faster, but with the right options, you can have it checked and invoke what in other languages would call a panic handler, but on UNIX is the SIGSEGV signal. You can then attach a debugger or read the core file and see what went wrong.
Note: I am not talking about "char *" for strings, that one is unsafe, but NULL is not. It is just a special case of an option type, or a result type, or an error union. Modern languages just extended this idea to objects other than pointers, and with more than one type of error.
> “When the compiler encounters [a given undefined construct] it is legal for it to make demons fly out of your nose” (the implication is that the compiler may choose any arbitrarily bizarre way to interpret the code without violating the ANSI C standard)
However this is limited by the compiler's actual abilities and code. The C standard doesn't forbid nasal demons, but the compiler isn't programmed to invoke nasal demons and isn't running on hardware with that capability, so it won't happen.
I've heard nasal demons invoked as a literal possibility. No they are not a literal possibility.
Maybe your compiler isn't, but GCC is designed to summon every nasal demon it can. It especially loves to elide null pointer checks. There was a PR that almost made it to Linux mainline that did some weird stuff in kernel space where memory mapped to address 0 is valid and contains important stuff. It got a pointer, read data from it, then checked if pointer is null before giving root permissions for some operation. Reading the code, everything seems alright. Even stepping through it with debugger shows that it works. But when compiled with -O2 or higher, GCC removed the null check and now everybody got root permissions unconditionally - because reading from null pointer is UB regardless of circumstances, and GCC treats UB as dead code even if it is reachable.
Sure, and char[] is no string because you can't tie anything with it. And don't get me started on the insects that allegedly live inside digital source files - that's an absurd concept, no self respecting computer specialist should be talking like that.
Wait, so if you're running on embedded, you're not allowed to store anything at address 0, requiring you to waste that little bit of RAM?
That's... a bizarre choice, considering how religious C is about not making any assumptions about what a machine can and cannot do, even going to the trouble of working with non-IEEE floats and int types that are not a multiple of 8 bits.
No. IIRC, you’re just not allowed to use a constant expression evaluating to 0 as a pointer to construct a non-null pointer to address zero.
Runtime pointer arithmetic resulting in a zero-valued pointer, is not a null pointer m, though it is also nonportable and could have the same effect (but hopefully not nasal demons, as I believe this falls under some catagory of nonportable behavior other than “undefined”).
There’s no requirement that the in-memory representation of a null pointer be all-zeroes, only that constant 0 represents a null pointer in source code.
The implicitness is exactly the problem. The workaround that caused this vulnerability was done because people assumed that crypt() would never return null.
You don't even need a fancy result type. A simple Option would've forced downstream users to consider the failure case, avoiding this situation in the first place.
The workaround was correctly designed: two error string tokens are used so that one can always be chosen that is not equal to the second argument.
Someone coded a defense against long inputs which ignored this important principle.
What turned the bug into a security disaster is that Musl stupidly chose two symbols such that one of them is exactly the same as a convention for disabled logins. So that the moment someone makes a change to the function which allows crypt("something", "*") to return "*", you have an incident.
If the same bug were made in Glibc, it would have limited impact because crypt("long .... string", "*0") returning "*0", while incorrect, wouldn't do anything in a system that doesn't use *0 in password files. *0 does not correspond to any widespread password field convention.
I think the argument would be that in C people will look at a char* and think "string" rather than "string or null". Thus consumers of the library not checking the result and the maintainer trying to be helpful by putting a "safe" value in there for the error case.
Whereas if the library returns something that is unambiguously an option type, consumers unambiguously call unwrap at their own risk.
Of course you do have a point (assuming I'm picking up what you're suggesting correctly) that all pointers in C are option types. But I think there's a cognitive ergonomics aspect to this where an option type more clearly relays the library authors' intent.
First you shouldn't have unwraps in prod code, and you can verify that in your CI/build process/whatever. Second, now you have a crash of the SSH server, which is arguably much better than some random logging in as root.
This isn't about "you", it's about whether any consumer of your code is using unwrap on the result, which currently is perfectly safe to do. The equivalent in C is returning NULL, which would also have caused a crash. They decided not crashing was preferable, which is how we ended up here.
"you shouldn't have unwraps in production code" is exactly the same form of argument as "you shouldn't be assuming crypt returns a valid non-null pointer in production code".
Absolutely not. Unwrap is a language rule, so everyone knows unwrap is not a safe thing to do and writing it even once is an explicit dismissal (and you should document it). On the other side, knowing that crypt does not necessarily return a hash is easily overlooked and forgotten, there is no code smell that can hint you that you are doing things wrong.
I don't really care about the form of the argument when the concrete results are that (i) “no unwrap in prod” is trivial to ensure through the toolchain, (ii) even though, its consequences are “damn it crashed” instead of “welp I guess this guy is root now”.
In this situation, the decisions to make crypt implementations stop returning null in favor of a string error token were made not because C programs that misuse crypt are difficult to fix, but because they are in the form of unknown binaries in the wild that you cannot fix.
And that analogous situation can happen in the safer language where crypt returns an Optional(String) which some bad programs assume is always present. You don't know where all such programs exist, and people use them who don't have the code and cannot fix them.
Returning an error token which is carefully different from crypt's second argument was an attempt to fix programs that cannot be hunted down, patched and recompiled, even if fixing them would be easy to do.
Of course the tooling for C pointers being null or not is not comparable to optional types; that's not debatable.
> And that analogous situation can happen in the safer language where crypt returns an Optional(String) which some bad programs assume is always present.
Once again, no, because these unknown programs would just crash on the `.unwrap()`; not get in a silently corrupted state. Same causes, different effects.
OK, so you are talking about running crypt() on bare metal microcontrollers or other small systems that don't have MMU support for trapping null pointers. Everywhere else, dereferencing an undisplaced null pointer is a instant CPU trap, even though it be formally undefined behavior in ISO C.
Note that even on systems that don't have hardware support for trapping null pointers, it can be implemented in the compiler as a code generation option.
For instance, in 1987, Microsoft Quick C introduced the /Zr option for doing this, as well as a #pragma check_pointer({on|off}).
In theory, having newtons and pounds force be represented as `float` is type safe. In practice, it destroyed a multi-million-dollar NASA spacecraft because of human error.
If NASA were to use the equivalent of `pub struct Newtons(f32); pub struct PoundsForce(f32);`, where the types Newton and PoundsForce were not interchangeable, this would never have happened[1].
[1] I'm aware that that story is a bit more complicated than that, that CSV files were involved (which are, again, not type safe), and that, in that particular scenario, more would have to be done to preserve type safety across formats, vendors and codebases, but my point still stands.
You might consider it type safe according to C's rules, but it is not type safe according to contemporary opinion because the success type and error type are the same.
Ok and Err are "thingies" which are in the type system, which give rise to distinctions that allow for compile-time diagnostics. They are part of the type.
Ok and Err are not just type constructors because they appear on the left hand side in pattern matching: they "do" matching and destructuring also.
The success and error types are not the same, because they are in different positions in the type, and so distinguished by different patterns.
If you have literally the above convention where a valid result is a String, and an error result is also a String (e.g. error message) that is still type safe.
match (whatever()) {
Ok(s) => println!("good result string: {s}"),
Err(s) => println!("error string: {s}"),
}
The success type and error type having to be different is not a good way to express the idea; rather that success and error cases are distinguished by type (even if the success and error data units are of the same type).
In other words, the "Ok" and "Err" entities are an ingredient of the type and that effectively makes the result and error be distinguished at the type level.
But we could also argue that Rust still allows you to write an integer-returning function such that -1 means error.
Basically, you need the whole 'success and error types are different' in a language like Python if you make the return type something like "int | string", where 'string' stands for your error message.
Of course that breaks down, when the success type is also an error message, eg when you are looking up something in a catalogue of error messages.
Rust has no trouble with this, for exactly the reasons you line out.
Soundness means that the program behaves in accordance with what its types express. You're right that C may be unsound due to undefined behaviour, but I don't think this is about soundness. What you're saying is that papers on types show how you can use types to express more constraints.
You may be right in general, but soundness comes from formal logic, where they're pretty fastidious with their definitions. Soundness certainly doesn't mean the expressivity of the types (i.e. what constraints they can represent); it's about the relationship between the runtime behaviour of a program and the types in the code.
Again I think you're trying to push a specific definition as more correct than OP's somehow. Why not just ask them what they meant to understand their intent/context instead of saying they are wrong?
Unless the point of your argument isn't to understand OP but simply to "win."
I think I understand their point and even said, "What you're saying is that papers on types show how you can use types to express more constraints," and that's correct. I merely pointed out that that's not soundness, which is something I think they'd like to know. There may be some leeway in the definition of soundness and how it's similar to or different from type safety, but either way, type expressivity is something different altogether (and C certainly has issues with both expressivity and type safety or soundness, but they're different issues). So the point isn't to win - the argument was right and I'm not claiming that C's type system is expressive - but to explain the different terms and their meaning in programming languages and formal semantics.
That's not the same as asserting that soundness requires adherence to the current fashion, sum types over results and errors. TIMTOWTDI.
You don't have to think C is sound or good to distinguish a disfavored API style from mathematical unsoundness. There are infinitely many type safe APIs and languages that are, despite their soundness, ugly and hard to use.
Let's not project every dimension along which a language can be good or bad onto the axis of safety.
Opinion among people who know better than to try to authoritatively explain subjective words with rigid dogma, because that just makes them look silly.
Exactly, whoever program in C knows that every function that returns a pointer may return NULL (even if not documented) and thus every pointer must be checked.
However a better API (to me) that I would have written was:
That way btw the function doesn't need to allocate memory inside, which is (being a mostly embedded programmer) something I don't like at all. Plus returning an error enum makes it clear instead of having to look at errno.
But POSIX API is not something well designed, it dates back to the '80 so... probably today would be written completely differently
We have that better API to some extent; it is called crypt_r. It originated in glibc, evidently, but hasn't made its way to POSIX. It fixes the reentrancy problem of crypt.
It is not immune to the present issue. If our reentrant crypt is given a long password and the salt "", and it decides this is an error, placing the output "" into the result buffer, then it has the problem. It has the problem in a thread-safe way, but it has the problem. :)
> It is just a special case of an option type, or a result type, or an error union.
These are not the same, and your next sentence suggests why. Option has but a single error type, but result types and error unions allow arbitrary types of error.
That depends on how you interpret the pointer type, but in any case there are multiple interpretation and just the fact that these exist will create confusion and type unsafety.
The idea of trying to provide a workaround for buggy code calling crypt() is terrifying. For sure, trying to grease the skids for buggy code calling printf(), go for it. But it is hard to imagine cases where slightly-incorrect usage of crypt() is a good idea.
I don't think so. Even in a language with a good type system like Rust, somebody could have made this API:
/// Hash the phrase with the given setting.
fn crypt(phrase: &[u8], setting: &[u8]) -> [u8; CRYPT_OUTPUT_SIZE];
Then, in a later version, `crypt` needs to become fallible, so they need to signal errors somehow. But because they really want to preserve backward compatibility, they can't change the output type to `Result<..., ...>`. And so they end up doing the same thing that was done in libcrypt - they output some error value and hackily make sure it's not equal to the `setting` string, under the mistaken assumption that the output of `crypt` would only ever be compared to the `setting` string. And then someone creates the exact same bug.
Maybe I'm wrong, but it looks to me like the exact same incentives and means to screw things up exist in C and in Rust for this particular bug.
IMO type safety requires both language features and a mindset where you take efforts to keep things type safe. Your example has only the former.
For example, in a Rust web app, you can choose to use String for encoded URLs, unencoded URL fragments, escaped html, unescaped HTML, usernames, etc. Or you can choose to use newtypes and carefully design all APIs to only take and return the correct newtypes.
The problem is not that you can write bad stuff in rust, the problem is that this particular usage pattern can't be sanely coded in C. This is a public API that should inform users and enforce certain properties. A Results type does exactly that, you would have to go way out of your way to force invalid stuff through that type alone.
I don't think so. In both Rust and C, the solution is the same - make the function explicitly fallible, instead of trying to preserve backward compatibility. In C, the function would probably just look like this, returning zero on success and negative on failure:
int crypt2(const char *phrase, const char *setting, char *out);
You can call such a signature "insane" if you want, and that's fine, but it would have averted the bug that is the topic of discussion.
what you are suggesting is a form of type safety. There are still degrees of type safety even in C, and this bug resulted from deciding that backwards compatibility is more important than type safety, which I think we agree on.
It would not have averted the bug. The bug is that out is the value "*" even when setting is also "*".
It would have averted the bug in a program which checks the int return value for error.
But we had that already: the null value, different from any valid string. That had to be done away with because programmers couldn't be relied upon to check for it.
Those same programmers are not going to check an integer value which doesn't even threaten them with a segfault.
(However, there is a GCC attribute that can be given to functions such that a diagnostic is produced if their value is not used: that could be profitably used with the above API.)
This isn't good either. Crypt is a pure computation that can't fail for arbitrary reasons. The former implementation could fail if the salt was invalid, which is something you could guarantee. Now it also fails if the password is invalid. That's new behaviour and no matter which way you wrap it up it has the potential to be incompatible. They should have added the maximum length recommendation to the man page, and let DoS attacks continue until the calling software was updated, not the crypt library.
But in your version, you can't preserve backwards compatibility, making an infallible function fallible is a breaking change. For something comparable to libc you might want to (eg Rust std still has mem::uninitialized), but even with security issues, I feel like people would be more inclined to deprecate for a new function (or panic, which does have its own problems).
A good type system also means things are written with better forwards compatibility, and can be safely changed later. The Rust equivalent to `crypt` likely wouldn't exist: a low-level library would just provide hash functions, leaving callers to mitigate DOS etc; a broader library might return some opaque handle with equality/serialization, which you couldn't try to compare to a passwordless user; a high-level library would parse the account store itself. None of these really need to change in incompatible ways, they have all the information to add features and fixes at their own level of abstraction.
You wouldn't be concerned about breaking backward compatibility with rust though, would you? A recompile would be needed anyway and the change would be detected and enforced. The rationale for the bad API here (two types in one variable) was that users were ignoring NULL - that's the part which wouldn't happen in rust.
Specifically, that the lack of tagged unions (or really, structured return types in general; you can pretend those are a tagged union if you believe hard enough) in C library interop conventions is a terrible foundation on which to build widely-used APIs.
And that just ends up indicting the C language itself. Can't send errors down a separate channel; folks might not inspect errno. Can't return NULL on error, folks might compare it blindly to another result. No tagged unions, so can't return one. Can't crash on error because there's no scoped error handling. Can't return a struct for checking .error/.result separately, because alignment and equality (and tearing and accidental clobbering and more) make that brittle as hell for widely-used APIs.
C is not a good language for library APIs. Like, really not. It has its uses/strengths, sure, but without even getting into the typical gripes people have about C, libraries--one of the things that it is used for the most--are one of the things for which it is the least well-suited.
Passing and returning structs by value has been part of C since C89. Most compilers pass small structs in individual registers, and nowadays with LTO they'll even elide the copy of bigger ones if they can. What's definitely the case though is that by-value structs weren't a common thing when most of the Unix API was written.
Another example that C programmers love: using 0 to mean infinity. Eg when you have a timeout setting, and 10 means ten seconds, but 0 means infinity seconds instead of zero seconds.
In general it's a problem of sentinel values. Some languages let you abstract this - I think in Rust and C++ you can define an "intrusive optional" which is like optional but uses an invalid value pattern to store the not-present state. In memory it's the same as a sentinel value but the compiler forces you to unwrap it like an optional.
Amusingly, Win API of all places generally evades this pattern: 0 timeout means "no waiting", and to wait indefinitely, you use timeout of INFINITE which is actually UINT_MAX or, equivalently, -1 (in Win API, timeouts are unsigned, not time_t).
That’s just a very bad API. You could do the same in Rust? There are nice C APIs that would use a nice struct that describes the timeout you want… eg GCD.
Not really. This issue is caused purely by the handling of the too-long input in the wrong way.
Firstly, why do crypt functions return a valid string to indicate an error? The problem is that the classic null return makes some programs crash which are not expecting it.
musl has a correct protocol for the error string, similar to the *0/*1 convention. It also uses two symbols, * and +. If the salt string is *, then it uses \+ specifically to thwart this issue: the program blindly assuming that the password is correct because the output of the function matches the hashed password field.
Where it seems they screwed up is this:
crypt("long .... string", "*") -> "*"
I.e. in this special anti-DDoS measure that checks for a too long input, they forgot to implement the key piece of logic that the output of the function, when an error is being reported, must be different from its second argument (the "salt", which is usually the entire hashed password or the contents of that field from a password file).
Thus the application ends up comparing "*" and "+", or "+" and "*" so the authentication fails.
Under no circumstances should "*" be returned when the second argument is "*", or "+" when it is "+", because it must be assumed that the app doesn't do any sanity checking, and blindly compares the second argument to the return value.
However OpenSSH passes a different hash to musl’s crypt() than what it later uses to compare, so musl doesn’t switch to the alternate return output because from its perspective, it’s already guaranteed not to accidentally compare the same as the input hash.
Is really the solution to a bug where a function had a wrapper that tried to solve its documented issue of returning *, to make yet another wrapper to try to solve the documented issue of returning true?
musl is actually safe against this use. OpenSSL takes a bizarre code path, detects that $hash is too short, calls crypt($pw, $other_string), forgets that it detected that the hash was too short, and checks $hash == return value.
I did note that musl would normally work just fine here. Though I appreciate that your other comment goes into much better detail about what was actually going on in the code, as my analysis basically stopped once I understood the high-level flow of what was going on.
Except OpenSSH qould have worked just fine if musl behaved correctly. Instead musl ignored the spec and tried to hack around badly written programs but ended up creating a security hole in a correctly written one!
Yep, the linked page doesn't /feel/ AI written (maybe it's using a grok or muse model whose tone I don't quite recognize yet?) but it kinda just states a lot of facts that make sense on their own but don't cohere into a whole vulnerability like the comment above does.
I think it's just Claude prompted to write in a casual, conversational, no caps style.
The surface-level hints are gone, but the attempt to make every sentence pithy, the decisions on what information to present and what to assume the audience knows, and the over-emotionalization are all still there.
It feels very AI written, it's just not the blatantly AI default style. Pretty much any prompt with a "write X in this style: [example text]" will produce writing like this.
and what same human would embed two linux systems (??) in a blog post just for a demo that prints "pass"/"fail" that hardly anyone will bother to interact with?
Because Claude wrote the linked page. Claude is impressively bad at communicating these days. with gpt-5.6+ approaching Claude level like 30% of time and Gemini much clearer but also the most annoying 7 year old ever, it’s a pretty lame environment especially for people who r new to everything
I wondered about this myself as the explanations and documents I have been maintaining are becoming increasingly 'slop', but a different kind of slop.
In the past I could get an explanation, where I was able to track the line of thought. Now when I read through the explanation it is like someone has taken two or three explanations and merged them into each other, like there is a combining of iterations without clarity.
example:
The tree is blue.
The tall tree has the color of blue.
the tree, while tall, is blue.
That finally gets squished into: the blue tall tree has height and is evident of blue.
Is this double speak or injected double effort of words?
Until Opus 5.0 I'd have agreed with you, but Opus 5.5 is honestly one of the best I've ever sparred with, vastly better than the rest, including Astra imho. Especially so when prompting for clear top-down, first-principles communication.
Disagree, I review a lot of Opus 5.5 code and its comments and docs are still pretty inane and Claudish. Is it better than the 4.x and 5.0 era, sure. But it's still nouning the verbs, using colons everywhere, a-not-b, "a foos the b" etc. A lot of lingual knots that need a human hand to untie.
Opus 5.5 is much better, but IME I always have to force it to review its writing (Opus 5.5) to get it to put the most important facts first (and not bury the lede). It loves to park the main point far away after intermediate clauses. It always writes “premise, so consequence.” It takes a prompt asking to review against rules in context for it to fix it.
The ‘concise’ output style setting is bad with Opus 5.5, as it will often spit out a cryptic (is this really English?!), allergic-to-commas concise response, which takes more time to parse.
I agree. The “excuse” for implementing those behaviors is just providing plausible deniability. The commit that lets crypt return “*”, followed closely by a change that lets you easily trigger that behavior sounds just like how somebody would sneakily introduce a back-door into tons of linux systems while making it look as innocent as possible.
Well, it was intentional, but not in the way I think you mean.
musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
That was a misfeature and should never have been implemented. But I bet it was an easy band-aid change to make!
But choosing "*" as the character to return is an insane choice for anyone who knows the basics of linux sysadmin, or hell anyone who's ever used a search field in any context and thus knows it is ubiquitously used as a wildcard operator.
But it fits the common Linux implementation of the function... e.g. the Debian manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an invalid hash to the _output_ field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
That says 'begin with,' which does not remotely require it be exactly the 1-character string '*' which probably was a bit of an insane choice under GP's logic.
And that's also only in Debian 11 and later. That man page said something entirely different in Debian 10 and earlier, when they shipped glibc's libcrypt. Now they ship libxcrypt instead.
Fedora asked libxcrypt to give them back glibc's behavior, returning NULL, and they obliged by adding ./configure --disable-failure-tokens. But it's an option that Debian never enabled.
It's almost sensible if you only think of one side. It's like if /etc/passwd put "NO_PASSWORD" in the field for disabled logins and someone was like, oh, I'll be clever and say crypt() returns "NO_PASSWORD" on error because that's obviously invalid and won't match. But that's exactly what *does* make it match.
> musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
Of all the things to mitigate...
1) crashing should be an option - it noisily indicates "there's a problem" and if it faults and dumps core, you've got a guided tour of your problem.
2) Is testing NULL as a passable option likely? I'd guess a more likely idiom is:
Wise words. I think early in my career I tended towards coding as defensively[1] as possible in a very specific way of "don't crash", as I definitely was not working in codebases I felt I had a full picture of. "I don't know who calls this method, and I don't want to get a call in the middle of the night that I broke prod" so I would say, return an empty string when asked to format a datetime except I'm passed a string or an int. (Yes, I was working in dynamic, untyped languages, too).
Later on, even in untyped languages, I came around more to "Throw loud and specific exceptions when asked to do things that are nonsensical" - it requires more overall cojones though because it may cause "your code" to crash. But the realization I came to was that it's better to crash ('a MaliciouslyLongPasswordError was uncaught') because "AuthManager asked PasswordService to check a password that was 2MB long" than to paper that over until something like this comes out. Or, the more common situation, you're passed garbage, you return something that won't choke the caller at the time, but what it saves to the database in that scenario ends up being corrupt data. 6 months later someone notices, and has to sort out a data mess that may or may not be recoverable.
[1] note that most "code defensively" advice is good. What I was doing could be thought of as a foolish misinterpretation of the phrase.
I think the issue is that a lot of callers were not aware that crypt() could error at all and all the issues are downstream of that. If OpenSSH knew that crypt() could error they would have handled it (possibly by inspecting errno).
I don't think that's the problem. The man page for crypt clearly says that erroring is a possibility and that it's not standardized how it errors:
> The behavior of crypt on errors isn't well standardized. Some implementations simply can't fail (except by crashing the program), others return a null pointer or a fixed string. Most implementations don't set errno, but some do. POSIX specifies returning a null pointer and setting errno, but it defines only one possible error, ENOSYS, in the case where crypt is not supported at all.
In my opinion the problem is that the behavior of crypt is underspecified - it isn't even specified how it signals failure, which is shocking.
Another disturbing decision that crypt's signature takes in the hash its output is supposed to be compared against - if it weren't for that, this half-baked "on error the output won't compare equal to the input" measure that crypt and crypt_* use wouldn't have been possible.
You can't check errno to see if there was a failure because, as the man page says, the value of errno is unspecified when crypt succeeds.
This specific manifestation of "C is terrible for shared code APIs" is rare.
But issues caused by folks forgetting to check errno, miscomparing NULL to NULL when calling multiple APIs, dereferencing external-API-returned NULL due to missed error checking, mishandling inherited errno after successful action, or--the most common--mishandling negative integer return sentinels meant to indicate errors? Those are incredibly common.
This specific library call is quite poorly designed.
Standard C api calls return an integer, which is 0 on success or nonzero on error. Ignoring it is an intentional choice. `errno` is the hack that shouldn't really exist because it's trivially easy to forget to check that and also has issues with threading (patched by making it a thread local) and more recently, coroutines (which admit no general purpose patch because they're a userspace construct).
Another reason why this is terrible is that it's unclear what the returned pointer's lifetime is. Is it until the next call to crypt? If so, is it thread-local like errno? If it's passed on to the user, is it safe to call free() on it, or is there a crypt_free function that should be called instead?
But all of this, and I mean all of it pales in comparison to returning '*' on error.
> But all of this, and I mean all of it pales in comparison to returning '*' on error.
100% agree. This is an egregious failure. But C's error-handling story would be bad even without this, and the badness of C's error handling is what motivated the (bad) decision to return *.
It'd be great if we wrote systems software in a language that supported advanced features like...
checks notes
...returning multiple values.
Hell, even in VBA you had to at least spend the keystrokes to type "on error resume next".
It's just that every other string than "*" would have been a better choice, including even the empty string (which would have been the next terrible choice).
It's not a C thing, it's a Linux thing. The asterisk is relevant to how crypt() is used for /etc/shadow.
If the password field contains some string that is not a valid result of crypt(3), for instance ! or *, the user will not
be able to use a unix password to log in (but the user may log in the system by other means).
No, it's a C thing. They previously returned NULL for invalid passwd lookups, but because there's no way in C to force API users to decide what to do if an error occurrs, that was causing program crashes due to NULL dereferences.
So they made the (very bad, but motivated by C's broken design/conventions) decision to return a magic string instead.
> but because there's no way in C to force API users to decide what to do if an error occurrs,
0. There are tons of libc functions that return NULL on failure, it is conventional way in C to indicate failure.
1. There's no much to decide, just fail to login if crypt() fails with NULL.
2. errno is standard way in C to check error code of a failing function, but in case of login you hardly need this, just fail!
I totally agree that fixing this issue in musl was a pretty dubious solution. Don't fix what's not broken, fix broken application instead. Publish a CVE if you cannot fix it.
I looked through FreeBSD source code for how they use crypt(), it's checked for NULL everywhere!
Nothing you said is incorrect, but I believe they meant that there’s no way to force a C program to check the NULL correctly—the compiler will happily compile code that gets a pointer back from a function but doesn’t check for NULL before dereferencing it.
In something like Rust the type system won’t let you touch the value until you’ve done something to deal with the error. Even if you just want to ignore it, you are forced to acknowledge that the function is fallible by the type system—it’s a compile error if you don’t.
Returning NULL on error is actually an excellent choice because most programs will crash if they try to use it without checking. A program, any program, crashing is better than this bug. Of course, modern programming languages have better constructs, but this was rare to nonexistent when C was started and if you think C is going away anytime soon you're delirious. musl's decision to change the return value to "*" is ununderstandable and likely intentionally evil.
> musl's decision to change the return value to "*" is ununderstandable and likely intentionally evil.
Dude, they documented their rationale. It was dumb and wrong but totally relatable. Irresponsible users were getting crashes, and I’m sorry to tell you that for a lot of library maintainers
> Returning NULL on error is actually an excellent choice because most programs will crash if they try to use it without checking.
…is not something they want to write in responses to bug reports, even though it is quite true. It’s human nature to want to prevent your stuff causing problems for others, even if the problems are fully their fault. Most people cannot and will not be Steve “so don’t hold it like that” Jobs, for fear that their decision will be just as short-sighted as his was.
In this case, that was obviously the dead wrong call. But I’ve been in that seat, and “closed: user error” for the 100th time feels bad to write. Add to that a touch of misplaced cleverness, and you get “hey…what if we just returned an impossible string instead of an error?”
Evil requires motive or malice, and I’m having trouble coming up with those in this case. These patches weren’t concealed or miscategorized a la xz, the behavior was discussed in the open. They were just, in hindsight, really poor decisions made for ordinary reasons. Hanlon’s Razor and all that.
That’s why I’ve been harping on C being trash for shared-code APIs in these comments: bad decisions are gonna happen, and irresponsible users are forever, so can we maybe use tools that require you to put on a seat belt (mandatory error handling) before starting the car?
> That’s why I’ve been harping on C being trash for shared-code API…
Agreed, but I can’t think of any other widespread ABIs that are better. Rust’s is not stable both C++ and Rust have concepts that don’t map into other languages well, so using them wholesale is a no go. So it would probably need to be a subset of those. Honestly just tagged unions would probably be enough. Maybe the solution is to get native tagged unions into C…
There is the concept of "defense in depth": having each layer of code and configuration be sane and secure. Even if you program in Rust, you should be vigilant in that sense. This kind of code amounts to removing one or two layers.
Of all the possible strings this seems like the absolute worst choice. If I were typing that string I'd pause to think "wait, what if the caller uses this with a regex?"
One has to remember the context here. The only error possible according to the SUS is ENOSYS, literally that crypt() is not actually implemented in the library. The traditional AT&T Unix crypt() was always implemented, and its encryption did not have failure modes, so there was never any error return to handle. This is why programs were written to never check the result of crypt(). It was only ever NULL on systems where one couldn't use the mechanism at all in the first place.
So the knock-on effect is that libraries that do have errors other than ENOSYS to accommodate, have to return magic 'impossible' values instead of setting errno and returning NULL, because of the weight of existing applications that expect nothing but the standard, traditional AT&T Unix (and indeed old BSD), behaviour.
Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
If you want to see real history here, check out the 1992 comment by Eric Andrew Young, that is still in the source for OpenBSD today (in des/des_fcrypt.c), on how SSLeay's crypt() had to be adjusted so that any single character password field, including "*", did not end up successfully matching itself.
It's worth pointing out that musl tried to do the same as SSLeay, and deliberately returns something different to the original salt so that they will not compare equal. It does not return "*" if the input is actually "*" in the first place, contrary to the headlined cartoon explanation. (The locus of the problem is the caller calling it with a salt that is not "*" and then comparing it with "*".)
> Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Although that's a fine distinction for the point being made, as it's still the manual page for the crypt() function that one gets on the operating system, which that other manual page is not.
Not really. For Linux-based operatings systems, the GNU libc manual cryptography section is a shadow of its former self. It has no crypt() function to accompany it any more, and has not had since 2023.
Notice that Debian didn't list your manual page, which doesn't accompany a crypt() function at all, since it isn't actually the GNU libc manual but a separate independent manual for Linux. The GNU libc manual nowadays acknowledges Michael Glad's work but no longer documents crypt(). It used to, and it used to explain some of the 'impossible' encrypted string values:
I suggest finding a code path where FreeBSD crypt() ever returns NULL, which would mean one of its internal functions returning non-zero, before getting too excited. They all end in return (0) . (-:
As I said, the callers of crypt() have for a long time been using the knowledge that crypt() on their platform simply never returns NULL. Unix has never been returning NULL from crypt() as far back as 7th Edition, when it simply always returned a pointer to a single internal static buffer.
The reality is that all C library people have to work with a corpus of Unix applications that do not check for crypt() returning NULL that goes back almost 50 years, and an error return mechanism that barely exists outwith the pages of the SUS.
That includes callers in single-user mode in process 1 in some configurations.
Well, crypt_blowfish() has a number of returns with (-1). I'm not familiar with Blowfish, so it's hard for me to trace the possibility of failure here. But what if in a distant future someone adds more hashing algorithms to the list without possessing historical knowledge of this "never return not 0" ?
Failing to check result of crypt() in init cannot do much harm, doing so is equivalent of going system reboot (crash init :-)). Besides, whoever has access to system console can do whatever they want with the system, no password can prevent from that.
I have no doubt, despite that Marshall Kirk McKusick is still committing in 2026, that FreeBSD developers will come along that do not know history. But they will get pointed to the manual page that (amusingly) says that returning NULL is 'not a standard behaviour' and that AT&T Unix never did that.
NetBSD's manual is more detailed on the subject, and explains the bind that library implementors have been in for decades, thanks to almost 50 years of applications knowing what crypt() 'really' does.
And this despite the SUS trying to shoehorn an error return mechanism into crypt(3) in order to accommodate the problems that operating systems had exporting encryption code, two decades afterwards. (The 1997 GoSolo book explains that this is why there's an ENOSYS error.)
The most charitable interpretation here is that musl authors are incompetent.
There is simply no universe in which a security-conscientious reviewer looks at the code, spots * being a valid return value, remembers that * is the default value in /etc/shadow for "no password login please" and then is all like:
This is fine, you check for * and return + instead.
There are more 32-long character sequences than atoms in the universe. Pick one, any of them instead.
Returning a string on error honestly is defensible (especially in a language like C where the type system doesn’t have exceptions or result types), the question is why the string was “*” instead of like “crypt ERROR: input password cannot be longer than 256 characters” lol
"*" is what's been historically used to represent an invalid/no value in passwd/shadow, so it makes some sense.
Of course given the chance that one might do a direct string comparison between crypt's return and the passwd/shadow file's hash field it was still a not well thought out choice.
It doesn’t help Google to create an AI-powered business which turns $1B in profit annually if it results in them losing a search/ads business which turns $5B in profit annually.
The CapEx spend for AI not only makes AI itself more expensive (and thus less profitable), it’s now driving everything else in technology to be more expensive (and thus less profitable).
AI is a profitability tax on the entire tech sector and it remains unclear if the profit AI will deliver is going to be worth that.
I would say they should slow down on build-out to balance supplier/labor availability with available cash flows, but I suspect all the Big Tech companies are worried about falling behind the other and being put in the position to need to rent out compute.
It’s more or less certain that Anthropic is already profitable if you discount the losses due to becoming too valuable.
What you are saying here
> The CapEx spend for AI not only makes AI itself more expensive (and thus less profitable),
There’s nothing inherently unique about this. It’s just supply and demand - the only problem is that the ecosystem of raw materials can’t keep up with the demand. It will catch up soon and the prices will stabilise.
Your only problem seems to be this aspect only: that RAM production hasn’t increased according to demand.
No I don’t agree they should slow down. I think RAM and other things should rather speed up to react appropriately to the demand.
It’s like saying the iPhone is not profitable because too many people want it and its price is going up.
People complain about the imbalance between economy supply and economy demand leading to inflation and debt concerns.
The complaint isn’t that the economy is “circular”, how else could it work with 3 or more people engaged in commerce? Neither of those things get fixed on different ways to manage the money supply either, it’s not as if they didn’t have national debts or inflation while on the gold standard.
It’s just a different format. Providing liquidity in the hope that more is generated from the economy. Except it’s doing more harm than good other than in true crisis. Other times it’s just creating a bubble, in national debt and in AI.
The introduction of the mouse led to developers paying less attention to keyboard shortcuts and function keys, but I don’t see anyone seriously saying computers should not have mice.
Or for that matter, that adding support for game controllers (something that has actually happened with Linux desktops) would led to curtailed development of non-controller parts.
> The introduction of the mouse led to developers paying less attention to keyboard shortcuts and function keys, but I don’t see anyone seriously saying computers should not have mice.
Well, a lot of devs did move back to the terminal ;)
I encourage it! I think having to deal with switching between mouse and keyboard actually accentuates why the terminal is nice as an interface so often. If it does nothing more than make someone building a terminal app think just a bit harder about where to be intentional about user workflows so that a mouse is not even required then it may even have helped keyboard-only uses.
It's sort of weird in a world that has moved to an DevOps-style ethos for most development.
But yes, before that it would be not at all unusual to have “test engineers” or other technical staff whose entire responsibility laid with setting up test environments and running QA of products. These people did not do product dev, though they may have been involved in writing supporting test platform libraries or even maintaining test infrastructure (like IBM Jazz).
Even today in the U.S. companies that haven’t fully jumped onto newer development practices probably still have QA and/or test departments.
>It's sort of weird in a world that has moved to an DevOps-style ethos for most development.
The world at large makes all sorts of trends, and that by itself doesn't make them smart. As a developer, you are likely to have blind spots when it comes to quality and reliability as a result of having written the code. Even a team of developers cross-testing each other's code will not have the same ability to find flaws (before they embarrass you in front of the customer/client/user) as a dedicated tester whose job it is solely to find new (and old) ways to break things (and document them.)
DevOps is a completely different thing (though they work together with testers.)
It is only weird to have dedicated QA on your team if you don't give a fuck about releasing shit with bugs in it for the user to bump into.
> The world at large makes all sorts of trends, and that by itself doesn't make them smart.
I never said it was smart. Or dumb, for that matter. Just that many product teams have been made to merge testing responsibilities as an internal responsibility, in line with having end-to-end ownership of the product from development to its operational deployment. And usually without any dedicated test engineer member of that team to do this.
> It is only weird to have dedicated QA on your team if you don't give a fuck about releasing shit with bugs in it for the user to bump into.
Well, has the behavior of software companies led you to believe they care strongly about this? I always got the impression they cared more about either maximizing feature velocity given a certain team size, or minimizing labor cost for a given feature velocity.
Isn't this still a thing? "Software Engineer in Test" being the newer term?
I just checked LinkedIn and they have quite a few job listings in this vein. Which makes sense: in the "AI and YOLO" development lifecycle, automated testing is more important than ever.
The solution is to avoid UB. std::unreachable() is just an easy-to-invoke way of hitting UB, mentioned for the sake of the discussion. They did later take your advice “Let’s say we change std::unreachable() to int* ptr = nullptr; *ptr = 99; in the previous example.”. They still got improper codegen out of the compiler.
Though, it would be better if programmers read deliberate instances of UB as if they literally said "std::unreachable()" at that point in the code, as that would better convey how the optimizer passes will see things after dozens of frontend and middle-end passes have run before it.
> it would be better if programmers read deliberate instances of UB as if they literally said "std::unreachable()"
Except that isn't true.
There are basically two kinds of undefined behavior.
The first is undefinable behavior, the kind that easily goes disastrously wrong, like writing to random memory locations.
The second is incidentally undefined behavior, something where there just isn't enough implementation consensus for the standard to settle on it, like realloc(p, 0).
There's no standard terminology I'm aware of to distinguish the two. But the distinction exists.
And where programmers read deliberate instances of UB, it's usually the latter, e.g., writing directly to an address memory mapped to device I/O in that particular environment.
Ironically, that led to a Agile™ consultant-industrial complex that tried to make it a cookie-cutter process again, but we still haven’t escaped the need to break out the IDE and actually put idea into something concrete, and I doubt we ever will.
reply