sshole

> $ret = crypt($pw, $hash); $ret == $hash

musl is actually safe against this use. OpenSSL takes a bizarre code path, detects that $hash is too short, calls crypt($pw, $other_string), forgets that it detected that the hash was too short, and checks $hash == return value.

See my other comment.

wtaf ?

A typo: `$ret = crypt($pw, $salt)`

OpenSSL??

So, any field de-reference in Java can produce a null pointer exception. Try testing all paths in something like:

try { foo.sideEffect1().sideEffect2()...sideEffectN(); } catch { bar.sideEffect1().sideEffect2()...sideEffectM(); } finally { quuz.sideEffect1().sideEffect2()...sideEffectP(); }

There are M * N * P paths. The only way to write testable code with exceptions is to panic on uncaught exception, and to never catch unexpected exceptions. Even then, you're relying on nothing unexpectedly throwing the exception type you caught.

I'd argue Result<T> >> NULL >> sentries >> exceptions for this reason. Sentries are terrible, but at least they're not computationally hard to use correctly.

https://en.wikipedia.org/wiki/Semipredicate_problem

If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.

With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.

When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".

> Turns out that sshd detects that the password hash is too short and does not forward the salt!

I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.

[1] https://lists.mindrot.org/pipermail/openssh-commits/2016-Jul...

My AI usage involves telling it to do things and setting up lint rules to catch all the bad programming it’s learnt.

One potential point of disagreement: it's easy to implement structural prohibition of magic strings in low-level languages too if you consider Rust a low-level language (which I've heard before, even though I didn't really consider it to be one for our application of it).

I'm not a Rust expert but do run a few thousand lines of it in prod for some perf/correctness critical code, and I wish it didn't have the reputation of being so difficult to comprehend. There's so much resistance to introducing it or new languages in our large org, that I'm surprised we were able to introduce our thin slice at all. I am not sure I'd use it where you need easy async primitives like goroutines, but it prevents whole swathes of bugs that are repeated over and over again with devastating impact, like the OP's bug, via thoughtful language design.

Maybe this resistance is an artifact of where I work more than large organizations generally, though. Curious about others' experiences. I'm not really an evangelist for the language but would really like to work in a more polyglot environment one day.

I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.

I think that's the case where LLMs might indentify most.

It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head

Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.

When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.

And now we have this.

And why because some dude was complaining sshd or whatever was crashing because it didn't handle nulls? Just handle the null my guy you already know why your bullshit is crashing.

Exhilarating, isn't it?

Integration of different software is the stage most likely to introduce uncaught bugs. That is what happened here: musl intentionally broke the documented API of a function used by OpenSSH. Then Alpine integrated this incorrect implementation with OpenSSH.

Also didn't have much luck on Alpine 3.20.10 with OpenSSH_9.7p1 which I've freshly installed from an official ISO.

Last two entries in /etc/shadow:

    test:*:20762:0:99999:7:::
    q3k:$6$snip.GgDQNTrqqX.A0:20762:0:99999:7:::

I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.

Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.

edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.

I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.

More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.

That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.

100%

Go ahead and program a flight controller this way, I'm sure nothing bad will happen.

Is there a test or check I can use to determine that software is 'good'?

Oh there isn't? Good/bad is subjective? We need to scape goating hacks as bad software, as if anyone is capable of perfect code. Perfect code would require knowing the system inside and out, the stack up and down - no one knows all that, no one is perfect. Sufficiently advanced AI though can rip through all of that, find the weakest spot and zero day it, bringing the entire house of cards down.

Software you thought was good can still be hacked. There is no way to create provably unhackable software.

Which means all software is packed with vulnerabilities just waiting to be discovered, no matter how good you think it is.

Hey, but at least it can’t touch the System32 folder.

Great job guys.

Objective-C ahem nil messaging...

Sadly, we will never learn this because failing fast runs counter to intuition, and a class of person will never override it.

Do you have an example? Looking at Alpine's advisory, I can only see that their cloud images (i.e. what you use to run a VM, not a container) have a default alpine user, no container images. And even with those, cloud images with cloud-init are unaffected with default config due to `PasswordAuthentication no` being inserted by cloud-init into sshd_config. So that leaves... Cloud images that use Tiny Cloud for provisioning. That's an instance bootstrapper authored by Alpine from the looks of it so I'm not sure how many people would prefer that over cloud-init (which a lot of VM/bare metal management software may already make easy to use) or just a non-cloud image.

Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".

[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.

However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.

This thread highlights why I often say that at some point in the future, the obsession with whether something was written by AI or not will be classified as a mental disorder.

It is tiring to virtually every day see this discussion play out in some HN thread or other. It's also just noise. I don't think your comment, nor the thread it inspired, added anything of value to the discussion.

  > root@OpenWrt:~# cat /etc/openwrt_release | grep DISTRIB_RELEASE
  > DISTRIB_RELEASE='25.12.6'

  > root@OpenWrt:~# which ssh
  > /usr/bin/ssh
  > root@OpenWrt:~# ssh -V
  > Dropbear v2025.89
  > root@OpenWrt:~# ldd /usr/bin/ssh
  >  /lib/ld-musl-aarch64.so.1 (0x7f8e120000)
  >  libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x7f8e0b8000)
  >  libc.so => /lib/ld-musl-aarch64.so.1 (0x7f8e120000)

  > root@OpenWrt:~# which dropbear
  > /usr/sbin/dropbear
  > root@OpenWrt:~# dropbear -V
  > Dropbear v2025.89
  > root@OpenWrt:~# ldd /usr/sbin/dropbear
  >  /lib/ld-musl-aarch64.so.1 (0x7fbb900000)
  >  libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x7fbb898000)
  >  libc.so => /lib/ld-musl-aarch64.so.1 (0x7fbb900000)

  > root@OpenWrt:~# apk search openssh
  > openssh-client-10.6_p1-r2
  > openssh-client-utils-10.6_p1-r2
  > openssh-keygen-10.6_p1-r2
  > openssh-moduli-10.6_p1-r2
  > openssh-server-10.6_p1-r2
  > openssh-server-pam-10.6_p1-r2
  > openssh-sftp-avahi-service-10.6_p1-r2
  > openssh-sftp-client-10.6_p1-r2
  > openssh-sftp-server-10.6_p1-r2
  > openssh-sk-helper-10.6_p1-r2

gregor@krekrm3 ~ % ssh root@192.168.1.1 root@192.168.1.1's password: Permission denied, please try again. root@192.168.1.1's password: Permission denied, please try again. root@192.168.1.1's password: Connection closed by 192.168.1.1 port 22

Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.

In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?

Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.

I will activate one for people to play around with for today:

    sftp hn@ping.ohftp.org

    PasswordAuthentication yes
    PermitEmptyPasswords yes

get so*

Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.

Edit 2: You really don't need to look very hard to find public container examples at million-download scale.

And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.

If you think about it that’s the whole idea of neural nets

We aren’t smart enough to design the function we want by hand as programmers so we need to learn it through weights

This lesson just keeps recurring in many different ways

This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.

This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?

Never enabled sshd there... except on my PostmarketOS PineBook Pro... oh shit!

Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug

You're saying it as if humans can't write that too.

ETA: I could trigger it. You need to enable passwords and add a second user with a valid password, then git@ works. Obviously this still means the default forgejo is safe as you say.

[0] https://en.wikipedia.org/wiki/Eternal_September

[1] https://git.musl-libc.org/cgit/musl/commit/?id=b3c4cc121f70f...

I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.

OpenBSD probably did (which doesn't use musl), RedHat probably did (or some other distro using glibc), but they all don't use musl.

> - sshd compiled with musl

So alpine as well as a bunch of other embedded Linuxes. Huge install base.

> - not using PAM

What you get if you try to have a minimal setup. Default sshd config if you install from source.

> - password logins enabled

The default. Super common.

> - at least one account with '*' as the password "hash"

Almost every Linux system.

> - at least one OTHER account with a valid password

Almost every Linux system.

So sure, there are caveats, but they cover a massive install base.

This is a super bad bug

Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.

This isn't going to end well.

---

Addendum (which may or may not be related to the above) --

Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.

The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.

Had to quickly scan the rest of the homepage of HN to confirm.

You had me there :D

LLMs will find it.

They work endlessly at finding the 'and' scenarios.

This should have been closed up a long time ago.

What? Pretty sure Cloudflare detected widespread attacks almost immediately after it was published. We are probably lucky that not much more happened because of responsible disclosure and the large media attention led to widespread admin awareness.

You can use \ to escape formatting.

- I found a way to get AI to help me while I made this

- I asked AI to make this

The people you're objecting to are complaining about the latter, this is an example of the former.

Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).

1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).

2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.