eMudhra to exit global SSL/TLS certificate business

On Mozilla’s tracker, eMudhra said the keys were created on 30 April 2025 in a key generation ceremony, a recorded procedure in which keys are produced inside hardware security modules (HSMs), tamper-resistant devices built to hold keys. The keys were meant to be moved to data centres in the United States and Europe for customers needing data stored in those regions. To move them, the company added a step to ‘wrap’ each key, or encrypt it for transfer, and ‘unwrap’ it at the other end. eMudhra said a flaw in the software script created a second, weaker set of keys during this step, and because the two sets had similar names, staff used the weaker ones by mistake. It said human errors occurred at multiple steps.

source:

What's the eMudhra security scare that has rattled the internet?

https://theprint.in/tech/whats-the-emudhra-security-scare-th...

[1] https://www.youtube.com/watch?v=hXUqwuzcGeU

It would be like me practicing surgery 1 hour a week and saying "well surgery only has a very small place in my 80-hour work week" in response to flagrant failure of medical care.

>According to eMudhra, proper strong keys were created in the HSM at the key ceremony on 30-04-2025. The script had been changed to test wrapping and unwrapping keys, because the intermediates were to be moved to data centres in the US and Europe, and a bug in that change created a second set of weak keys. The keys had almost identical names, and the weak ones were used for the certificate requests. The same request was signed by eMudhra's own root and sent to Sectigo, which cross-signed it. The strong keys never made it into any certificate (Bugzilla).

https://www.fairssl.se/en/articles/emudhra-ca-noegler-knaekk...

If they expect people to still trust them on their other services they have obviously no clue on how Trust works, meaning they shouldn't be in the game in the frist place.

As a trust provider you get one chance. Squander it and you're out. That is how it should be.

In any legal dispute involving them, Exhibit A will be "look at how they ran their SSL/TLS business". That won't go well.

Unsurprising if you are familiar with the indian honor code „izzat“. To quote an old kiwifarms thread:

Hi. I didn't see this term mentioned in the thread, so I wanted to share this all with you. It's about Izzat. Some posts have mentioned 'saving face', which is Izzat, but I wanted to elaborate on it.

Izzat has no direct translation into English. We only have terms that can broach the same concept such as 'honour' or 'reputation' or 'face'. Izzat is so much more than that. It's a zero-sum game of collective honour shared by whole groups of people, all of whom take it very, very seriously. A system like this isn't just foreign to Enlightenment values, but I'd argue it's antithetical to every sensible form of governance on the planet. It will destroy any system that assumes good faith.

Izzat conflicts are not about who is right and who is wrong. It's about who wins and who loses. This means it's a zero-sum game where just about any action is justified (including murder) to restore the lost Izzat. Izzat is a limited social currency and the easiest way to get it is to take it from someone else. Winning is righteous in Izzat. Losing is unrighteous. This means that if someone plays the game of izzat well enough, they can get away with just about anything (murder, rape, scamming, cheating, stealing). The only morality in Izzat is the protection of your group's collective ego. The only appropriate response when your Izzat is attacked is the complete destruction of whoever insulted it.

Two people get into an argument. They might escalate, but chances are it won't be that bad. i.e. 'Sorry I broke your garden gnome.' With two Indians in an argument, the stakes are always deadly thanks to Izzat. Neither of them can back down, nor can they admit fault. Admitting fault is seen as deliberately humiliating yourself. Not only that, but because Izzat is shared, you are shaming everyone who shares your Izzat. So, admitting fault or taking responsibility for a problem is a form of social suicide. This means even if the dispute was over something completely fucking stupid or trivial (like a broken garden gnome), it could spiral into a decades long honour feud. Izzat is also the reason why police are reluctant to get involved in disputes. Because Izzat is a zero-sum game, all participants are involved in the game. If a policeman sides with one family over another in an Izzat dispute, this means that he's deliberately taking the side of that family and dishonouring the other. This marks him for retribution by the offending party. Without parties being able to be impartial, then centralized authority cannot effectively function.

Let's say someone in India complains about a broken water pipe. Instead of the problem being addressed, the official responsible for the water pipe denies it's a problem and counterattacks him instead, because daring to question his efficacy in his role was challenging his Izzat. So the official destroys the person who brought the problem up. The water pipe never gets fixed.

Izzat is also the reason why Indian managers are so infamous for hiring more Indians. It's because from the manager's perspective, he's using his position to gain an invisible social currency. Merit and actual qualifications come second to that idea. If he hired a westerner, he would not gain or even lose Izzat by doing so. He has a very strong cultural incentive not to be impartial. Meanwhile, if he can strong-arm dozens of Indians into a company, he is gaining huge quantities of Izzat and conspirators who owe an absolute debt to him.

It's not uncommon to see Indians gloat about their success in the west. And yes, izzat is very much a system that enables short term success. But the fundamental reality is the prosperity that these Indians find so attractive in the first place wouldn't exist if the west practiced something similar to Izzat. Our systems can only exist on the assumption of good faith, and not a majority of people exploiting them for destructive short term gain. On top of that, if an Indian causes the systems and companies he comes into contact with to collapse, then he can just go back to India with his plunder. He has no stakes in the long-term prosperity, functionality or stability of these systems. The stakes are completely asymmetrical in the Indian's favour.

You all know about that infamous video of that Indian scamming a food bank. Once again, winning is righteous. Losing is not. By employing 'clever' means (jugaad), he successfully extracted more resources for himself and his family. From the perspective of jugaad, a rule is not something to respect, but merely an obstacle to the Indian's own gain. Since he cheated the system and wasn't caught, he is seen as a righteous and dignified man in the Izzat framework.

I read a story in India about a man offering to pay another man to use a public toilet instead of defecating openly. The latter left and came back with several friends and beat the former to death. The problem was not that he was defecating openly, but that he was criticised for doing so. And with Izzat, it's not an eye-for-an-eye. Izzat is often an eye-for-a-whole-head. Disproportionately and brutally annihilating your enemies is the correct move to make because you take back your lost Izzat and then some. This kind of vindictiveness would be rightly seen as horrifying and disgusting in most of the world.

The king was very careful to ensure all the gatekeepers were following the process correctly. It did not occur to anybody that one of the gatekeepers could be using a dirty rag to jam their gate closed instead of a lock.

Once discovered, the wall could be trivially bypassed. The security of the whole internet was immediately broken. Rebuilding the wall to block up the dodgy gate took much effort from very many people.

Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.

Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.

  for the non technical folks:  imagine there was a password that if you guess   it, you can impersonate *any website on the internet*
  
  now imagine the password is literally the number 4

> "We set up a system where every government and a bunch of random banks and IT companies in the world holds the secret codes to everyone's encryption. Then we make them pinky swear not to abuse it, with stern talkings to if they do. And now it's all on fire, because one of them was irresponsible!"

You can know the intractability of the problem that PKI is solving, and the (worse) alternatives that have been tried, but that doesn't make this system not-insane to a normal person, and its at least a little bit the fault of the collective "us".

And it's not like it is an impossible problem; we rely on TOFU for SSH, we have remotely-attestable TPMs everywhere, ZKPPs / PAKEs exist, session keys and federation protocols are widespread. Somewhere in the set of "possible solutions" are options that don't sound unhinged when you explain how we got to this eMuhdra situation.

Your computer has a built-in list of signatures that it trusts and expects to see on the certificates, and one of those signatures was eMudhra’s. These signatures are basically just a proof that you know a secret number without revealing the number itself. If you know the secret number, you can sign the certificate.

There are an unimaginable number of secret numbers. Imagine each atom in the Great Pyramid of Giza is a secret number. And then imagine you have as many Great Pyramids of Giza as there are atoms in the universe. And then double it.

Now imagine you choose any of the numbers represented by all those atoms to sign the certificate with, and you chose 4.

It’s not technically wrong: 4 is indeed a number. But if anyone wants to guess your number, and they start at 0, it won’t take them very long to find your secret number.

Once they’ve found your secret number, they can now sign certificates pretending to be you. So someone could pretend to be your bank, and present a certificate saying “yes, I am your bank” with a signature on it that your computer trusts.

Yours is much funnier!

And when that clicks, extend the metaphor, literally, to a key with ~384 grooves.

I start by saying this is about the tech behind the S in HTTPS. I say that the main point of this is that when you talk to foo.com, it has to prove that the people behind foo.com are on the other end, so it's safe to send them your encrypted password or whatever.

Then I say a bit about how the system works by having a small number of trustworthy companies vouch for other people's servers. This doesn't need too much technical detail to get the point across.

Then I say one of those "trustworthy" companies put an encryption key of 4, where it's supposed to be a long number that's hard to crack. This makes the verification easy to forge.

Then I say that everybody who runs a server needs to remove this company's bad key, and Windows or Mac or your phone needs updates too. And that your web browser probably already quietly removed it, which removes a lot of the risk.

I feel like when I explain it this way people get it.

The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).

Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!

Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.

Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”

house collapses and people get injured.

concrete manufacture says we no longer build houses cause it’s too small of a business.

“That's the stupidest combination I've ever heard in my life! That's the kinda thing an idiot would have on his luggage!”

Then they locked it with the key you find on a chew toy for a baby.

I'm actually being serious, I do judge companies by their logo and website design taste, and more modern-looking companies also tend to pay more attention to doing things properly.

This was a good read on the situation, thanks for the link.

Is the Mozilla bugtracker usually a / the Go to party for these issues? Or just a coincidence in this case?

It would need to be a 3, 5, or 7, since a prime is required.

Also note that using small numbers like 65537 for RSA public exponents is common practice.

Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.

The "4" would be valid in hex, decimal, and even octal.

The key needs to mention which curve it is using.

They just didn't mention that now anyone can be their own Certificate Authority.

It was ~15hrs 30 minutes to first revocation

https://news.ycombinator.com/item?id=50189521

I suspect Apple published no update and is relying on the CA's CRL and/or OCSP to do its job. But it's rate-limited, the connections time out, and iOS is hard-coded to soft-fail.

Like you my trust store is months old.

That said, I found separate installs of Firefox and Microsoft Edge that accepted *all* the bad certificates without a care.

The only browser that consistently rejected all of them was — you guessed it — Internet Explorer. A browser so old it couldn't even run the test properly but it did what it needed to and blocked all the servers.

Just now on M1 Pro Safari.

On M1 Prp Firefox: none.

On iPhone 17: Safari: none, Firefox: none.

It looks like them explaining that certs are only a small part of their business did not fly with shareholders.

It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.

I can see the case on phones: the OS may no longer be supported by the vendor, so browsers shipping their own set makes sense. And on embedded there isn't (and on IOT there might not be) an OS, but then you should probably just ship the the single CA that you use, not all of them.

But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem

The problem is that those same people tend to disappear as soon as you ask "So what should we do instead?"

The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.

It raises the question why these CAs agree to participate on this forum and then complain about doing so.

1: https://bugzilla.mozilla.org/show_bug.cgi?id=1950144

You apparently don't understand Indians.

e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault

learn about Izzat and Jugaad in their "culture". That explains the problem itself, as well as the reaction ;)

I see this more as "we didn't know this was so complex and the juice no longer looks worth the squeeze."

Probably just a botched translation though

But in all honesty there are serious questions to be asked about:

a) The initial approval process for ANY CA

b) The ongoing monitoring process for ANY CA

The fact this CA was based in India ultimately has nothing to do with it if we are being completely honest here.

Clearly there is a fundamental problem with the overarching approval and monitoring process which means it could have happened anywhere. This time it just happened to be India.

Secondly, in direct reference to the "far away company" comment India does have quite strict controls on both certain technical things (speaking generally here, not specifically in relation to certs ... e.g. use of sat phones). And in general it is quite difficult (read: almost impossible) to turn up in India and do business as an external company. So there is probably a genuine need for an Indian company CA since not every body can/wants to use Let's Encrypt or whatever.

Far away from the default country?

This is a sample on what it looks like during covid lockdown : https://bugzilla.mozilla.org/show_bug.cgi?id=1728790

Will they send a "strongly worded letter"? That's the best way to make sure such an incident never occurs again. /s

Apparently they generated them like this: https://xkcd.com/221/

What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?

I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...

1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc. 2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.

That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.

The blast radius of that is every subdomain of whoever got compromised. Having only ICANN being capable of breaking the entire internet is a world of difference from what we have today, and may be the best we can do... and we should be using that, but it's not perfect either.

If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).

#1 Executive bonuses

#2 Shareholder value

...

...

--//--

...

...

#NaN customer trust

#NaN security

"Oh sorry, we erased the evidence because we no longer had use for it"

Or silently spin out a subsidiary under a new name who takes over the cert business, and redirect existing customers there for renewals.

Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.

First one I clicked on?

Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.

My trust in those vetting the CAs is completely broken now.

You might have heard of DigiCert or IdenTrust, but what, exactly, makes them trustworthy, and why eMudhra wouldn't be considered trustworthy (a week ago)?

That’s sort of the problem the CA forum (badly) addresses. The consumers of the CAs are not specifically exposed to failures of the CA they choose.

One man's "little known company with no cybersecurity muscle" is another one's "sovereign certificate infrastructure."

https://www.deccanherald.com/india/meity-forms-team-to-probe...

I think that's for the best; there are much bigger problems with DANE/DNSSEC than TLS reliability.

The reason we don't have DANE support in browsers is of course because Google decided not to implement it in Chrome, because (ultimately) the key lengths securing the root DNS server zones weren't large enough to suit them (at the time, 1024 bits):

https://www.imperialviolet.org/2015/01/17/notdane.html

And if Google wasn't going to do it, it didn't make sense for any other browser to do it either.

However, I believe the root zone keys are now longer, so I could (potentially) see Google implementing DANE in Chrome now, maybe with a Chrome-specific restriction of "TLSA records that have keys under 2048 bits in their chain of trust are ignored" (since the root zone keys are not the only keys in the chain and they want to eliminate 1024-bit RSA from Chrome). And if Google does it in Chrome, then everybody else has to do it too, so the situation flips.

I wish any company that gets sunk by this a very happy go-out-of-business.