Salesforce Global Outage

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

Have you tried turning it off and then on again?

> We're no longer pursuing restarts as a path to remediation.

Oh you have

Unplanned outage timing is never good but this is really not good.

https://www.salesforce.com/dreamforce/

Sept 15-17

Cause: Legacy Salesforce login service got into a resource-exhaustion cascade.

Fix: Rolling some unspecified fix they proved in testing out over the fleet seemingly very slowly (After their earlier attempts to roll something out faster failed).

Details at https://status.salesforce.com/incidents/20004433

That status page is the most salesforce thing ever.

Scroll down. >_<

You can click on any of the instances and then the service that is down to read the updates. It’s not 100% clear but some sort of issue with a “legacy login service”. The latest updates say a fix is rolling out.

At this point OpenAI really ought to let us know when they're testing again.

You better bet someone started their agents with a prompt "Make a salesforce clone but with 100% uptime"

  ⎿  You've hit your session limit · resets 2:53am (48°52.6′S, 123°23.6′W Etc/GMT+8)
  /upgrade to increase your usage limit.

This is certainly a unique status page.

It's dns isn't it

Haven't they got some kind of new fancy ai interface they can use to fix it?

Perfect timing with Dreamforce this week.

Here is link to incident details: https://status.salesforce.com/incidents/20004433

My gut instinct is that this is about when all of their on prem servers were EOL and their /public cloud solution was required. This must have had something to do with that

Leetcode developers win again

Seems it is back up now. Damn.

ClaudeForce in action!

Dreamforce, wake up. You’ve overslept!

Obligatory https://xkcd.com/2347/

Could it be people doing Claude/GPT automations and they just can't handle it?

Wow, the intern must have tripped over a very big power cable this time

So I guess today everyone gets actual work done

Sam speaks at Salesforce.

Salesforce goes down.

No causation here...move on.

And nothing of value was lost. God, I hate everything about Salesforce. Sometimes I have to integrate against their services, and it is always a pain, not to mention what the core project actually is: optimization of marketing and spam.

Ah yes. Exactly what a status page should look like: an endless list of random ID’s that don’t mean anything and no information whatsoever

At least salesforce is consistent with their design language

It’s akways DNS or login

Was it DNS? Any guesses? :)

Remind me please, what are folks currently paying per seat for this glorified CRUD app?

Never before in the history of global compute outages was so little lost by so many down servers, whose purpose was known to so few.

Kind of ironic. Salesforce is basically one of the major spiritual grandfathers of Slop. It is not uncommon in production systems to find that objects like Contact and Account have hundreds of custom fields. Sometimes, you find out that several of them have the same meaning and semantics, but were used at different times. Digging out you discover that some Marketing guy that used to work at the company did some task in a certain way that was lost when he was gone, and then a few months later his substitute had the same need and went ahead and created the same field with a slightly different name.

Doing data engineering work with Salesforce data is an exercise on archeology, psychology and organizational politics.

Slop is basically the ontological and teleological philosophy behind Salesforce very existence. Despite the official discourse that the "No Software" meant no infrastructure, no toil with updates and configuration, the subtext as intended for executives was very clear: "No need for you to be blocked by those pricks from engineering and their stupid, bureaucratic and gatekeeping rules".

"No software" was a call-to-arms to a certain subset of managers that were radicalized by Nicholas Carr's 2023 HBR article "IT Doesn't matter". It doesn't matter that Carr was a journalist and a writer with a masters in English that has never ever run even a small bodega, or has never managed an IT department. Anti-intellectualism and the abundance of capital brought in by the petrodollar that allowed the US government to run deficits year by year while exporting the ensuing inflationary effects to rest of world, would ensure that this message would ressonate and then even be amplified during the years of ZIRP and the Baillouts. Play fast and loose, first come, first served, a rising tide rises all boats and all that jazz. Wall Street favors bold, and the heck with the long term! This quarter will only live once!

Frankly, this is just poetic justice: Kill by slop, be killed by slop.

"Trust just got personal!"

At least now we can figure out what Salesforce does.

[dead]

[flagged]

[dead]

[dead]

[dead]

[flagged]

[flagged]