Degraded availability for GitHub Actions, GitHub Pages and Webhooks

The latest problems correlate pretty closely when they tried to migrate the entire platform onto azure in an insane timeline, something like ~1 year (for a project of that size, that's borderline insane).

These are incredibly complex and massive systems, they do not fail instantly. The failures happen incrementally until they become insurmountable and the system collapse. Incompetence doesn't show its head immediately, it can take quite a while.

Either way, it's clear the acquisition has not helped github in any meaningful way. GitHub is owned by a multi trillion dollar transnational corporation. They have enough money, resources, and human labor to achieve whatever they want, but this is what they want and this is what they've achieved.

Agreed. It's not at all clear that GitHub as an independent company would be doing much better at dealing with this new world than GitHub within Microsoft is doing.

> I'm not sure I would agree that the core reason in this case is "the acquisition"

Would data change your mind?

https://damrnelson.github.io/github-historical-uptime/

It doesn't matter when GitHub has been acquired. The only thing that matters is whether it's part of a broader pattern, which it is. Remember Skype?

Would be very interested in reading that, do you have a link or a hint so i can find it?

it's almost like they should have a test environment with proper load testing...

Great ideas if working remote, can’t imagine busting out my Steam library in the office though.

That’s a good way to get blamed and let go.

But sometimes it’s a catch 22. I’ve had numerous times where systems simply didn’t work.

I brought it to my managers attention and they didn’t care.

Then later I leave in disgust or get fired. All my fault. I should have used my magical powers to convince my manager he’s wrong while not making him feel stupid.

Ultimately if you’re on the titanic it’s easier to grab a life boat vs trying to avoid the ice berg.

Isn't this basically https://xkcd.com/303/ but "Github is down"?

(Man, unbelievable the mileage we're getting out of this comic.)

I mean this really seriously. Maybe not a game on your computer, that's unprofessional (unless your TL/manager does it first). But

> Not your fault if the company can't provide the tools to do the job, go find something better to do.

is just Truth. What should I do in this case to best deliver value to the company, fly to SF and hand Github a ringbound volume of my changeset? Patch our corporate network so we can interface to Github via RFC 1149?

More "professional" ideas:

- read a book; a novel or TAOCP for nerdmaxxing

- play a game on your phone

- learn to draw. Draw hands!

The attention for ideas like that usually fades as quickly as it came after the downtime is over.

After we spent a couple of years ripping and replacing the previous solution? You may have a point.

I guess I can embrace the corporate way, repackage the old solution with a few improvements as something brand new, and pitch to execs.

> which creates very different access patterns from how humans use GitHub.

it can

> They could, hypothetically, filter bot traffic;

Nobody seems to be able to do this reliably for a whole huge list of reasons, including: the LLMs themselves can simply mimic human access patterns, timing, and behavior.

No, their decision to focus on a move to azure is the indicator, as the reliability issues started shortly after that, but before their boom in agentic commits

Claude/codex has made the stickiness much less of a problem. We just say "no new repos in GitHub" and then put agents on refactoring the existing repos for GitLab compatibility.

Self-hosted $whatever, managed your way. CI/CD doesn't have to be dog-slow or ever go down when you need it, only when you do your upgrades yourself, as god intended it.

I personally use Forgejo + their built-in Actions stuff (mostly GitHub compatible), have one old Linux desktop as a server which runs Linux + Windows runners, one Mac Mini for macOS/iOS builds, using NixOS so I don't have to remember how things work, and it just keeps running 24/7, always there when I need it.

Forgejo, and Gitea are decent alternatives if you want to fully move off Github! If you are hoping to move away from only Github Actions, I've been working on a ~drop-in replacement for their control-plane + runner for the past few months. Each each job runs in a cross-platform vm so you can run your ci locally and update your checks. It's still beta, so I certainly wouldnt use it in prod just yet, but it's great if you want to run some of your CI locally. it allows me to extend it where possible to introduce some new features like JIT pipelines, improve performance or security. https://github.com/preloopdev/preloop if anyone's curious!

The alternative is, I’m guessing, probably any other hosted git + CI/CD service that doesn’t offer a free tier to zillions of users (human and LLM alike) so that their computing capacity isn’t in a constant arms race with growth.

What makes it sticky? I just don't see it. The declarative Github Actions manifests?

I imagine very few people use the wiki, because it's so subpar.

The issues are a problem, but this is why I always thought issues should be a core part of the Git repo, checked into it, sort of how Fossil does it. And so should the wiki, for that matter.

Absolutely anything? GitHub slowly became a standard through market share. Concourse CI is much better, in fact nearly anything is better than GitHub as it doesn't have first class container support.

See, my main issue is with Actions. If that breaks, there's little alternative for the more complex flows. Minor infrastructure changes could be done manually through AWS, but complex deployments would have to be replaced. That's a big task even with AI.

The pull request system and related machinery is useful, but Gitlab and the ilk can do the same. Even self-hosted solutions like Gitea. But I don't particularly have a problem with that living in Github. If there's an outage, break glass, force push to main.

The actual Git repository hosting part is trivial. If that breaks, Git is distributed anyways, push and pull to any random remote storage. All you need is SSH. The main headache is that automations would break as they are usually hard-coded to a given origin. That is solvable in a pinch, and we have setup mirrors in the past.

It really depends on your use-case and how much you are tied to them (Github authentication?)

swarmfile was built to deal with this type of problem

[flagged]

Churn to what exactly?

You can free tier but still sensible rate limits and QoS policy. You can have separate region or cluster for your paying customers. Finally you can spend time re-architecting GitHub Actions or just aquehire one of many competitors that do better co/cd.

Where are the sharks, though? I just see minnows. GitLab has a lot of outages/brown-outs on a much smaller load and Gitea is so confident in their ability to handle load that they host their own development on GitHub.

Because East River Source Control hasn't gotten back to me yet.

You are basically saying that you, from the outside, have solved the problem perfectly and the whole team of engineers at GitHub just don't know about this one simple solution you are proposing.

Have you considered that you don't have the full context of the problem and it's not "just" a few small tweaks?

Then just reduce free tier if they cannot handle. I paying for a service but I get best effort freeware quality.

What if it's the paying customers causing the outages...

Yes. An acknowledgement of the issue should be fine, even if they don’t know why it happened. If a random dev with one PR can see that no jobs are picked up, GitHub should see it as well and add an entry to their status “hey, actions aren’t getting picked up, we are looking into it”. I wasted 20 minutes looking into out actions config looking got an issue, thought maybe we messed something up, but in turned out it was GitHub.

Yes then you know the problem is not in your set up and so do not have to waste time looking into that possibility.

The status page should detect the thing they are looking into. Kinda like a Check Engine light or gauge reading notifies you to investigate a problem with your engine.

swarmfile supports git, self-hosted runners, is self-hosted swarmfile.com/public/swarmfile/swarmfile-server

You could self-host GHE. Or Gitlab. Or Forejo.

No https://cursor.com/blog/git-at-any-scale

No, it's a separate forge. You can choose to mirror repos between Github and Origin, but you can also have repos on just Origin and w/ no dependency on Github.

GitLab is often recommended when GitHub issues occur, but GitLab Cloud is not any better in terms of reliability and has plenty of outages https://status.gitlab.com/pages/history/5b36dc6502d06804c083...

GitHub also has more traffic, compare to GitLab

Community sentiment on GitLab has been poor. I want something newer and better like Cursor's.

So you are saying paying 25K isn't enough we need to pay more.. oof I got it, I am too broke for this world.