The development pipeline is a production system

For 26 years I have tried and failed to explain this to my different managers at so many different jobs. I gave up.

This is correct, but there's additional nuance here, which is the security of your development pipeline plays a part in the production system too.

Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.

( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )

I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra?

Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?

Agreed. I've started calling our non-production infrastructure environments 'engineering' rather than dev, and nobody's invited. We'll pull in platform teams that are exposed to this shame environmental offset issue, but no customers.

It's bigger than that even. Ensuring that the code you are shipping has not been tampered with, for example.

> In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages.

In my experience, we treat "not being able to deploy to prod" as an warning light, because our ability to fix an actual issue or outage in prod is impacted.

i.e. if there is a prod issue, then we can often ether fix it in code, or roll back to an earlier release. But we need the deploy pipeline to be working in order to do that. The longer a pipeline outage persists over time and across multiple services, the more likely it is that there will be some impact.

It's like driving without a seatbelt - you're fine so long as nothing else goes wrong, so it's not recommended as a state to stay in.

Worked for a company with a 3D web application (among other things.)

Those things can't be tested with Playwright et al. So the team had a "every pull request gets an instance like pr143.company.org" pipeline where QA approved. They must still have it.

The other companies tested "on demand."

We have a QA team but it’s small in comparison to the dev team. We didn’t have one before as we thought automated tests and “good developers” who take testing seriously was enough. Unfortunately, even with extreme automation and huge test suites, our bug count per release was unacceptable. Having a QA team was a big help in fixing that. We have two weeks of testing before a release goes out. But we sell a security product so for us bugs must be far between. If you just deploy web apps and you can update every hour, I guess you don’t need it as much.

Yes. We call them BAs and they have deep product knowledge, so they also get used for training, escalated support requests, estimations on effort required in new integrations, etc.

Here in Finland my company has a dedicated testing team that's almost at large at the actual engineering staff. We sell a combined hardware-software stack, though, and it's basically a contractual requirement in our industry. They are very helpful to be on good terms with when working on a given project, they often know ways to work with the gestalt I never would have guessed in a million years.

> Genuine question, does anyone here ITT working in software still have dedicated QA?

A few years ago I worked at a company who had a team of QAs dedicated to do manual acceptance tests. Up until the day where a critical regression went unnoticed and it turned out QAs were going through the test suite without actually checking the behavior. Following that the company got rid of all QAs except the software engineer in testing who was responsible for the whole automated test infrastructure, and instead of QAs the team held test days to go through the same manual test suite while we progressively automated them away.

And the whole process improved all across the board.

We have 2 QA and I’m grateful every day for them. The devs generate slop at light speed but nothing gets passed those guys, they’re fantastic.

Different country probably, here QA is still often part of the team. More often than not as part of a team with a title like "test automation engineer".

Yes. I work in games and we heavily rely on manual QA. Too heavily, really.

My experience is 50/50 - half have dropped them, half still have them.

I also don't find most QA to be worthwhile compared to a good, quality focused dev.

There are several cultural issues with how QA are treated (low pay) and how they think they're expected to behave (e.g. hands off the code, hands off types, hands off unit tests) which make them less useful than a dev who is really good at engineering for reliability.

Where we had good engineering practices and that type of "black box" QA I noticed that product tended to find the majority of bugs, followed by devs, followed by customers, followed by QA.

Only when engineering practices sucked and they were used as a crutch by devs did they really catch a lot of bugs.

Of course, in these days of vibe coded madness they're probably getting more useful, not less.

The problem with QA is that from the POV of management, all they do is find issues and delay releases. If the QA team did their jobs well, then once all of the reported bugs are found and fixed (passes QA), the software runs fine and there are no future problems. This sounds good in theory, but a good release doesn't get the credit it deserves - it's "business as usual".

A buggy release is fine, if it came out faster, and costed less. The bugs in the release can be fixed in the future, and the "heroic" fixes are "creditworthy".

Therefore, over time, the QA team is politically relegated as a "useless" team in the department, and thus given how politics play out, they get sidelined until the entire team is gone.

When we do updates, I mandate that our teams do blue/green deployments meaning they setup the exact VM side by side with the actual production data and test with the new version of software (eg GitLab) before doing the DNS flip. This is on prem. It’s an ephemeral sandbox.

Not seeing this get traction, but this is exactly what I thought while reading this. The problem with calling something "a" top priority is you can only have one top priority. That's how priority works. A customer facing outage is a higher priority than a company-wide developer tool outage, which is higher priority than a single developer's personal workstation failing, etc. etc. That doesn't mean the lower-priority things don't matter, but you can't make any and every failure of anything an all hands on deck situation. That's a certain path to burning out your entire company.

That sounds nice and pleasantly confirms our biases, but it's blatantly false.

Existence of recruiters and recruitment companies as separate job entities is a proof positive that you do not, in fact, need a working recruitment pipeline.

If anything, recruitment in our industry was always broken. And yet here we are.

> CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you.

To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be in control of the dependencies you as a company rely on and to not unnecessarily put load on free third party repositories.

Had not heard of your solution. More often than not I come across Harbor, Nexus or things like GCPs internal stuff.

The fix is easy and ubiquitous. You host your own artifact repository and docker registry. This also makes sure your supply chain remains easily auditable.

Mechanized production was designed by people to produce commodities and was designed with planned tradeoffs, tolerances, and failure modes. That’s not what agents are, though I get why an analogy to the Industrial Revolution is useful for CEOs.

Having spent too many years in CI/CD I'd agree but there is an even better solution; decouple deployments from releases. Make it possible to deploy without releasing, and release without deploying. (An implementation of this could be, for example: feature flags.)

> What's the confusion here?

Acknowledging that developers are in fact customers, is often where it comes undone in my experience.

In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.

Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.

[dead]

2:34 AM

zZzzZ...

2:35 AM

ALERT #42 - CRITICAL

!!!

ALERT #42 - CRITICAL - Joan forgot feir password to npm

Help feir!

This is a great way to encourage your devs to jump ship as soon as anything even a little better comes up.

It's trendy to act like devs are fungible resources or that AI will replace them, but the truth is that hiring smart people, paying them lots of money, and then forcing them to spend half their time dealing with prod issues that they're not empowered to fix is a staggering waste of resources.

Since deployment or testing issues are mostly invisible to upper management (who never have to spend half of THEIR time fixing them) this doesn't come with any recognition or promotions, so the only thing left to absorb the impact is morale and goodwill towards the company.

Sadly, engineer-driven companies are increasingly rare.