Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

Building the right incentives around that can be tricky, those incentives need to ensure the highest levels of management aren't themselves disincentivising their directs & their departments from surfacing pain & problems - but it's also pretty common for people to mask those signals purely out of a well-intentioned desire to help. It's important to coach people on the idea that in large group sizes, it's more efficient to let certain kinds of problems play out and not be so reactive to them.

Too many companies ground their performance incentives & processes around oversimplified ideas that don't match the reality of human behaviour

My colleague in the IT department had one idea, replace our commercial certificates with Let's Encrypt and drop the EV requirement. In total he'd stand to get a bonus of a little over €2000. He never got the money, because things like that was part of his job apparently.

When things come up with other teams, you’ll have a catalog of tasks that were done to show why you didn’t have the same issue. The work was done, just at a better time to avoid downtime.

E.g. that time a central media controls power supply broke down which would have made using one of the most prestigious rooms impossible. I fixed it myself by swapping in a spare power supply from a used unit, then went on to remind them twice a year that we are now living on borrowed time and I take no responsibilities if a fault I predict to happen and get no funds to fix will in fact happen. 4 years later I got the funds.

Having stuff costs money. Everybody wants to invest funds once, but nobody wants to keep paying for maintenance.

When things run smoothly, very few people notice. When things break, everyone notices

You can't fix this. Out of sight, out of mind. It is hard-wired into us. It's all about the optics, and will always be.

It’s extremely advanced technology, though, and most CEOs would rather rent seek / camp than give up some decision-making power (and very few are even aware it’s possible).

s/in this industry//

I disagree with the implied idea here that "engineers are better managers". The solution is to have good management, not to assume that "engineers are better managers". I have seen good and bad managers, and in both groups there were engineers and non-engineers.

Alas, for many parts of society there is a large amount of people that would rather be reactive than proactive. It means it is easier today but harder long term.

We've become too comfortable, since actual toil is no longer seen in the company: Manufacturing is overseas, customer support is overseas, logistics is an afterthought with established guarantees. Thus we want the mild weather and smooth meetings. If your engineering team is too smooth, maybe you should already branch out to help other related but "struggling" teams to get your hands dirty and noticed.

"Accounted for X situation" "Added gaurdrails to protect against Y"

When working as a business analyst i have to do this sort of thing all hte time or else id get no credit for half my work

It’s not a problem in this industry, it’s a problem everywhere.

> I'm not sure how we solve this, other than having management come from engineering.

You mean the engineers who are causing the chaos you’re complaining about?

Engineers aren’t some magic group of people who know better than others - we’re just as fallible as other people.

This is a very game-able system, and I'd wager a decent amount that any senior engineers on those teams know exactly what they are doing. In a lot of (broken, but aren't they all) management structures, it's better to be seen to swoop in with the save than to quietly fix it ahead of time.

And if your management is structuring rewards like this, it leads to your seniors anticipating a bunch of these failures, lining up 90% of the fix before hand, so that they can jump on the oncall escalation with a 100% "Hail Mary" of a fix...

Now I'm at the other kind of place and it sucks. They'll fire the performative engineers though during layoff season. It's almost like they like playing politics until it really matters.

This is the same thing. We need to reward things never going wrong as a society since this is pervasive.

I finally moved on to be an IC. Same story, same pressure :) You need to present to directors not because they need to know, but because your managers have a quota of N presentations per quarter, and if you back out, someone else needs to step up.

Needless to say my productivity reduces by half and sometimes to almost zero during the week or fortnight of presentations every quarter.

Given the whole point of management is to work to ensure their own survival and growth, it would in their interest to kill genuine competition when its coming up.

Who wants to raise their new competition and lose to them, no one!

Heroes are lauded even if they solve problems they themselves are the cause of - which is conveniently either forgotten or denied - or they are solving non-issues that are deemed important by the ignorami-class. Politics, for example, is utterly dominated by this dynamic.

It's the first instinct: let the expert run the show. However, one of the (many) ways to let a complex project fall apart completely is to hand over full control to engineers. I'm one myself, but I know what I'm good at and what not. Dunning-Kruger is often mentioned in these discussions, but don't forget it also applies to engineers that often lack any management or leadership experience of any appreciable kind. They vastly overestimate their ability to handle management and organization-wide issues and tend to not only miss the forest for the trees but actually miss the trees for the leaves.

"Unix: A History and a Memoir" by Brian Kernighan actually mentions how proper management was crucial to their success. It's a detail that's frequently conveniently forgotten by the engineers who think themselves better than the "suits". For the record, I don't claim engineers are the primary problem, but it's not just management's either. Quotes like "who holds the company standing" and "who understands how to double click" are enormous smells and IMO make quite clear what's happening here.

I don't have ready-made solutions unfortunately, but I do wish we would look further than "it's the suits". It's a systemic, human problem that I believe is a natural result of operating under informational constraints and, very human, cognitive biases on all sides.

"My second brother casually fixes bugs as soon as they appear, so his skills are known to the entire tech department.

"As for me, I rush around everyday putting out fires everywhere, so everyone in the company knows of me."

the article is from June 2024. this is "recently"?

("I have made this longer than usual, only because I have not had the time to make it shorter.")

Blaise Pascal

The director then clearly advised that they should use the complicated way because that's how you get published: not because you're clever, but because your solutions sound complicated.

It resonates perfectly with your comment and it's an unfortunate reality that most people don't bother for beautiful solutions and praise complicated processes. That's how we neded up with bureaucracy, probably :D

But when someone comes up with something simple but effective, it always looks so obvious in retrospect.

  H: So, Watson.
  W: Hmm.
  H: You do not propose to invest in South African securities?
  W: How on earth do you know that?
  H: Now, confess, you are utterly taken aback.
  W: I am!
  H: I should make you sign a paper to that effect.
  W: Why?
  H: Because in a few minutes you will say it is all so absurdly simple.
  W: I should say nothing of the kind!
  H: You see, my dear Watson, it is not really difficult to construct a series of inferences, each dependent upon its predecessor and each simple in itself. If, after doing so, one simply knocks out the central inferences and presents one's audience with the starting point and the conclusion, one may produce a startling, though possibly a meretricious, effect.
  H: I can tell by an inspection of the groove between your left forefinger and thumb, that you have decided not to invest your small capital in the gold fields.
  W: I can see no connection.
  H: Very likely not; but I can quickly give you a close connection.
  H: Here are the missing links in the very simple chain: You had chalk between your forefinger and thumb when you returned from the club last night. You put chalk there when you play billiards, to ease the cue. You never play billiards except with Thurston. Now, Thurston, you told me, four weeks ago, had an option on some South African security which expired in a month, and which he desired you to share with him. Your checkbook is locked in my drawer, and you have not asked for the key. So, you do not propose to invest your money in that manner.
  W: How absurdly simple!
  H: Quite so. Every problem is absurdly simple when it is explained to you.

In the long run, IME, you'll be recognized either by management or your peers if you keep doing that over and over again.

  > elegant solutions

And of course that's true. We push for speed, absent of direction, while praising velocity. To be honest, at this point I'm disappointed the engineers gave up and just started becoming business people.

"This is what my regular customers pay me. If I hired one of my friends or relatives I see it as my duty to pay them at least what they are worth, this is the way you raised me."

I believe this to be true btw. If someone is really your friend, you want them to do well and that means you pay what they usually get or you don't bother them and get someone else.

Note that there is also the flip side of the coin, people who spend all their time worrying about things that never happen, so it's not like you can just reward a defensive attitude things are more complicated than that.

And people ask why I hate humans.

That's what the moneys for ...

If online comments are anything to go by, I'm not alone.

If you're in the Bay Area and you can get a Sonic fiber connection, I would highly recommend them over AT&T/Comcast/etc.

Also, telsa self-driving. yes, we know about the greatly publicized accidents, or the tweets of the founder, but the avoided incidents not so much.

Originally it was engineers from the top down, but over the last 15-20 years those leaders with engineering backgrounds have retired and been replaced by non-engineer MBA's. And the more I look around, the more I see that as a common trope is the US.

https://www.boeing.com/company/bios/kelly-ortberg

  When the Master governs, the people
  are hardly aware that he exists.
  Next best is a leader who is loved.
  Next, one who is feared.
  The worst is one who is despised.

  If you don’t trust the people,
  you make them untrustworthy.

  The Master doesn’t talk, he acts.
  When his work is done,
  the people say, “Amazing:
  we did it, all by ourselves!”

Alas, that doesn't always fly with the populace.

The assertion may have been unfounded, but I think it's just as unreasonable to assert the opposite. Bugs have cascading effects and in a sufficiently complex piece of software they can create chaos with unpredictable outcomes.

I earned my first house deposit helping the team fixing the water and gas company in Wales, UK. Their entire system was running off a set of COBOL programs on a mainframe, none of which had been properly documented over the years, and the whole thing used 2-digit dates. It would have caused actual deaths if not fixed; everything would have shut down, and no water and no heating in a British winter is potentially lethal. And then it would have sent everyone in Wales a bill for 100 years of water and gas.

They were bribing retired software devs to come out of retirement with huge stacks of money, because that was cheaper than training new COBOL devs and getting them familiar with the spaghetti system.

It worked, no-one died, life went on. So obviously it was all fake rolls eyes

People are gonna criticize by saying "see? it was an overreaction to the problem, since there's not been many homicides at all!", when in fact the homicides were prevented by fixing the original problem. Same way with the electrician: "how much are you gonna charge again? And you're charging for a fix to a problem that didn't happen yet? Nah, I'll call you when the problem happens".

Its maddening.

Not if nobody knew he'd fixed it.

https://en.wikipedia.org/wiki/Stanislav_Petrov#Aftermath

> Petrov underwent intense questioning by his superiors about his judgment. Initially, he was praised for his decision.[2] Colonel-general Yuri Votintsev, the then-commander of the Soviet Air Defense's Missile Defense Units, who was the first to hear Petrov's report of the incident (and the first to reveal it to the public in the 1990s), states that Petrov's "correct actions" were "duly noted".[2] Petrov himself states he was initially praised by Votintsev and promised a reward,[2][22] but recalls that he was also reprimanded for improper filing of paperwork because he had not described the incident in the war diary.[22][23]

> Petrov has said that he was neither rewarded nor punished for his actions.[24] According to Petrov, he received no reward because the incident and other bugs found in the missile detection system embarrassed his superiors and the scientists who were responsible for it, so that if he had been officially rewarded, they would have had to be punished.[2][24][22][23] He was reassigned to a less sensitive post,[23] took early retirement (although he emphasized that he was not "forced out" of the army),[22] and suffered a nervous breakdown.[23]

Then we soon see non-technical people start leading the same, pushing for some people to be recognized for this every sprint. Meaningless recognition starts coming in. The process fades out in just a couple weeks.

The problem is there are these people in the mix, often leading, who do not understand.

Which loop it belongs to in the model is left as an exercise for the reader.