The Dunning-Kruger effect may just be a data artefact (2020)

It’s obviously real, at least as used in conversation, whether it meets some rigorous definition I’m sure there’s an out, but we’ve all known these people. With vibe coding they’re everywhere. Is this going to be a modern “begging the question” where everyone knows what you mean but someone pipes up that actually the technical meaning is different?

  [ Unconscious Incompetence ]  -->  "You don't know what you don't know"
             ↓
  [ Conscious Incompetence ]    -->  "You know what you don't know"
             ↓
  [ Conscious Competence ]      -->  "You know, but you have to think about it"
             ↓
  [ Unconscious Competence ]    -->  "You know it so well, it's second nature"

The key claim: "Random data actually mimics the effect really well."

This makes some sense. If people are asked to guess a number between 1 and 6 and then roll a die, the people who roll low are more likely to overestimate and the people who roll high are more likely to underestimate. But the key is precisely how well random data mimics the effect.

Even if it isn’t true, it’s got the feeling of truthiness (1).

I don’t expect it to ever go out of the public consciousness. Like other things that were never real like Stockholm Syndrome I suspect it’s just stuck in the zeitgeist now.

1. https://en.wikipedia.org/wiki/Truthiness

Replication crisis. More than half of all psychology studies are not reproducible.

I'm at the point honestly, where I don't even consider psychology to be a science anymore.

Very hard to understand the meat behind all the fluff of the article, especially as the simulation code is not available, and as the presented simulated and original graphs are effectively the same (I don't see a disagreement).

It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.

And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.

A similar argument was made a couple of years ago, here's the rebuttal of the argument from back then [2022]: https://andersource.dev/2022/04/19/dk-autocorrelation.html

Claude, please write a paper proving Dunning-Kruger is false. I'm not an expert, but you'll have no difficulty finding sources as it's clearly 100% false.

The strict academic definition hasn’t followed the colloquial usage for a long time. Maybe ever:

If a specific novice is over-confident and out of their depth, we say “Dunning-Kruger”

If a specific is under-confident and performing better than their self-estimate, that’s not commonly considered Dunning Kruger, in the colloquial use. It’s called imposter syndrome, or not labeled at all.

The researchers aren’t really disagreeing with that. They found that novices had a wider range of self-estimates of their performance than experienced people. So in the novice group you were more likely to find someone who was grossly over-confident in their abilities, but you also found people who underestimated themselves.

> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.

Which doesn’t precisely contradict the idea that among novices you can find people who overestimate their skills. Which is how it’s commonly used.

So I can believe it’s a statistical wash when averaging across all subjects. But I never considered the common use of Dunning-Kruger to be applied to averaged groups of people. It was always brought out for those outliers on the long tail of the novice grout who thought didn’t even know what they didn’t know.

My favourite aspect of DK, which isn't changed by this argument really, is that people who reference it very often believe it said that low skill people are MORE confident in their abilities than high skill people, when the authors never claimed that. Which is kind of like doing a DK yourself when trying to deploy the DK findings.

Just because two graphs looks similar doesn’t mean they are “same”. Shouldn’t we subtract the random bias from the real world data and see if there is any slope remaining (or something similar)?

Is any of the raw datasets of such an experiment available? I would like to see a scatter plot of self-assesed score vs actual score instead of the data aggregated into four bins.

Hm... I vaguely remember a different article debunking the Dunning-Kruger. Basically the conclusion was that the data from the experiment shows that people's estimations of their results are all over the place, with people scoring high being actually slightly (but only slightly) more confident that they did well.

UPD: probably this one https://economicsfromthetopdown.com/2022/04/08/the-dunning-k...

The article in the post is older though

The plot in the blog post seems to be very symmetrical around 50% (to the point where there must be some identity going on). The plot from the paper seems to cross over around 75%. So the 3rd quartile still has some explaining to do, right?

Maybe the core reasons are buried in this, but the amount of empty statements makes it hard to find

There are so many strange things about the original Dunning-Kruger plot. Why use quartiles for one axis and percentile for the other? Why use higher precision for the subject's estimate, which is by definition imprecise, and lower precision for the true score, which is known precisely?

I think the only conclusion you can draw from that plot is everyone thinks they'll be in the third quartile.

The autocorrelation argument explains the graph but not the people. Anyone who's managed a team has watched this play out in real time.

While the pop-culture notion of the Dunning-Kruger Effect is "idiots don't know they're idiots," the actual results of the paper were (essentially) that F students thought they were D students, whereas the A students thought they were B students. The argument here seems to be that the original effect is explained as essentially a kind of reversion of the mean argument (people assume themselves to be more average than they are), but I don't entirely buy that--especially since the simulation results they present don't really look like the original Dunning-Kruger results, since the crossover point is in the wrong place, and that's actually kind of significant in the original analysis...

Regardless of whether Dunning-Kruger is real the solution is the same. DK concerns poor performing people who cannot accurately address their performance relative to a group. Forget DK. The bigger problem is missing objectivity, which is a very real concern. So, just measure for objectivity.

Can they measure things or do they just guess? Are they willing to seek evidence? Even if evidence is immediately available will they use it? Everybody has bias, but is their bias primarily self-oriented?

The consequences for poor objectivity are profound and measurable, but then its an invisible failure for people that struggle with this in the first place. In many industries poor objectivity can result in termination, law suits, criminal penalties, physical harm, and more. Software just seems to pretend this is vapor.

The article may not take into account the possible effect of knowing about the Dunning-Kruger effect (or cultural sayings that goes in a similar direction) may bias measurements. Before it was widely enough known it was not a factor.

Also, negative knowledge comes in two flavours, what you know that you don't know and what you don't know that you don't know. There it may be ground for that effect, but also changes in culture may affect that, specially with exposure to internet/global culture and attitudes, that may make you more aware of what you don't know, and stories of success/fail for taking the wrong approach.

I think it's mostly misapplied. The best example of Dunning-Kruger is an intelligent, competent, Ph.D. in physics thinking 9/11 was faked because "jet fuel can't melt steel", not realizing that steel loses significant tensile strength as it heats up without necessarily melting, which I think most engineers would be aware of. His great knowledge in one area blinds him to his woeful lack of knowledge in another.

The AI-Kruger effect is here today...

...when very mediocre vibe coders think they are brilliant developers....

I mean, we all remember the cases where it was true, but do you really think most people think they are good at computer programming? Or speaking Russian? Or playing the harp? Or gardening? In the vast majority of cases, people who are not skilled at something, know that they are not. There are, sure, a few people who are overconfident, but the D-K effect as generally used in conversation was always pretty obviously untrue.

I think what people need to realize is that the Dunning-Kruger effect is mostly "not real" because, on average, everyone (regardless of competence) overestimates themselves. Saying that incompetent people overestimate themselves doesn't prove Dunning-Kruger is real, because it doesn't negate the fact that competent people also do this.

This article does not make its case. He shows a graph of "random" data, and then just kind of keeps going. But that random data is the meat of the whole thing.

Cut out 60% of the useless text, and focus on explaining why random data should look like that.

So this is a case of the Dunning-Kruger Effect?

Isn't trying to discount published research with a glorified blog post the Dunning-Kruger Effect in action?

I've always found it somewhat ironic that the people who are least familiar with the actual research on the Dunning-Kruger effect tend to be the most confident in discussing it.

It's a sort of recursive Dunning-Kruger effect.

Having worked in tech my entire life, no amount of research will convince me that the Dunning-Kruger effect is not real. You might as well tell me that this isn't air that I'm breathing.

Ok. Let’s read the papers cited:

> Our results further confirm that experts are more proficient in self-assessing their abilities than novices.

I'm 100% confident it's real but I have no expertise on the subject. Checkmate, Atheists.

Regardless of whether it's real or not, there's a significant public perception that it doesn't matter. There's a real "fake it till you make it" sentiment coupled with anti-intellectualism that will make people overestimate their ability.

On that note, I don't think the "random simulation" is described well enough. If I randomly assign a self-assessment and an actual score on a test, all of the quartiles will be the same because they should be uniformly distributed. I read through the papers mentioned and in these simulations, they hard-coded the correlation, "As in the Kruger and Dunning (1999) comparison, these random variables were correlated r=0.19"[1] so of course the graph will look similar.

On the other hand, the original Kruger and Dunning comparison could have been explained through differences in expected test scoring. It looks like people who thought they would get a D (60 percentile) or an F (50 percentile) objectively scored within the ranges of a 10 to 40 percent. While this is an overestimation, perhaps further studies can instead bucket the test scores according to how we expect them on a grading scale instead.

My takeaway is that the conclusions and discussions from the Dunning-Kruger effect study are valid. At the same time, the methodology and statistical significance is different from how popular science presents it. Also, there might be better ways to measure this phenomenon, and I would be interested in understanding how those who believe college is worth it and those who don't compare on objective testing.

[1] https://www.sciencedirect.com/science/article/abs/pii/S01918...

It is absolutely real.

A lot of Dunning-Kruger specialists here, apparently.

Excessive willful/unwillful ignorance is the root cause of someone exhibiting the Dunning-Kruger Effect. We've all at some point worked or lived with someone with real illusions/delusions about their abilities, and the root of it is ignorance. There's little we can do in our workplaces to mitigate these people. Word of advice from my experience: Never co-found a vc-backed software startup with someone that's done genuine innovation....and been completely ignorant and oblivious about everything else.

Damn if only this article actually explained why you see this effect from random data. Unfortunately it doesn't seem like they understand the maths enough to know. Does anyone fancy reading those papers and giving us a TL;DR?

Maybe this is reveals more about me than anyone else but the whole usage of dunning-kruger is just another arrow in the quiver for media to talk down to a group that they dismiss because they have different priorities.

I find references to the effect in pop culture are almost always used in an insulting, smug manner.

> The Dunning-Kruger Effect Is Probably Not Real

Self-deception by any other name is still self-deception.

The Dunning-Kruger effect also applies to smart people. You don't stop when you are estimating your ability correctly. As you learn more, you gain more awareness of your ignorance and continue being conservative with your self-estimates.

But overall I think real intelligence by definition requires empathy and humility.

One has to realize that we can't know the things we don't know, which includes the fact that we can't always trust our own beliefs and opinions because we might be relying on faulty or incomplete information, or we might be suffering from a mental health problem, whether we are aware of it or not.

"As a rule, strong feelings about issues do not emerge from deep understanding." -Sloman and Fernbach

This article seems really dumb (no Dunning-Kruger joke intended).

The two lines on the graph are basically linear (for the "actual performance" the quasi-linearity is obvious by the design, for the "estimated performance" it still means that even though dumber people over-estimate their performance, all group still think they do best, when they actually do best, in a relative linear way)

And when they do their simple model (we assume they just generated "real performance" from a gaussian, then added some gaussian noise for the "performance" and another gaussian noise for the "self-assessment") they still (obviously) got two linear graphs that crossed each other.

And then they conclude that this means there is no effect, because "the graphs are eerily similar" (whatever that means)

But obviously the simple model is going to make two lines cross (in particular if you use a min(100, max(0, actual_performance + noise)) since at each extreme, then min and max will tend to skew the line). To put it simply: someone really stupid will STILL not pretend that he's "negatively stupid".

The argument "I can make a simple model without using actual humans which shows some kind of bias that vaguely ressembles the result of a paper" doesn't mean that the actual paper is wrong...

But what if that is the Dunning-Kruger effect?

I feel like the only important point in the article would be to explain how the random data was generated, yet it was relegated to the single sentence: "There was no bias in the coding that would lead these fictitious students to guess they had done really well when their actual score was very low."

Because on the surface, it doesn't make any sense for two sets of "random" numbers between 0-100 selected in pairs to deviate from each other based on whether the first number in the pair was low or not. You would not expect the first number chosen in a pair to influence the second number. Whether the first number was between 0-25 or 76-100, you would expect the second number to be about 50.

So this is obviously some sort of structured randomness that may be entirely justifiable, but the only way to find that out would be to read the two articles that this article purports to summarize for the layman. Instead there's over 1300 words of slop before this sentence, then nearly 700 words of slop after this sentence. Turns out we don't need AI for this. Speaking of random, I don't think that 2000 words is random.

-----

edit:

maybe the point of the papers is that low scorers can't underestimate their abilities - as in they literally don't have enough room? If so, that just means that the Dunning-Kruger affect is unavoidable. But the fact is that people are not choosing numbers at random, they are choosing them based on their expectations. People who got zero questions right and expected 100% are as likely as anyone else from a random number generator, and non-existent from actual people.

edit2:

OK, I've worked it out. I was making the mistake of thinking that they were evaluating absolute performance rather than relative performance. So each of the first numbers in the pair is unique. But that still leaves the fact that the random draw still predictably sits at 50% where the Dunning-Kruger data is around 65% based on the graph. Seems like norming that with the random data would give you better information.

edit3:

> In Dr. Nuhfer’s own papers [...] his team [...] showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.

How is "narrower range" not an indication of more accurate self-evaluation? With that, and since people on the higher end of the scale have less room to overestimate their standing, and people on the lower end of the scale have less room to underestimate their standing, wouldn't you expect "Dunning-Kruger"? People on the low end of the scale would have wild swings that would be gated at zero, and people on the high end of the scale would have small swings that would be gated at 100. That would lead to small underestimates at the top, and large overestimates at the bottom. More accurate self-evaluation at the top of the scale is exactly what Dunning-Kruger is about, and the direction of the mistakes is predictable if this is true.

final, tldr:

Honestly, the entire debate is garbled. People are not being asked about their performance on a test, they're being asked about their standing within a sampling of people chosen by the experimenter, something which they have no reason to know anything about other than on the experimenter's word.

I think how people interpret Dunning-Kruger, and the only interesting thing about it, is that people who have more knowledge of a subject are more accurate in their assessment of how much they know about that subject. This seems likely (but not evidently) to be true, due to the range of (relative) self-assessment error being narrower in the top quartile as compared to the bottom quartile. This is what people found intuitive and compelling.

If it is true, the top quartile would tend to small underestimation (because of the narrower range and that they can't choose numbers higher than 100) and the bottom quartile would tend to larger overestimation (because of the wider range and that they can't choose numbers lower than 1.) That the average direction of over- and underestimation is forced by the nature of the evaluation doesn't make the effect any less true.

[dead]