It’s obviously real, at least as used in conversation, whether it meets some rigorous definition I’m sure there’s an out, but we’ve all known these people. With vibe coding they’re everywhere. Is this going to be a modern “begging the question” where everyone knows what you mean but someone pipes up that actually the technical meaning is different?
The problem is that the original formulation was "most people" are unaware of being unskilled, but by now the name is used to mean "some group of people" is unaware of being unskilled. In that sense the article confirms that “a small number are", 5-6%, and therefore the effect exists in the colloquial sense but not in the scientific sense.
Mentioning the Dunning-Krueger effect incorrectly is wonderfully meta...
My experience with new CS grads was that most of them greatly overestimated what they knew, or alternatively, underestimated how much they did not know.
My experience with every person was that most of them overestimate what they know regardless of experience level. You just notice that more in new grads since its easier to tell when people are wrong about simple things than when they are wrong about more difficult things.
Even if it isn’t true, it’s got the feeling of truthiness (1).
I don’t expect it to ever go out of the public consciousness. Like other things that were never real like Stockholm Syndrome I suspect it’s just stuck in the zeitgeist now.
Very hard to understand the meat behind all the fluff of the article, especially as the simulation code is not available, and as the presented simulated and original graphs are effectively the same (I don't see a disagreement).
It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.
And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.
The strict academic definition hasn’t followed the colloquial usage for a long time. Maybe ever:
If a specific novice is over-confident and out of their depth, we say “Dunning-Kruger”
If a specific is under-confident and performing better than their self-estimate, that’s not commonly considered Dunning Kruger, in the colloquial use. It’s called imposter syndrome, or not labeled at all.
The researchers aren’t really disagreeing with that. They found that novices had a wider range of self-estimates of their performance than experienced people. So in the novice group you were more likely to find someone who was grossly over-confident in their abilities, but you also found people who underestimated themselves.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
Which doesn’t precisely contradict the idea that among novices you can find people who overestimate their skills. Which is how it’s commonly used.
So I can believe it’s a statistical wash when averaging across all subjects. But I never considered the common use of Dunning-Kruger to be applied to averaged groups of people. It was always brought out for those outliers on the long tail of the novice grout who thought didn’t even know what they didn’t know.
I wouldn’t assume the cause of the replication crisis is 100% due to a lack of rigor in psychology and the other relevant fields. These fields deal with concepts and phenomena that are often abstract and difficult to measure. In some sense, it’s just harder.
Besides, deciding what is and isn’t science is a question for the philosophy of science, not science itself.
There are so many strange things about the original Dunning-Kruger plot. Why use quartiles for one axis percentile for the other? Why use higher precision for the subject's estimate, which is by definition imprecise, and lower precision for the true score, which is known precisely?
I think the only conclusion you can draw from that plot is everyone thinks they'll be in the third quartile.
While the pop-culture notion of the Dunning-Kruger Effect is "idiots don't know they're idiots," the actual results of the paper were (essentially) that F students thought they were D students, whereas the A students thought they were B students. The argument here seems to be that the original effect is explained as essentially a kind of reversion of the mean argument (people assume themselves to be more average than they are), but I don't entirely buy that--especially since the simulation results they present don't really look like the original Dunning-Kruger results, since the crossover point is in the wrong place, and that's actually kind of significant in the original analysis...
Regardless of whether Dunning-Kruger is real the solution is the same. DK concerns poor performing people who cannot accurately address their performance relative to a group. Forget DK. The bigger problem is missing objectivity, which is a very real concern. So, just measure for objectivity.
Can they measure things or do they just guess? Are they willing to seek evidence? Even if evidence is immediately available will they use it? Everybody has bias, but is their bias primarily self-oriented?
The consequences for poor objectivity are profound and measurable, but then its an invisible failure for people that struggle with this in the first place. In many industries poor objectivity can result in termination, law suits, criminal penalties, physical harm, and more. Software just seems to pretend this is vapor.
The article may not take into account the possible effect of knowing about the Dunning-Kruger effect (or cultural sayings that goes in a similar direction) may bias measurements. Before it was widely enough known it was not a factor.
Also, negative knowledge comes in two flavours, what you know that you don't know and what you don't know that you don't know. There it may be ground for that effect, but also changes in culture may affect that, specially with exposure to internet/global culture and attitudes, that may make you more aware of what you don't know, and stories of success/fail for taking the wrong approach.
Having worked in tech my entire life, no amount of research will convince me that the Dunning-Kruger effect is not real. You might as well tell me that this isn't air that I'm breathing.
I think what people need to realize is that the Dunning-Kruger effect is mostly "not real" because, on average, everyone (regardless of competence) overestimates themselves. Saying that incompetent people overestimate themselves doesn't prove Dunning-Kruger is real, because it doesn't negate the fact that competent people also do this.
Excessive willful/unwillful ignorance is the root cause of someone exhibiting the Dunning-Kruger Effect. We've all at some point worked or lived with someone with real illusions/delusions about their abilities, and the root of it is ignorance. There's little we can do in our workplaces to mitigate these people. Word of advice from my experience: Never co-found a vc-backed software startup with someone that's done genuine innovation....and been completely ignorant and oblivious about everything else.
This article does not make it's case. He shows a graph of "random" data, and then just kind of keeps going. But that random data is the meat of the whole thing.
Cut out 60% of the useless text, and focus on explaining why random data should look like that.
Agreed. Also it seems like, if the actual test scores and perceived test scores were both sampled iid, the “perceived” line should just be flat, with everyone being at 50th percentile. The fact that the displayed graph deviates from 50 in a way that more resembles the empirical data makes it seem like a cherry-picked random sample
Do you think there's some element in published research that makes it automagically correct?
Papers saying that lead in petrol was totally safe were "published research", as were the papers saying that replacing tetraethyl lead with benzine made it safer.
Both of those turned out to be pretty majorly wrong, but they were "published research".
But really, the article seems to be going out of the way to make the author's particular point... but reads to me that the original paper is often understood... it simply shows that "specialists who are very knowledgeable about a subject are more likely to accurately identify gaps in their own knoweldge, when compared to any population less knoweldgeable on the same subject."
For example, I am apparently the most knowledgeable birder in my family. I've taken graduate-level ornithology courses, identify a fair number of N. American birds by their calls, etc. However, I recognize that I know nothing about birds compared to anyone who actually works in the field with them... I don't know enough to even estimate what I don't know.
Maybe this is reveals more about me than anyone else but the whole usage of dunning-kruger is just another arrow in the quiver for media to talk down to a group that they dismiss because they have different priorities.
I find references to the effect in pop culture are almost always used in an insulting, smug manner.
Self-deception by any other name is still self-deception.
The Dunning-Kruger effect also applies to smart people. You don't stop when you are estimating your ability correctly. As you learn more, you gain more awareness of your ignorance and continue being conservative with your self-estimates.
But overall I think real intelligence by definition requires empathy and humility.
One has to realize that we can't know the things we don't know, which includes the fact that we can't always trust our own beliefs and opinions because we might be relying on faulty or incomplete information, or we might be suffering from a mental health problem, whether we are aware of it or not.
"As a rule, strong feelings about issues do not emerge from deep understanding." -Sloman and Fernbach
It’s obviously real, at least as used in conversation, whether it meets some rigorous definition I’m sure there’s an out, but we’ve all known these people. With vibe coding they’re everywhere. Is this going to be a modern “begging the question” where everyone knows what you mean but someone pipes up that actually the technical meaning is different?
The problem is that the original formulation was "most people" are unaware of being unskilled, but by now the name is used to mean "some group of people" is unaware of being unskilled. In that sense the article confirms that “a small number are", 5-6%, and therefore the effect exists in the colloquial sense but not in the scientific sense.
Mentioning the Dunning-Krueger effect incorrectly is wonderfully meta...
I've seen at least 3 different ideas on this topic:
- people not skilled in a thing are bad at estimating their skills
- people are generally bad at estimating their skills
- people skilled in a particular areas often feel they are intellectually fit in other areas
Out of these three I feel like there's some truth in it, at least anecdotally.
> people skilled in a particular areas often feel they are intellectually fit in other areas
This I think is a separate phenomenon, maybe Nobel Disease but there might be a more general term, for example that includes celebrities.
My experience with new CS grads was that most of them greatly overestimated what they knew, or alternatively, underestimated how much they did not know.
This then boils down to people being generally bad at estimating their own level of ability.
My experience with every person was that most of them overestimate what they know regardless of experience level. You just notice that more in new grads since its easier to tell when people are wrong about simple things than when they are wrong about more difficult things.
Even if it isn’t true, it’s got the feeling of truthiness (1).
I don’t expect it to ever go out of the public consciousness. Like other things that were never real like Stockholm Syndrome I suspect it’s just stuck in the zeitgeist now.
1. https://en.wikipedia.org/wiki/Truthiness
Very hard to understand the meat behind all the fluff of the article, especially as the simulation code is not available, and as the presented simulated and original graphs are effectively the same (I don't see a disagreement).
It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.
And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.
The strict academic definition hasn’t followed the colloquial usage for a long time. Maybe ever:
If a specific novice is over-confident and out of their depth, we say “Dunning-Kruger”
If a specific is under-confident and performing better than their self-estimate, that’s not commonly considered Dunning Kruger, in the colloquial use. It’s called imposter syndrome, or not labeled at all.
The researchers aren’t really disagreeing with that. They found that novices had a wider range of self-estimates of their performance than experienced people. So in the novice group you were more likely to find someone who was grossly over-confident in their abilities, but you also found people who underestimated themselves.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
Which doesn’t precisely contradict the idea that among novices you can find people who overestimate their skills. Which is how it’s commonly used.
So I can believe it’s a statistical wash when averaging across all subjects. But I never considered the common use of Dunning-Kruger to be applied to averaged groups of people. It was always brought out for those outliers on the long tail of the novice grout who thought didn’t even know what they didn’t know.
You've hit the nail on the head, it's a conditional defenition, rather than an absolute one.
Replication crisis. More than half of all psychology studies are not reproducible.
I'm at the point honestly, where I don't even consider psychology to be a science anymore.
I wouldn’t assume the cause of the replication crisis is 100% due to a lack of rigor in psychology and the other relevant fields. These fields deal with concepts and phenomena that are often abstract and difficult to measure. In some sense, it’s just harder.
Besides, deciding what is and isn’t science is a question for the philosophy of science, not science itself.
> More than half of all psychology studies are not reproducible.
Do you have a source for this?
Maybe the core reasons are buried in this, but the amount of empty statements makes it hard to find
There are so many strange things about the original Dunning-Kruger plot. Why use quartiles for one axis percentile for the other? Why use higher precision for the subject's estimate, which is by definition imprecise, and lower precision for the true score, which is known precisely?
I think the only conclusion you can draw from that plot is everyone thinks they'll be in the third quartile.
While the pop-culture notion of the Dunning-Kruger Effect is "idiots don't know they're idiots," the actual results of the paper were (essentially) that F students thought they were D students, whereas the A students thought they were B students. The argument here seems to be that the original effect is explained as essentially a kind of reversion of the mean argument (people assume themselves to be more average than they are), but I don't entirely buy that--especially since the simulation results they present don't really look like the original Dunning-Kruger results, since the crossover point is in the wrong place, and that's actually kind of significant in the original analysis...
Regardless of whether Dunning-Kruger is real the solution is the same. DK concerns poor performing people who cannot accurately address their performance relative to a group. Forget DK. The bigger problem is missing objectivity, which is a very real concern. So, just measure for objectivity.
Can they measure things or do they just guess? Are they willing to seek evidence? Even if evidence is immediately available will they use it? Everybody has bias, but is their bias primarily self-oriented?
The consequences for poor objectivity are profound and measurable, but then its an invisible failure for people that struggle with this in the first place. In many industries poor objectivity can result in termination, law suits, criminal penalties, physical harm, and more. Software just seems to pretend this is vapor.
The article may not take into account the possible effect of knowing about the Dunning-Kruger effect (or cultural sayings that goes in a similar direction) may bias measurements. Before it was widely enough known it was not a factor.
Also, negative knowledge comes in two flavours, what you know that you don't know and what you don't know that you don't know. There it may be ground for that effect, but also changes in culture may affect that, specially with exposure to internet/global culture and attitudes, that may make you more aware of what you don't know, and stories of success/fail for taking the wrong approach.
Having worked in tech my entire life, no amount of research will convince me that the Dunning-Kruger effect is not real. You might as well tell me that this isn't air that I'm breathing.
I think what people need to realize is that the Dunning-Kruger effect is mostly "not real" because, on average, everyone (regardless of competence) overestimates themselves. Saying that incompetent people overestimate themselves doesn't prove Dunning-Kruger is real, because it doesn't negate the fact that competent people also do this.
Excessive willful/unwillful ignorance is the root cause of someone exhibiting the Dunning-Kruger Effect. We've all at some point worked or lived with someone with real illusions/delusions about their abilities, and the root of it is ignorance. There's little we can do in our workplaces to mitigate these people. Word of advice from my experience: Never co-found a vc-backed software startup with someone that's done genuine innovation....and been completely ignorant and oblivious about everything else.
This article does not make it's case. He shows a graph of "random" data, and then just kind of keeps going. But that random data is the meat of the whole thing.
Cut out 60% of the useless text, and focus on explaining why random data should look like that.
Agreed. Also it seems like, if the actual test scores and perceived test scores were both sampled iid, the “perceived” line should just be flat, with everyone being at 50th percentile. The fact that the displayed graph deviates from 50 in a way that more resembles the empirical data makes it seem like a cherry-picked random sample
Isn't trying to discount published research with a glorified blog post the Dunning-Kruger Effect in action?
Do you think there's some element in published research that makes it automagically correct?
Papers saying that lead in petrol was totally safe were "published research", as were the papers saying that replacing tetraethyl lead with benzine made it safer.
Both of those turned out to be pretty majorly wrong, but they were "published research".
I completely agree.
It is absolutely real.
So this is a case of the Dunning-Kruger Effect?
I was going to say: is this a joke?
But really, the article seems to be going out of the way to make the author's particular point... but reads to me that the original paper is often understood... it simply shows that "specialists who are very knowledgeable about a subject are more likely to accurately identify gaps in their own knoweldge, when compared to any population less knoweldgeable on the same subject."
For example, I am apparently the most knowledgeable birder in my family. I've taken graduate-level ornithology courses, identify a fair number of N. American birds by their calls, etc. However, I recognize that I know nothing about birds compared to anyone who actually works in the field with them... I don't know enough to even estimate what I don't know.
Maybe this is reveals more about me than anyone else but the whole usage of dunning-kruger is just another arrow in the quiver for media to talk down to a group that they dismiss because they have different priorities.
I find references to the effect in pop culture are almost always used in an insulting, smug manner.
> The Dunning-Kruger Effect Is Probably Not Real
Self-deception by any other name is still self-deception.
The Dunning-Kruger effect also applies to smart people. You don't stop when you are estimating your ability correctly. As you learn more, you gain more awareness of your ignorance and continue being conservative with your self-estimates.
But overall I think real intelligence by definition requires empathy and humility.
One has to realize that we can't know the things we don't know, which includes the fact that we can't always trust our own beliefs and opinions because we might be relying on faulty or incomplete information, or we might be suffering from a mental health problem, whether we are aware of it or not.
"As a rule, strong feelings about issues do not emerge from deep understanding." -Sloman and Fernbach
I think the Dunning-Kreuger effect is more pernicious in smart people. I think Engineer’s Disease is basically DK by another name.
But what if that is the Dunning-Kruger effect?