Which is More Important: Confidence or Doubt?*

*Doesn’t matter, either way, you’re probably not as great as you think you are (and neither am I), so let’s instead panic (just a bit) and keep going.

Part 0: Before Self meets Confidence

Let’s start with a story; it takes place way before 2016. It’s a story about confidence and overconfidence.

One night some friends and I decided to meet for dinner and live music. Worried that I might get stuck listening to bad music, I volunteered to find a nice bar with awesome live music and great food. I started my usual process¹ to find things that looked “interesting”². After shortlisting several places, I finally chose one place because 1) it was a perfect size, 2) none of us had been there before, 3) the pictures and reviews were great, and 4) they had great live music scheduled that night. Everyone agreed to meet there. 

We got there, sat down, ordered food and drinks. After some time there, everyone started getting impatient because no band had performed yet! I walked up to the bartender and asked when the band will start. He looked at me like I was telling him that Trump-will-run-for-presidency-in-2016-AND-win-the-election and says “Never — we’ve never had live music here — in the history of this place.” I thought to myself, “In the history of this place?!?! Well, someone’s being over-confident. I know I read about live music online.”

After such a crappy experience, I picked a good place next time — that’s called learning and updating my model. After that good experience, I picked a new disaster — I swear it looked interesting.

Now imagine a fairly similar cycle for almost all social things I planned. According to my friends, after all these years if you averaged my success rate of picking good experiences, it would be a solid 50% — and that’s with a bit of effort³.

So am I running on too much confidence?⁴ If I am, is that a problem? Should I even care? I haven’t lost any friends over this so is overconfidence really a problem here?

Let’s start another story to explore these questions.

Part 1: Self Meets Confidence

What do we mean by Confidence?

Like most things that are influenced by perspective, what confidence means to you depends on who you are. So how do you think about confidence?

1. What do you mean by “Confidence”?
2. What do you think others mean by “Confidence”?
3. What do you think others get wrong about “Confidence”?

Most of us have either heard or said one, few, or all of the following (some of these I found on #confidence on Twitter):

“Be more confident”, “How to be successful”, “How to be intelligent”, “How to be competent”, “How to influence others”, “Self-confidence is the key to success”, “Believe in yourself”, “Act confident”,“Act confident and you are confident”, “Stop doubting yourself”, “Confidence is overrated”, etc.

Our culture reveres confidence and our personal development gurus preach it like it could single-handedly achieve both inner peace and world peace. And why wouldn’t they? Confidence makes it easier to get what you want more often. Confidence puts a rainbow-colored shiny halo around you that everyone is drawn to. 

Confidence allows you to be more liked by more people.

But what exactly is confidence and how can we think about it constructively? [Go away question, we’ll come to you later.]

Let’s see what Google says about some of the above phrases.

“Believe in yourself”

“How to be more confident”

“Be Confident”

“How to reason”

We care about confidence. A lot. Even if we don’t take these charts that seriously, we can informally say that we’re using phrases like “how to be more confident” or “believe in yourself” more than we used to earlier. Thankfully we also care about reasoning, just no more than we used to earlier.

Why do we care about Confidence so much?

Nobody really invented confidence out of some want or need. Humans evolved in dangerous and uncertain times, so confidence probably evolved to give us an evolutionary advantage back then.

Let’s say hi to Self. Self is a human. Let’s see how confidence could have helped him back then.

Back-then-Self

Back-the-Self lived in dangerous and uncertain times. He needed to be liked, appreciated, and accepted. Most importantly he needed to be safe.

Confidence motivates us to take action in the face of uncertainty. And life back then was filled with uncertainty. Self was constantly in new situations fearing rejection and competition. And actual rejection and failure really sucked so he had to avoid it.

Back-then-Self had to convince others that he was better than others, especially his competitors. When the rewards were high, such as finding a great mate or negotiating who owns what land or fighting for new-found resources, etc., he needed to convince others that he was the most competent one. And to do that, he needed to believe that he truly was better than others.

Back-then-Self needed to do this without really knowing others’ strengths. He needed to win without knowing others’ strengths and strategies. And it is in these uncertain and high reward situations that confidence (actually overconfidence) helped. The paper says, “…under plausible conditions for the value of rewards, the cost of conflict, and uncertainty about the capability of competitors, there can be material rewards for holding incorrect beliefs about one’s own capability. These adaptive advantages of overconfidence may explain its emergence and spread in humans, other animals or indeed any interacting entities, whether by a process of trial and error, imitation, learning or selection.”

Back-then-Self had to reduce the uncertainty of his environment quickly. He needed to make sense of his situations and act. Overconfidence helped him reduce the uncertainty of his world by converting the ton of actually-hard-to-predict-variables to a few easy-to-formulate-and-understand-sentences. Like many other biases, overconfidence is also due to our erroneous attempt to process and interpret information in the world around us. It’s faulty, but it helped Back-then-Self form flexible models quickly and act.

Who cares about Back-then-Self, what about Here-and-now-Self?

Being judged negatively by a small number of people had massive consequences on Back-then-Self’s well-being. But does Here-and-now-Self need to worry so much? The consequences of being disliked by a small number of people are small these days. Who cares if he can’t influence those people, there are plenty of other people who he can be with.

Why, then, is Here-and-now-Self so obsessed with the dozens of articles written on confidence, all of them telling him how he needs to “become more confident”? Don’t we have better tools to process and interpret information?

Yes, we do.

Why is Now-and-Here-Self behaving like Back-Then-Self?

Because Now-and-Here-Self still has the brain of a Back-Then-Self. And his brain loves shortcuts.

We are (also) Monkbots — monkey bots with a massive drinking problem

Remember when you did something with confidence (actually, with overconfidence) which you now know you shouldn’t have done? You miscalculated something major. That decision was mostly made by Monkbots (monkey bots with a massive drinking problem that are in your brain). They can’t assess confidence reliably and are experts at ruining your short-term life.

Think of Monkbots as our crude instinct for guessing whether the result of our actions will be successful or not. And they don’t really have the tools to do a good job. They are happiest when they are lazy and when they’re happy, we feel good about ourselves because things seem easy for us.

Overconfidence is often a result of Monkbots’ attempt to simplify information processing.

They make decisions, many times insanely stupid decisions, based on a) what information comes in first, b) how many sources are giving them similar information, and c) how their simple brains interpret the information they get. They see no reason to pause. They see no reason to get off their bums to go find Reasoners for help.

Monkbots aren’t completely useless though, they allow us to make decisions quickly. When the cost is low, it’s ok if they’re in charge. But if it is something really important to us in our modern society, like finding a good career or finding a compatible life partner or making useful medical decisions, Monkbots will almost definitely screw up. And that’s because they don’t bother to involve Reasoners in decision-making.

Monkbots confuse confidence with a million other super-nuanced-and-reliable things, i.e. they use confidence as a proxy for competence. And because they seem to incorrectly assume that confidence equals competence, they encourage us to have confidence even when we probably shouldn’t because what we might really need is actual competence. Confidence makes us feel better about ourselves so we naturally overestimate our ability.

Self’s brain (= Here-and-Now-Self’s brain = Back-then-Self’s brain) also has Monkbots. So let’s see how Self’s brain makes sense of Confidence.

What is Confidence?

Before we proceed, let’s go to a dictionary to find out what confidence is because, you know, “The beginning of wisdom, is the definition of terms” [Socrates] and all that.

Merriam-Webster’s definition

1 a : a feeling or consciousness of one’s powers or of reliance on one’s circumstances

b : faith* or belief that one will act in a right, proper, or effective way

2 the quality or state of being certain : certitude

Oh dear! 

 Hey dictionary, are you really saying that if I want to be a confident person, then I just need to have faith that I will “act in a right, proper, or effective way”? Sooooo I wouldn’t need to show evidence? I am confident = I have faith that I act in a right, proper, or effective way = I just know.

That’s just ridiculous! I just need to believe in my abilities and don’t need to show any evidence. Then why would I (or anyone else) think that I can be trusted? Remember Monkbots? They make sure I am mostly doing things I feel good about, even if those are not the most useful things. So, could you really trust me to act in an effective way all the time? Nope, you can’t.

But this isn’t about me. It’s about Self. Self, meet Confidence.

Self thinks Confidence and him are perfect for each other. They don’t need anyone else.

We better find ways to reason with Self, else, like a crazy narcissist, he’ll run with the idea of being super awesome and land himself or someone else in trouble.

Neighborhoods of Confidence

Let’s see what else Merriam-Webster says about confidence — maybe we will find some tools there. Merriam-Webster also gives related words. I went through each of the related words and categorized them as red, green, and yellow (I know I know, a super-scientific methodology) and here’s what I found:

At this point Self’s shock:comfort ratio should be at least 40:7, but this is Self’s state.

He thinks red words are faaarrrrrr away and he won’t be affected. Let’s try to help Self think about this a little better.

Think of Confidence as a neighborhood. It has all those feel-good things Self wants. He is awesome in this neighborhood of Confidence because he can:

• Influence others
• Start (and pursue) things
• Recognize his fear and manage it
• Stand up for himself and his beliefs
• Hold a long-term view
• Carry on despite setbacks
• Stretch his limits
• Seek out new opportunities
• And many other things

The related words, then, are other nearby neighborhoods. The red words represent neighborhoods near Confidence that are dangerous and land Self in trouble. If Self intentionally or accidentally steps outside of the neighborhood of Confidence, he’s more likely to land up in a dangerous (= red word) neighborhood.

Ending up in one of the neighborhoods could mean he:

• Ignores failures and doesn’t learn or improve
• Assumes opposite views are always wrong
• Believes he is superior because he can influence others
• Believes his competence is transferable to any other domain
• Pisses off friends, family, colleagues, and even strangers
• Performs inaccurate self-assessment (in his favor)
• Thinks that confidence will eventually cause competence
• And many other things

Confidence lives in a fairly dangerous space.

Just like Self needs guidelines and strategies to navigate new or dangerous geographic neighborhoods, he needs guidelines and strategies to navigate Confidence and its neighborhoods.

How unchecked confidence grows vs how Self thinks it grows

Left to himself, Self’s confidence could keep growing.

He may start thinking that his confidence is a result of his competence, and even if it isn’t, then it will eventually cause competence. He begins to form self-assured models quickly and it becomes harder and harder for him to see reality or reason.

Initially, his confidence may grow slowly, so even a small challenge from one person might be enough to make him pause and re-evaluate his approaches. But, if his confidence goes unchallenged, and does so for a long enough time, then he may only pause and re-evaluate if he is hit with either a) a massive challenge or b) many smaller challenges from a large number of people. The higher the rate of growth, the bigger the challenges need to be, ranging from doh-hand-to-his-forehead type situations to fall-to-the-floor-because-life-is-falling-apart situations.

But Self probably doesn’t realize how hard it is to assess himself accurately.

Remember, Self has Monkbots running around in his head and as we said they look for the first thing that comes to his mind, especially if it comes from a large number of people. Self thinks his confidence growth is a result of his competence growth. He thinks he is actually improving so why pause?

Time to give Self a dose of reality because there could be an insane number of variations that are more likely to be true.

Self’s Reality Check

Reality is very different from what Self imagines. In some areas, reality matches up to his assessments, but in most cases, the reality is much harsher than his assessments of himself.

This disconnect is one of the biggest reasons he’s likely to stay in red-words neighborhoods.

He attributes most value to his thoughts, while reality attributes most value to his actions (= evidence).

He thinks, “I’m a good person. I want to make a massive dent in the universe. I can reason about things better than most people. I have a lot of empathy. I’m ambitious and kind. I’m fair even when life throws crap at me. I’m more competent in my domain than those other people. I try my best. I’m a loving friend/family member/human….”

Reality looks at your actions and says, “What impact you Lazy-Self — you really haven’t done much in 342 days. Have you seen the list of your unexamined axioms — if biases are for other people, then you are Other-Self. You behave like a jerk when you’re running late Unkind-Self, you blame others for your parking tickets It’s-never-my-fault-Self, you think that startup that got $30 million in funding didn’t deserve it, but forget that you’re getting more than most of the world Ungrateful-and-envious-Self…”

Reality looks at all of his actions, but he looks at only some of his actions. It is likely that if Self is competent, then he will have confidence, but just because he seems confident (but-could-also-be-any-1-of-the-40-red-words), he need not have competence. His first and best strategy to not end up in those red-words neighborhoods for too long is to find ways to differentiate between confidence and competence. And to do that, Self needs to learn to doubt and search for evidence.

Part 2: Self meets Doubt

What is Doubt?

Our culture also tells us to avoid doubt like it’s a black hole. So Self avoids doubt like it’s a black hole.

Doubt creates uncertainty about our skills, skills of others, the feasibility of our ideas, and the trustworthiness of people. It creates tension between us and those around us because it could make us angry when others challenge our beliefs (and vice versa). Think about all those times when you challenged someone’s beliefs or axioms and they reacted violently — introducing doubt when someone isn’t ready creates volatility. Doubt can also create paralysis and force us to play life too safe. Playing life too safe wasn’t really great for Self back then, especially if it meant he couldn’t act quickly enough. He needed to make sure he could act fast. Analysis paralysis was very counterproductive then, if not deadly.

So why should Self learn doubt? Because he doesn’t live in ancient times anymore. In the present, he needs better ways to analyze the tons of information coming his way so he can at least avoid unnecessary and uncalculated risks.

These days, doubt is not only advantageous, but it is also necessary. Doubt encourages him to evaluate evidence, both for and against his beliefs. It motivates him to continue exploring. It challenges the status quo. It ignores preconceived notions of ideas.

Doubt is Self’s internal warning system. Remember how quick Monkbots make decisions? Well, Doubt triggers alarms in Self’s brain that force Monkbots to pause, get off their lazy bums and actually pass the information to the Reasoners. It also forces the Reasoners to wake up and look for information. Under normal circumstances, there is nothing forcing the Monkbots to pass on the information they receive. But, Doubt’s alarm forces them to get the Reasoners involved in decision making and update their models.

Doubt forces Self to search for evidence in situations where he would otherwise run with his unexamined axioms.

But we also know Self can’t doubt everything all the time. And this is where things get tricky. Similar to how Monkbots confuse confidence with a million other super-useful-nuanced-and-reliable things, they also confuse doubt with a million other super-judgemental-and-not-so-related-things.

Maybe if we go a dictionary to find out what doubt is, we can get some help.

Merriam-Webster’s definition

1 archaic

a : fear

b : suspect

2 to call into question the truth of : to be uncertain or in doubt about

3 a : to lack confidence in : distrust

b : to consider unlikely

This means: I doubt myself = I lack confidence in myself = I cannot be trusted

This also seems ridiculous! If I call into question any belief (that is held dearly by society) then the risk of judgment is massive. I might be labeled as annoying, incompetent or untrustworthy. And that hurts. A lot sometimes. And since my ancient brain is wired to keep me doing things I only feel good about, it’ll try to convince me to not doubt and instead assert things confidently.

Let’s introduce Self and Doubt and see how they interact.

Self Meets Doubt

Self thinks Doubt is just being unnecessarily annoying and he has no energy to manage Doubt every day. Maybe if we look at related words, we can show him that there are good-for-you-green-words-neighborhoods he can explore if he befriends Doubt.

Neighborhoods of Doubt

Let’s look at Merriam-Webster’s related words.

Let’s look at Merriam-Webster’s related words map.

Nope, there are no green-words neighborhoods to comfort Self — thanks Dictionary and Society. Doubt lives in a fairly dangerous and unpredictable space.

These neighborhoods have all those things that seriously scare Self. They have characteristics (and pathways) that:

• Give him analysis paralysis
• Scare him into not starting (and pursuing) things
• Burden him with fear
• Force him to question status-quo and new perspectives
• Challenge him most precious and personal views
• Make him feel less qualified
• Tire him with constant questioning
• And many other things

No wonder this is Self’s state after learning all this:

Self’s fear is not totally unjustified. Because if he explores neighborhoods of Doubt, he is likely to get stuck in those neighborhoods. We haven’t given him any tools to get unstuck or to get out of those dangerous neighborhoods. He doesn’t know how long he could end up being stuck in one of those neighborhoods, he’d much rather just avoid Doubt, similar to how we humans much rather avoid black holes.

How unchecked doubt grows

So far Self has no tools or methods to control or quiet his doubt, ever. As Self goes through life, if he doesn’t find ways to tackle, at least, some of his doubt, it may keep growing and he may get stuck in one of those dangerous neighborhoods.

Initially, when his doubt is low, even the smallest meaningful interaction with one person may keep him going. But, if he doesn’t get sufficient support often enough, then his doubt will keep growing and he’ll need much bigger support systems or many different smaller support systems from a large number of people to help him get through even one day.

The higher the rate of growth, the scarier the consequences on Self’s life and the stronger the support systems need to be. These range from one-person-helps-study-for-the-test-so-i-pass to friends-family-therapist-magic-and-more-needed-to-take-even-one-step. He may begin distrust himself, but even worse he may reject his capability to improve.

Here’s the thing though: Doubt can also act as an internal alarm system. So Self should befriend Doubt. He just needs to make sure he does not get lost in any dangerous neighborhood for too long by finding people and systems that support him, those that increase his competence and hence build his confidence.

His first and best strategy is to find a way to balance Confidence and Doubt. And to do that, Self must find ways to differentiate between his thoughts and reality. And that seems hard.

Self needs both Confidence and Doubt. Self needs Confidence to inspire him to act. He also needs Doubt to help ensure he gets things reasonably right.

But how can he balance the two?

Part 3: Pulling Confidence and Doubt together

Let’s come back to the idea that Doubt acts as an internal warning system. Doubt sounds the alarm in our brains that forces Monkbots and Reasoners to work together to avoid overconfidence:

When Self is an absolute beginner, Monkbots involve Reasoners

“Absolute beginners can be perfectly conscious and cautious about what they don’t know; the unconscious incompetence is instead something they grow into.“

When we’re beginners and we actually view ourselves as beginners, we tend to frequently doubt ourselves. That doubt reduces our confidence. Remember those Monkbots — well, the information coming in seems so new and fast that they don’t yet know how to deal with it. They don’t yet have models they can use or update. Doubt is constantly sounding the alarm because Monkbots aren’t able to process this information quickly enough. This forces all the Monkbots and Reasoners to work together because they all think you’re going to get thrown out of a tribe or get labeled as an un-mateable loser or get killed.

Monkbots eventually get the hang of it and begin to process this information quickly. They build somewhat shitty models quickly. And nope, that’s not super helpful either because they don’t update those models.

With just a bit of experience, Self’s confidence starts to grow faster than competence. Monkbots are at it again.

“ A little experience replaces their caution with a false sense of competence.”

“The Beginner’s Bubble: We found that people slowly and gradually learned how to perform this task, though they found it quite challenging. Their performance incrementally improved with each patient. Confidence, however, took quite a different journey. In each study, participants started out well-calibrated about how accurate their diagnoses would prove to be. They began thinking they were right 50% of the time, when their actual accuracy rate was 55%. However, after just a few patients, their confidence began skyrocketing, far ahead of any accuracy they achieved. Soon, participants estimated their accuracy rate was 73% when it had not hit even 60%”

When we start gaining some experience, we begin to view ourselves as more experienced, competent, and knowledgeable than we actually are. Monkbots start becoming really active because they have those models they can use. They start getting comfortable, which starts making us feel life is easy and good. We become less doubtful of ourselves because we focus more on our feel-good thoughts than our actual actions. Without an internal alarm system, Monkbots and Reasoners no longer have any reason to collaborate.

The trick is to find a way to introduce doubt at this stage — via feedback. We need to get the brain’s alarm system to go off and force Reasoners to get involved. Then we might be able to constrain this unwarranted confidence growth.

All this because of forming erroneous self-assured models quickly — because model building and updating is slow and hard.

“What produced this quick inflation of confidence? In a follow-up study, we found that it arose because participants far too exuberantly formed quick, self-assured ideas…”

“Small bits of data, however, are often filled with noise and misleading signs. It usually takes a large amount of data to strip away the chaos of the world, to finally see the worthwhile signal. However, classic research has shown that people do not have a feel for this fact. They assume that every small sequence of data represents the world just as well as long sequences do.”

1. Shitty self-assured models — check.
2. Quick over-simplified conclusions — check.
3. Noisy as hell data — check.
4. Ton of inaccurate data — check.
5. Massive effort to clean this data — nope, Monkbots can’t do.
6. Massive effort to build robust data pipelines — nope, Monkbots can’t do.
7. Represent and understand data in multiple ways — nope, Monkbots can’t do.
8. Update models regularly and check metrics — nope, Monkbots definitely can’t do.
9. Massive biases and errors that favor feel-good things over reality — check, check, check.

We learn, but then we also forget. Monkbots and Reasoners continue their love-hate relationship.

“But our studies suggested that people do eventually learn — somewhat. After participants formed their bubble, their overconfidence often leveled off and slightly declined. People soon learned that they had to correct their initial, frequently misguided theories, and they did.”

“But after a correction phase, confidence began to rise again, with accuracy never rising enough to meet it. It is important to note that although we did not predict the second peak in confidence, it consistently appeared throughout all of our studies.”

Eventually, someone catches on that we’re making mistakes or not improving or asks us questions we can’t answer or presents us with situations we can’t understand, etc., i.e. we get some new information that forces the Monkbots to slow down a bit, again. Reasoners get involved, again. We update our models a bit. This slows us down for a while and our confidence takes a hit. Later Monkbots catch on and get comfortable and off we go again.

It’s crappy, but it’s just very hard to know ourselves well: “The real world follows this pattern.”

Confidence is not dangerous if Self has it in the right amount. And the right amount has to be aligned with Self’s actual ability and maybe even his ability to gauge his ability. That’s where Doubt can help — by constraining Confidence within a range that is aligned with Self’s abilities.

Doubt can keep Self within a manageable and realistic range of Confidence.

Self will need to frequently trade the perception of certainty that comes with Confidence with several less-precise and slight-panic-inducing guidelines. Oddly, better approximations of Self’s confidence can come about by emphasizing doubt at each step.

How does Self find a set of guidelines to navigate his way through potentially dangerous areas? How does Self create enough doubt to constrain his confidence, ensuring it’s reasonably aligned with his ability? All of this seems like a ton of work for Self.

So far Self doesn’t have the tools to answer these questions. And it doesn’t have to be a ton of work.

Overconfidence is so hard to correct because it’s so deeply tied to the structure of our thought. We can’t ever eliminate it from all dimensions of our lives unless we get rid of our brains. But we can become slower and more aware, at least in most important dimensions of our lives. And then we can try to build strong foundations in these dimensions

In the next blog, I’ll attempt to create a simple framework to help both Self and us create our balanced-selves.

Footnotes

1. It starts with me opening enough tabs (in multiple windows) to exercise my memory, attention, flexibility, and speed so thoroughly that it would crush those Lumosity exercises. Next, comes the simple step of making a decision, done as follows: read about food/history-of-food/chefs/science-of-food/whatever-questions-this-brain-asks, then take a break to eat a snack or two, then daydream about how learning + working on solving a problem + traveling + eating is my dream life, then feel lousy for not working on anything meaningful, then transfer that struggle to food so I can figure out whether I am in the mood for a particular cuisine or want to be surprised, then depending on the cuisine, decide if I want a restaurant with 5 stars from only 27 reviewers or 4 stars from 168 reviewers, then search text reviews for foods I like to make sure reviewers have actually rated items I care about, then think about how current recommendation systems don’t seem to be taking strong dislikes for food items into consideration (do they? how can I trust someone who doesn’t like dessert?), then read reviews to see which ones provide the most entertainment and hence relevance, then find pictures of items I like, then finally pretend the machine in my head has the best end-to-end prediction system and put all this together. Talk about a run-on mind!

2. Should we be allowed to use this word? What do we really want to say? Which other words would we use if we were forced to replace it each time? Who knows. Back then I didn’t know better so I used it a lot, now I know better so I make sure to add a footnote.

3. For those of you wondering why I don’t have a better strategy, I’d like to remind you of a better question: Why do my friends still listen to me? Clearly they’re the ones who need a better strategy. [Footnote to a footnote: I have some idea about this, it might be related to explore vs. exploit and what type of a personality I have, but that’s for another post.]

4. I suspect if I take the perspective of optimizing each decision to find the most favorable end goal, then I could probably improve my success rate. But I regularly change my goals (in this context) so the most favorable outcome adds negligible joy. The most joy is assigned a) to the exploration process and b) to creating the journey that itself is a funny and memorable event. In this context, the actual outcome is sort of like the cherry on top of my favorite ice cream sundae. I don’t really care if the cherry is not there, but if it is there it looks pretty and gives me one extra new and interesting bite.