Noise in our judgments

I am sure you have played this game a few times before: someone asks players to individually guess the number of jelly beans in a jar. The winner is the one whose guess comes the closest. If you are like me, then chances are, you have never come even close to winning one.

Guessing the number of jelly beans is like target practice at a shooting range. If we are lucky, we may hit the bullseye. But, most of the time, we don’t. The outcomes are often scattered around the bullseye.

This is analogous to the judgments and decisions we make in our lives. Whether we are bidding on a job, assessing a safety risk, forecasting revenue, or hiring an employee, we are trying to hit a bullseye. However, most of the time, we hit around but not the bullseye. In other words, our judgments are noisy.

Of course, the noise in my jelly beans guesses is going to be much higher than the noise in my target practice, because I may know a thing or two more about target shooting than jelly bean guessing. In other words, the better you are at it, the closer your shots are to the target. Nonetheless, unless you are a superhero in Hollywood’s imagination, you are not going to hit the bullseye all the time. Regardless of their expertise, the judgments people make in their personal or professional lives are going to be noisy. The big question then is, how big and costly is the noise in their decisions?

Researchers studied insurance underwriting done by several experts and discovered that the median difference in the price of the same policy was more than 50%. In other words, for the same policy, if one underwriter sets the premium at $10,000, another set is at more than $16,500. Similarly, for claim adjusters, the median difference was 43%. A similar analysis of the fair value of a stock by experienced investors at an asset management firm found the median to be 41%.

Keep in mind that these are not folks like me guessing the number of jelly beans in a jar. These are results from decisions made by expert bean counters in their respective fields. If the decisions made by these experts can have such a large noise, there is no reason to believe that the decision made by other experts such as job estimators, risk analysts, or an HR manager is going to be any less noisy.

Much like a target practice at a shooting range, we sometimes have the luxury of knowing how well we performed in our judgments. Medical diagnosis, weather forecast, and revenue forecasts are judgments where we will eventually know how close we came to the bullseye. For an ensemble of judgments, we can even measure the noise and the bias in them, understand their impact, and take corrective actions.

However, for many or perhaps most of our judgments, we do not have a bullseye to compare our results. An estimator will never know if his bid price for a project hits the optimum target. An insurance adjuster will never know if the claim payout she negotiated was the right one. Of the 20 candidates we interviewed for one vacancy, we will never know what the outcome would have been if we had hired someone else among the 19 candidates we rejected.

We often think of noise as something that has an averaging effect and contend with our results as long as the average of the outcomes meets our objectives. For instance, for insurance claim payouts, lower than ideal payout may make up for the loss due to higher payout. Unfortunately, many of our judgments do not have such a luxury. Bid too high and we risk losing the project. Bid too low and we risk losing money on the project. We may contend with multiple good hiring decisions to make up for one poor hiring decision but no amount of correct medical diagnosis can make up for even one wrong medical diagnosis.

And there are once-in-a-lifetime or blue-moon decisions for which there is no bullseye and there is no do-over. For example, starting a business or acquiring another business is not the kind of decision we make every day. Our judgments in such decisions are made using the same underlying cognitive processes that we use to make other judgments. They suffer from the same kind of noise as other judgments do. It is just that those singular judgments likely have a bigger impact and we will never know the degree of noise in those judgments.

Going back to the target practice, the reason we don’t hit the bullseye is due to the combination of noise and biases in the system. Noise, such as shaky hands and our heartbeats, causes our shots to be scattered around the bullseye. A bias, such as a misaligned scope or wind, causes them to be clustered on one side or the other of the bullseye. In the case of our judgments, the reason we don’t hit targets is due to psychological biases and noise in our decision systems.

Most of us are now familiar with psychological biases in our judgments. Concepts like confirmation bias, anchoring, groupthink, and overconfidence bias have almost become part of our everyday conversation. In my previous posts, I have written about psychological biases that influence our judgments. But where is the noise coming from and how pervasive and costly is it?

Daniel Kahneman, along with two other well-renowned psychologists, tries to answer these and many other questions in their latest publication aptly named Noise. Daniel Kahneman is a renowned psychologist and winner of the Nobel Prize in Economics credited for his groundbreaking work on behavioral psychology that became the foundation of behavioral economics. His bestselling book Thinking Fast & Slow, which takes users on a fascinating journey of the human mind, is a must-read for those who want to understand psychological biases.

In Noise, the authors take on a similar journey to help us understand the noise in our decision process. In it, the authors expose its pervasiveness, its impact (including the examples I cited above), and strategies we can follow to mitigate it.

In almost every aspect of our lives, whether it is personal or professional, and whether it is recurring or once-in-a-lifetime, we make decisions that are based on our judgments. In fact, if we could not do so, our jobs would not be satisfying. We all thrive when we think freely and make judgment calls. And in each of those judgments, noise is inevitable. However, what may not be inevitable is the degree of the noise, something we will explore in our future posts as we follow along with Daniel Kahneman and his co-authors in their journey of noise.