Where Data-Driven Decision Making Goes Wrong [and How to Fix It]

But businesses operate around uncontrolled variables and quickly evolving conditions. It’s very easy to be deceived by data and make poor decisions based on the fallacy that numbers don’t lie.

In fact, numbers do lie. And when they’re interpreted incorrectly, numbers have the power to create real and lasting problems. 

We’re not advocating for ignoring the incredible metrics you have at your fingertips — just that you make sure you’re being cautious while using them.

Not data-driven. Data-informed.

Data is an important tool to help arrive at decisions, but should not be the decision maker itself. No matter how powerful and rich a dataset may be, companies cannot abdicate their decision making responsibilities and outsource them to data.

Why?

Data isn’t accountable.

Wouldn’t it be great if you made a poor decision and could then pass the blame to “data”?

There is a natural human tendency to avoid decision-making pressure at all costs. Saying you are making a decision based on data is one common way to deflect the burden of responsibility.

Imagine going to a restaurant and receiving horrible service, while other patrons seem to be treated quite well. Confused, you confront the wait staff, and hear the response “our dataset indicated you fit the profile of someone who prefers not to be spoken to, or smiled at.” Obviously, this would not be a suitable excuse. Similarly, standing in front of your team and saying “we have to lay you all off because our data made a decision that cost us a lot of money” is not a suitable excuse. 

Using data to justify or excuse imprudent choices may be common, but ultimately it’s the decision maker who made the call, not the numbers. When we rely too heavily on data, it can be seen as a weapon, or as a limit to a team’s agency. Taking personal ownership helps ensure your team sees data as the growth-enabling asset it can be.

Likewise, when a business leader uses data as an excuse for poor performance, it can create a culture in which people seek scapegoats for mistakes, rather than accepting responsibility and learning from them.

Data is rarely conclusive.

Nuance is no fun. Life is much simpler when there are definite answers. You’ve probably seen this in your own life; people tend to seek out certainty in complex situations.

This false sense of certainty leads to the twisting of data to support the narratives that give us the most comfort.

But life (and business) is full of uncertain situations. One piece of data can be framed to tell many different stories, depending on your agenda. If we’re not careful, data can solidify our biases rather than help us overcome them and inhibit creative or intuition-based decision making. Groupon’s Former CEO Andrew Mason learned this through experience:

"...When you’re in hypergrowth like this, you don’t have time to see what is going to happen to the data in the long term... There are some things where you have to say, ‘I’m sorry. I’m not going to look at the data on that. This is just what we’re going to do. We know that it’s right, and there’s nothing that’s going to shake us from that.’’'

Data is only as good as the quality of its collection.

Data collection is the first step toward becoming a data-informed company. Making choices based on inaccurate or incomplete data is often worse than making decisions with no data at all.

For that reason, it’s important to constantly ensure the efficacy of your data collection systems.

You may have heard the mantra “people, process, technology.” This is a textbook application. Accurate numbers generally result from people following the correct processes with the correct technology.

Say you want to know your sales team’s close rate for different types of deals in order to gain insight into customer retention. But only some of your team members record closed deal information some of the time, and you have no written process in place for tracking customer retention. That’s an incomplete data set — using the information you do have might lead you to the wrong conclusion.

The technology component is also critical. Web traffic looks suspiciously high or low? Chances are it’s artificial. Tracking codes may not have been set up correctly and a bot attack may have warped the numbers. Before proceeding as though the data is gospel, it’s imperative to check for all those possible explanations first.

This is particularly true when it comes it to revenue numbers. Poor data collection on the sales side makes it nearly impossible to forecast bottom line realities, which in turn prevents scalable growth.

How can we fix data-driven decision making?

One of the reasons we tend to see data mishandled in business contexts is that businesspeople aren’t usually scientists or statisticians. We weren’t trained in standard methods for using data to answer questions. Familiarizing ourselves with some of these methods can be helpful. They include:

• Ensuring your sample size is large enough
• Using separate control groups to test conclusions
• Consulting multiple sources of data

Unfortunately, these tactics are often too expensive and difficult to make time for in the fast-moving world of business.

Here’s a simpler, but still effective approach: business leaders should review their data with as many different parties as possible.

Start with your internal leadership team. Have a meeting in which findings are presented alongside your conclusions. Solicit feedback and encourage dissent. It’s inevitable that others in the group will have a different perspective and challenge your conclusions. They might even challenge the accuracy of the data — and that’s a good thing!

Odds are, at least some of the conclusions will need to be revised based on leadership team feedback because some important considerations were missed.

Once that process is complete, show the data and revised conclusions to a third-party data analytics firm, consultant, or trusted peer. This fresh set of eyes can be invaluable in helping thwart groupthink and internal bias.

Conclusion

Just remember, there are really only three data points that truly matter for most businesses:

• Revenue
• Profit
• Expenses

All other data can certainly be helpful, but usually within the context of supporting improvement of the above three metrics.

 At Salted Stone, we believe data-informed decision making contributes to better business outcomes -- if your organization chooses to buy in with the necessary time and resources. Otherwise, it could very well hurt more than help.