Roni Kobrosly Ph.D.'s Website

The great Benn Stancil, founder of Mode has a fantastic substack. He writes on topics like:

• "Is Excel immortal?"
• "How a startup feels"
• "How to create a winning pitch deck"
• and the hilariously titled "Ed Harris would like to connect with you on LinkedIn"

He wrote a blog post way back in the summer of 2022 entitled "Do data-driven companies actually win?", and I just can't shake it from my mind. We take for granted that collecting data, mining it for insights, and building ML gives organizations a competitive edge. Is that true though? Have we rigoriously assessed that? It's sort of funny, but in this blog post I'm basically questioning the raison d'être of my profession 😂.

In this post he sets up a thought experiment where he compares a few theoretical, E-commerce startups selling clothes. They're run by very different types of experts. The companies are as follows:

1) "Long View": Their executives have been working in fashion for decades. They know the market, they say; our experience, and the intuition we’ve developed on top of it, will make us successful.

2) "Bolder": is led by executives who’ve been in the industry for less time. But true to their name, they believe in moving fast and making things, in not overthinking strategy, and that decisiveness is often more important than being right. As the last slide of their pitch deck says, “We fail when we look back.”

3) "Prodigious Daughter": is run by a thirty year-old wonder kid. In just a few years, she’s already put her stamp on fast fashion. Though her company is average in other ways, it has her generational talent.

4) "Square Corner": emphasizes operational excellence. Their leadership team writes emails with military precision, never never misses 7:30 a.m. standup, and always sends out board meeting slides, a pre-read, and a Loom explainer video exactly five days early.

5) The final startup is "MTRX": believes data will be their competitive edge. They've taken all of the MBA courses at Wharton data-driven business practices. Their prior experience is on par with that of Bolder’s founders, and has taught them that fashion is fickle and hard to predict. The most iconic brands, they say, will be built by companies that find and respond to new opportunities in the market before anyone else does. Fashion may be art, but running a business is a science.

Benn actually polled his readers for which company would be most succesful, but frustratingly I can't find the results!

To get to the point though, Benn makes the following statement:

Honestly, if it’s my money, MTRX ain’t getting it. My stack rank is 1) Prodigious Daughter, 2) Bolder, 3) Square Corner, 4) MTRX, and 5) Long View. For early to mid-stage companies, give me talent and intuition over everything; if not that, I’ll trade away analytical rigor for speed and decisiveness. 

Still, I don’t think that the statement that “data-driven companies win” is entirely wrong—it’s just data operates through a different mechanism than we might assume. 

Most often, we say that data helps us make better decisions. We can devise better strategies, and be smarter operators. The implication here is that when we’re faced with a choice of what to do, with data, we’re wise; without it, we’re foolish. 

I think this dramatically overstates data’s usefulness. Business problems are extraordinarily complicated, and analytical recommendations are mostly educated guesses. Great data teams likely make somewhat better guesses, but at the end of the day, we’re all still gambling.

I mostly agree with him.

How would we even go about rigoriously testing whether a data-driven company has an edge?

• Run an experiment? It's obviously not feasible to create a batch of new startups and randomize them to embrace the best data practices vs no
• A natural experiment? As in, look at companies that previously weren't data-driven, identify a point at which they suddenly embraced the best data-driven practices? Perhaps a new CEO joins, with visions of realtime dashboards and Generative AI gleaming in their eyes. How would you know any change in revenue or other KPIs could be attributed to the data practices specifically? Maybe it's a change in management style or team organization or some third factor?
• Run an analysis of observational data on the 1000s of startups over the years? How do we go about quantifying how data-driven they are? Furthermore, there is likely severe confounding factors at play that make the analysis suspect: maybe executives that embrace the latest market tactics and understand the market dynamics tend to succeed more and also tend to embrace data. Maybe data has nothing to do with it at all.

I'm not aware of any rigorous attempt to measure the impact of an org being data-driven, but I'm very curious if something like this exists. Please let me know if you learn of anything on that topic!

In my experience and from hearing the stories of others, it feels like 1 out of 20 organizations have the data maturity to:

• properly create clean data, and have solid practices in place to ensuring data stays clean
• organize/structure it in a way that data teams can use meaningfully use it
• hire data engineers to lay a solid foundation before hiring sexy data scientists / ML engineers
• understand that data-driven decision making is the way to go, instead of decision-driven data making ("Hey, this analysis didn't show the result I wanted to see. Can you try slicing and dicing the data again so it shows a better result?")

I could go on and on. This all sounds really negative, but I'm hopeful that as a data-savvy generation of tech folks make their way into the C-suite things will get better 🙏🏼.