For those who place their faith in scientific-like testing (and big data), the findings of the Reproducibility Project ought to be astonishing. Two-thirds of the original studies tested proved fallible and even those that could be replicated demonstrated irregular statistical variations. Specifically, the magnitude of the effect tested was frequently half as small as the original finding.
Never place too much faith in any marketing formula.
Sure, there is plenty to be gained by running A/B tests in an attempt to convert your business thinking from "we think" to "we know." There are many successful examples. But just because the results of testing turn out one way or another doesn't ensure success. A/B testing isn't a sure thing in marketing.
The truth is that we must stop treating single studies as unassailable truths, especially when other variables could be influencing the outcome of any finding, outcome, or assumption. True scientific thinking, after all, comes with a critical mindset rather than a yielding one. And we need to be more critical now than ever before, especially as people attempt to manipulate our thinking daily.
You can find evidence everywhere. Journalists are more likely to write attention-grabbing narratives first and then find examples to fill in the blanks than ever before. Scientists are more likely to build studies based upon biased theories than rely on objective observations. And marketers, whether they admit it or not, generally attempt to validate their work more than they produce better outcomes.
And no, it isn't always intentional. Anyone who has ever gone to an eye doctor only to be prescribed an inferior prescription knows how easy it is for mistakes to happen. No matter how meticulous the doctor or technician might be in the office, you eventually have to try it in the real world.
It recently happened to me. In the office, it seemed monovision — wearing a distance vision contact in one eye and a near vision correction in the other — was a suitable option for my slight presbyopia. In the real world, it didn't work at all. Too much of my interaction with the world relies on intermediate vision for monovision to be effective. The same thing can happen in scientific studies.
There are reasons humans are mostly unpredictable.
If you truly want to understand psychology and sociology as it applies to marketing, you have to make a real effort to understand humans. First and foremost, you have understand that humans are the only creatures on this planet that form flexible and scalable cooperatives based on abstract concepts.
Yuval Noah Harari, author of the international bestseller Sapiens: A Brief History of Humankind, is especially intuitive on this point. As he explains it, bees and ants can form scalable cooperatives but aren't flexible in their ability to change their social structure. Whereas chimps and dolphins are flexible in how they cooperate, they are only able to do so in relatively small numbers.
The reason, it seems to me, it that humans are also the only creatures on this planet to operate with a dual reality, a perceptional concept studied in depth by Donald Hoffman, professor of cognitive science at the University of California, Irvine. In sum, humans perceive an objective reality (what is true) and a conceptual reality (what we accept as truth) at the same time.
For example, the money in your wallet is a piece of paper. The concept that it has value is a fiction that we have collectively agreed to accept as truth. And, to be clear, it is this dual reality often discussed by Hoffman that provides us our unique ability to form flexible and scalable cooperatives.
Marketing and communication, at their core, only has one purpose: to change behavior. And as such, marketers usually try to change behavior by drawing attention to an objective reality or attempting to elevate (or diminish) a conceptual reality. And what makes this especially interesting is that in the last decade, especially with the advent of social media (and likely to become more prevalent with the rise of augmented and virtual reality), is that marketers spend more time targeting the conceptual reality.
So what? The greater the emphasis on conceptual reality, the greater the unpredictability of testing because humans, throughout history, have proven to be consistently inconsistent. And in knowing this, maybe it is time to treat your approach to the science side of marketing as an exercise in adjustment and not in the collection of unassailable truths that will one day be proven false. Good luck.
