Prologue.
Recent Conversation with CEO B:
Me: I saw your ad at the airport the other day. I have a sense for your challenges and Marketing objectives. I’m sorry, but that ad was heartbreaking. I sense it is trying to be too clever. And, it demands an immense amount of attention to understand, who in their right mind would give you that? You know I love Creative Pre-Testing (CpT). You should test the ad for effectiveness, before you spend money.
CEO B: First: You are right. After a few months of the campaign, we have seen no impact. Second: Avinash, I listen to you. We did pre-test the ad creative. It passed with flying colors!
Me: REALLY!? Can you please send me the vendor’s report?
Typically, failure is rooted in not giving at least a passing glance to data.
This is a significantly worse scenario:
Failure rooted in listening to data, while not recognizing foundational flaws in the data/analysis.
This is a massive hit on the credibility of the Agency, Analyst, Tool provider.
Today, though the lens of CpT, the story of Analytical flaws.
The critical questioning approach I take below applies to Conversion Rate computations, to the terrible habit of “organic social data mining,” to being able to cut through “AI” solutions hype, and to the inflated impact of Marketing (the reason no one believes your CMO).
Creative Pre-Testing | The Extraordinary Value.
I’m obsessed with Creative.
It is your video ad – on TV or YouTube. It is the text in your Search ad. It is the hideous LV billboard in Manhattan of Roger and Rafa walking in t-shirts (!) on top of an ice mountain. It is your Display ads.
My obsession is sourced in two extraordinary facts:
Without exceptional, sustained, Brand Marketing it is impossible to gain market share.
Fully 55% – 70% of a Marketing campaign’s success is influenced by the Creative!
There is literally nothing in Marketing that is more important.
By pre-testing the creatives you are, wisely, choosing to pre-guarantee success by identifying and eliminating the "loser creatives" – before you spend any of your precious media dollars.
CpT forms one of only two key parts of my win before you spend strategies – which CFOs love.
Creative Pre-Testing’s Awesome World.
There are numerous Creative Pre-Testing solutions in the market. Some have been around since the 13th century, others are brand new.
That history explains the prevalence of a multitude of approaches taken to conducting pre-testing. The crucial dimensions to evaluate closely include:
1. How the test is conducted (tiny focus groups, in 2024!!).
2. What data is collected (verbally expressed feelings vs, quant, qual, facial encoding).
3. The methodology applied to computing the results (we averaged it all).
4. The result itself (here’s a compound metric, happy birthday!).
5. Why should you believe the result (trust us, and did you hear our influencer at Cannes).
6. The ability to go from indexing against industry norms to category norms to company norms, through large scale sophisticated meta-analysis.
7. Identification of the relationship between media weight and passed pre-tested creative that is key to delivering success. (Humbly: The biggest miss by most CpT providers.)
Having put in a couple of decades into this work, I have a book’s worth of content on everything above. In future editions of my newsletter, TMAI Premium, I'll unpack each dimension, to help you become a dangerous asker of difficult questions to your Agency, Vendors.
Given the heartbreak of my conversation with CEO B above, let’s look at #4 today.
Let me help you recognize when you might be (inadvertently) led astray by your pre-testing provider. I'll flag what to look super closely when 59 slides of results with big tables and fancy charts are puked at you.
Pre-Context. :)
The use case today is creatives being used for a Brand Marketing campaign.
(Though, everything also applies to Performance as well.)
The creatives could be used on any channel: YouTube, Facebook, billboards, TV, Magazines.
There are two main types of CpT:
CpT A. Testing creative concepts. (First green box, on the left.)
CpT B. Testing creative executions.
CpT A is sixty-seven times more important than CpT B because getting the creative concept wrong results in all the executions of that concept failing (and campaign death).
CpT A is also a lot harder than CpT B – 70% due to human ego/internal politics reasons, 30% due to quality of pre-testing required.
Expected CpT Outcome: The thousands/millions spent on media amplifying passed creatives will deliver a causal lift in the Big 6 KPIs – three for short, medium, and long-term financial success, and three for long-term brand relevance and resilience.
[TMAI Premium Members: Please see editions #403 & #404: Big 6 KPIs: Delivering The Great Leap Forward. Please email me if you can't find them, they are absolutely crucial.]
What's your Creative Pre-Testing vendor actually measuring?
Of the seven dimensions above to ensure your CpT program is not delivering a false positive, (much to CEO B's heartbreak), I'm focusing on #4 today.
I’ve worked with many CpT vendors – those that are independent and those that are a part of an ad platform's offering (say YouTube, Facebook, others).
Every ad platform offers "creative best practices" for their ad units. You should read them, listen to them, activate them to the fullest extent possible. There is a lot of good that can come from that, even the worst-case impact of that is you will suck less – this is great!
It is important to emphasize that ad platform provided "creative best practices" are not a replacement for pre-testing your creatives – especially for creatives that will take up big chunks of your media budgets. You need a very specific trust, but verified proof that those creative will deliver profits. For these, you'll use the independent CpT vendors.
Let’s look at three vendors whose approaches to CpT are representative of all.
Thermo.
(Real company, fake name.)
Thermo has an extensive approach to pre-testing creative. As is common with many CpT vendors, it leverages academic research around human reactions and emotions.
Thermo, other vendors, often pre-test with gen pop (a wide slather of the population). It is wise to insist that the pre-test audience is a close replica of the media audience. You pre-test against a random human collection, creatives pass. This offers zero guarantee the creatives will work for the specific media audience! Campaign fails.
When Thermo presents its analysis, it reports "ad perception."
What does Thermo measure:
“effective simulated emotions”
“intrigue and motivation”
“believable”
“enjoyable”
“feeling of control and ease”
“makes me feel”
With sincerity and affection, I want to emphasize that these are fascinating assessments.
It feels valuable to know if the Creative is enjoyable or that it simulated emotions. I'm sure simulation has something to do with Creative effectiveness.
But, like you, I can get distracted by activity. Real life is about outcomes.
What does Thermo not measure:
Brand lift.
Thermo does not measure, they do not tell you, if the Creative will have any impact on the outcome of the campaign.
So… What's the point?
Is it helpful to know if the audience was simulated, but not know if that simulation will deliver (short-term) lift in the brand metric and (long-term) impact on the financial metric?
No.
I'm sorry.
I don't care.
The campaign objective is to deliver a +3 lift for brand KPI Consideration.
Without a prediction of the impact on the purpose of the campaign… Knowing easy to understand, enjoyable, makes me feel etc. is fun, distracting work for the Analysts and my Agency. It is just meeting fillers for my creative team and maybe CMO. Without knowing the impact of all that on my campaign objective… It is simply a waste of money (for the company).
The Analyst/Agency made a grave error in not realizing (/pointing out) that the 59 slides said a lot, but only about that blue line you see in the above image. It said nothing about the outcome.
The case of CEO B is a prefect example of this.
They used Thermo to assess that there was a lot of simulation. They proceeded to spend their considerable media budget on those Creatives. The campaign failed to deliver any lift – and obviously zero financial anything.
So…
Is your CpT vendor, like Thermo, analyzing, predicting, simulated emotions, or outcomes (on your campaign's success KPI)?
[Note: The critically needed concept of incrementality does not exist in Thermo's assessment.]
PlatformU.
(Real company, fake name.)
PlatformU (PU) is among the cohort of CpT providers who have been around for a looooong time. Its age is reflected in the massive number of clients they have, and in the traditional methodologies their solution leverages.
PU, like Thermo, but with a very different methodology, assesses self-reported viewer emotion.
At the end of its assessment, PU reports a compound metric: DUST Index.
What does PU measure?
"long-term market share"
"gain"
The pre-tested creative is given a DUST Index score of 7 (high), the implication communicated is that that will drive "long-term market share" gains. Honestly, which CMO can resist that implication? No wonder PU has so many clients!
The implication is not tied to a specific creative's impact on a specific campaign. See that black line above? PU is saying: More of this creative, and in the future it will have a business impact that could be positive.
What does PU not measure:
Brand lift.
Or Profit or even Market Share.
Specifically… If you are running a Valentine's Day campaign for your brand, or one for Back to School or Labor Day or for the football season…
There is absolutely no prediction regarding the impact of the pre-tested Creatives on the business outcomes from the specific campaigns using those Creatives.
As an Analyst, a critical thinker, it is crucial to recognize when this is happening in a tool, an analysis, in your reports to senior leaders:
Spend money on measurement.
Deploy a custom methodology with custom metrics with custom reports. (This is ok)
Present results that can never be verified.
That is what's happening when you use PU.
The DUST Index and resulting comprehensive and colorful analysis of individual creatives is only loosely implying something positive might happen in the future.
This mindset does not take into account the reality that companies use new and different creatives for new and different campaigns – which solve for different business purposes.
It also ignores the sad reality that little to nothing is strategized, executed, optimized in Marketing for the long-term. I wish that was not true. It is. As an Analyst, it is unwise to smoke pipe dreams.
Using PU is a little like promising you can use a bicycle to climb Mount Everest.
So…
Is your CpT vendor, like PU, reporting a compound metric loosely implying a future brand impact, or predicting the success of your pre-tested creative on your current campaign?
[Note: The critically needed concept of incrementality does not exist in PlatformU's assessment.]
Human Made Machines (HMM).
(Real company, real name.)
HMM is among the newer cluster of CpT providers. I've used them when I was at Google, across Marketing initiatives globally, and I continue to do so now for current clients.
HMM's core methodology divides the campaign's intended media audience into Test and Control clusters, to assess the impact of the Creative on the Success KPI of the campaign.
What does HMM measure?
Statistically Significant Brand Lift (in Campaign's KPI).
Diagnostic metrics (brand appeal, relevance, understandability, etc.)
The primary measures of Brand Lift are the three you've heard from me time and time again in context of effective Brand Measurement: 1. Unaided Awareness. 2. Consideration. 3. Intent.
As is also true for Thermo & PU, HMM provides an assessment of key themes present in the qualitative feedback that is collected from participants in the study. I find this to contain invaluable reflections on why a Creative might have failed or passed pre-testing.
The most important dimension of HMM's methodology is that it actually predicts outcomes: What will happen at the end of the campaign if you put all the media weight (budget) behind the Creatives that HMM scores as Passed?
The second most important dimension is that it is measured on the basis of incrementality.
If you listen to the data and do something (Cause), this is what's going to happen (Effect).
As an Analyst, I have an additional selfish benefit: I can trust, but verify.
HMM reports, say, Creatives A, C, F passed, Creatives B, D, E failed. I can:
Put most of the media budget on Creatives A, C, F, and measure if HMM's prediction of Brand Lift is delivered in real-world circumstances.
Put some of the media budget on Creatives B and D (F scored a 5/100, a dud!), and measure what happens in the real-world circumstances – do they fail to deliver lift?
My independent ability to trust, but verify, within the confines of the campaign for which the Creative was built, is the secret superpower that I'm entirely missing with Thermo and PU.
You should gravitate towards any data methodology that lets you trust, but verify – because Analysts, Vendors, Agencies, should never be trusted simply based on their words or jazzy slides, and extroverted humans, in the sales process.
What does HMM not measure?
Above and beyond what they are supposed to measure, see above, I am always hungry for more and more and more long-term meta-analysis!
So…
Is your CpT vendor predicting the incremental Brand Lift the Creative will deliver – for the Success KPI you've pre-identified for your campaign?
[Note: I've used Kantar for CpT, especially offline Creatives. I love that they also use test & control groups as their foundational methodology.]
Here's a little summary sketch for you…
[TMAI Premium Members: 1. If you would like a copy of this slide for your use, email me. 2. Building impactful Creatives is the key. Please see TMAI #401, #402 (Win Before You Spend) for six patterns that winning Creatives have, and nearly 30 examples that illustrate those patterns. If you can't find them, just email me.]
Bottom line:
Nothing's more worthy of your time as a Marketer as the Creative that tells your story. Brand or Performance Marketing.
Pre-testing that Creative, having the courage to use only the passed Creatives, is the single most effective way to pre-guarantee success.
Applying critical thinking when you assess your Creative Pre-Testing provider, ensures your pre-guarantee is not a mirage.
Carpe diem.
Via 
Like this post? Share it:
Or you could Print or PDF it!