Analyze This: 5 Rules For Awesome Impromptu Web Analysis

The hardest kind of "analysis" to provide is in response to open ended questions. That is why I love asking open ended questions!

They expose a person's critical thinking ability (something I highly recommend you test when you hire web analysts: Interviewing Tip: Stress Test Critical Thinking. Please).

They also help you understand if someone really grasps key concepts.

Recently on behalf of Market Motive, my start up that focuses on online marketing education, I had the opportunity to offer one scholarship for the latest round of Master Certification in Web Analytics.

So at the end of my 10 Fundamental Web Analytics Truths blog post I requested readers who were interested in the scholarship to complete this simple task:

Pick a site you love and tell me three things you would change about it, and why.

Seems straight forward right? It is not!

First I must say that I was overwhelmed by the responses (thanks!) and I was impressed with the time people took to do the analysis. I got wonderfully created pdfs / Word docs and well written emails. I was amazed at the creativity on display (which validated the fact that I have chosen to be in the right industry!).

Based on the responses, some wonderful and some not quite as wonderful (!), in this post I thought I'll share with you some tips should someone (like me!) ask you an open ended question ("what would you and why").

The first part covers 5 rules, sourced mostly from what people did not do. The second part contains 4 things people did that delighted me.

Let's go.

When someone asks you an open ended question, at least connected to web analysis, here's what's important. . .

1. Don't offer your opinion, at least not right away.

This is a very very hard temptation to resist. But try.

These were most common fixes people wanted to make on sites they loved:

I have to tell you that the last thing anyone wants to hear, in this context, is your opinion.

Not your boss. Not your friend. Certainly not the HiPPO (Highest Paid Person's Opinion).

Even if you believe that you are "absolutely right"! [Note: I often think I am "absolutely right". :)]

You and I are poor proxies for the customer. And just because you don't like something… how should I put it so you'll understand…. oh let's try this…. you not liking something is not a statistically significant sample of data!

On a serious note… offering your opinion on something, unsupported by any data except "I think", is probably a really poor way to start a conversation with anyone in the Analytics field.

If you express your opinion then present it in the from of a hypothesis that can be tested. Win-win.

So for example consider saying something like:

See the difference?

It is ok that you started with a hunch. You went and got some kind of data. Finally you offer a hypothesis that I can test, and you were clever enough to point to two things of value to the business (both of which can be measured!).

Your HiPPO / Boss is much much more likely to listen to you and accept your wisdom.

In the rarest of rare cases if you must express your opinion, present your credentials. Something like:

See that would be acceptable. :)

Overall: if you can, try not to offer your opinions (at least not in the opening statement).

2. Always offer alternatives / Think things through.

One of the persistent flaws in Web Analysts (and Marketers as well I am afraid) is that far too often we take a siloed view of things. We only see our slice of data. We only see our little world. We only care about what bothers us / what makes us happy.

You should always take a much more expansive view of things and when you make recommendations think of the big picture, think things through.

Here is a good example.

I was astonished at how many Ninja's included this in their fixes: Remove Ads.

Now I love adblock as much as the next guy and wish advertising (especially Display) were more relevant.

But when you as an Analyst recommend removing ads because you find them annoying (and they can be super annoying) you are essentially recommending the removal of a revenue stream.

Ok so if I accept your recommendation of removing ads what do you recommend I do about the revenue stream?

The "remove ads" recommendations did not consider that implication of their recommendation.

Now I don't expect you to be an expert on the intricacies of the business you are analyzing when I give you an assignment to do "impromptu analysis". But I would have loved to know that you thought about the big picture, what you thought about the implications of your recommendations.

You could have said:

Or:

Or:

Give me a clue that you have: 1. Thought through the implications of your recommendations. 2. Have some alternatives handy, no matter how pie in the sky.

Here is another recommendation that is more nuanced, and something I think we as Analysts rarely think through.

The recommendation was that Flickr should allow posting of anonymous comments because it will likely result in more comments being published on pictures which will potentially increase User Engagement.

A very nice suggestion.

But by now it has been well established that anonymous comments very quickly lead to unintended consequences. [New York Times article.] All kinds of people jump in and, quite literally, say all kinds of things.

I would have loved to hear what your suggestion was to deal with this absolutely sure to happen outcome from your recommendation.

Think things through. As an Analyst, as someone who thinks more broadly.

[Note: I am not saying comments are bad. I am not saying all anonymous comments are bad. I am not saying comments should be 100% moderated and neutered before being posted. There is a happy medium and there are many wonderful options to deal with this problem.]

3. Offer data, even when you don't have access to the site's data.

Alec shared a guidance with me after the contest was announced. He said, and I am paraphrasing, "award the scholarship to the person who says that they can't make any recommendations to fix the site they love because they don't have access to the data".

Really good point.

I had very much kept my question open ended because I really wanted to see if people got creative with how they arrived at the recommendations (beyond the "I think").

I am afraid no one provided data.

On the surface it is understandable. You are doing analysis, impromptu analysis, on a site that you don't own. Of course you don't have access to data to base your opinions on.

Unfortunately that is not quite true.

You ALWAYS have access to data. For ANY website.

If you want to understand the clickstream data for any website you could go to Compete (here's ESPN's data, or this blog's). If you want data for a international site use Google Trends for Websites (here's H M V's data, and here's data for people from Switzerland who read the French newspaper LeMonde).

Sure the data is not 100% accurate, but it is directionally accurate and it will take a few minutes on either Compete or Trends to dig a bit and find something interesting you could base your recommendations on. It should take you a few more minutes to compare data for one site to its direct competitor and identify something even more interesting.

If you want to understand the search engine ecosystem then use Insights for Search. Check out how much delightful data is available to you: Acne vs. Poison. [Look out, poison making a massive come back!!]

Spend time understanding the keyword market and consumer interest for the business you are analyzing. Find strengths and weaknesses. Find opportunities (by geographic region or in the cluster of top related searches or, my fav, fastest rising searches). There are so many sources, so many possibilities (many free!).

If you want to get demographic or psychographic segmentation data use the DoubleClick Ad Planner. In a few minutes you can understand the demographic make up of any site.

Male – female, age, education, household income, audience interest and more. In a few more minutes you can get down identifying the psychographic segments. Affluent 100k+? Brides-to-be? Gossip Gurus? Home Buyers? Moms? Technology Geeks? Who are we talking to? Who do we want to talk to?

And these are just the basics. Check out: The Definitive Guide To (8) Competitive Intelligence Data Sources.

You always have access to data. Regardless of if you own the site or not.

If you are put in a position where you have to offer impromptu analysis please use these (and other) data sources to add the kind of power to your recommendations that can only come from being backed up with data. Some data.

4. Always, always, always state what you think the Objectives are.

This is such a common mistake when we present our analysis. We make recommendations without saying what we are actually solving for.

Before you present your recommendations first tell me what you think the website's objectives are. What you think the purpose of the website is. What you think the site is solving for.

Often analysis is not valued very highly not because it is stinky, it is because the producer and the receiver disagree on what the objectives of the site are.

I might think the purpose is: Orders, Leads, Job Applications.

You might think the purpose is: Facebook followers, Brand Perception Lift, Product Reviews.

If you don't tell me what you assumed the objectives were you'll see very quickly why I might think you produced nothing of value.

So make it clear.

I might still think your analysis was poor (or awesome!), but at least I know what you were solving for.

I have context within which I can place your analysis.

You might think that it is obvious what the purpose of GoNomad or NBA.com or SFAF is. But I assure you that it is not obvious. So make it obvious, we'll both come to your analysis / recommendations from the same perspective.

In your daily jobs you should never present your analysis without having shared vision around the objectives. Otherwise the best result is no action will be taken on your recommendations. The worst result is… we'll I don't have to say it do I? :)

[Use this if it helps: Web Analytics Measurement Framework. Though for impromptu analysis you don't have to get that detailed. Just keep the framework at the back of your mind.]

5. Focus on the obvious, and the non-obvious.

Even if you spend only 30 mins on doing some analysis try to say something that I won't anticipate by spending 5 mins on the site's home page.

Surprise me [/ your boss / your audience / children / god].

Here is an example.

I can guess the Macro Conversion on site in two seconds. So tell me about the three Micro Conversions that are not obvious but of great value to the site.

Say you looked at Williams-Sonoma. Points for telling me about ecommerce. Bonus points for grasping and telling me how to improve qualified sign-ups for the Williams-Sonoma Catalog (which brings a lot more revenue in the long term than a quickie online order). Or how to improve number of brides creating Wedding Registries (huge money there). Or memberships to the Wine Club. Or Gift Cards (which are essentially customers making interest free loans to Williams-Sonoma!).

Surprise me.

Visit the website of the site's biggest competitor and tell me two things they do well that you think your site should.

Dig out industry standard scores for Customer Satisfaction & Task Completion Rates and use that to tell me areas of opportunities.

Give me three specific ideas for A/B or Multivariate tests and state your hypothesis for what will change.

Present your analysis / recommendations in a different format.

Shock me by including a framework you use for your recommendations (which one person did, it looked like a house! so amazing!).

Postulate a good enough reason to use Social Media (not just because everyone is doing it).

Tell me about how the inevitable demographic shifts in the US population will destroy the current business that this company has.

Surprise me.

If Scott or Brett or Dai or Trevor or someone else can spend a few minutes on the website and come to the exact same conclusions as you then it is unlikely that your analysis will be as impressive as you think it should be.

So… focus on the things that will be obvious to many and then include at least one non-obvious thing that almost no one will focus on because only you, the unique awesome genius person that you are, will see it.

Summary: Don't just offer opinions, think things through, offer data, clarify what you are solving for and finally do at least one thing that falls in the non-obvious category.

Amongst the submissions that was presented there were some common themes in the I was quite delighted by.

Here are a few of them, you should do these too when you do analysis…

1. "Why before the how"

Almost everyone focused on redesigning the home page, with one holy goal in mind: Make the value proposition of the company really clear really fast.

I love that!

One person framed it so well: "Address the why before the how."

Brilliantly put.

Use that mantra every day.

Some things were common in many submissions, and these I really really liked:

2. Obsess about SEO.

Some folks diligently focused on SEO, and I LOVE SEO!

From garbled urls to missing title tags to poorly linked internal pages to missing site maps. I am so happy people found these things (and EVERYONE of you can too with basic knowledge of SEO!).

It is "free" traffic, but more than that it is investing in the long term success. It is pretty attractive to jump to Paid Search recommendations or doing more Email Campaigns. You should do that, but if you come to me with that and not mention SEO you are going to break my heart.

[Even if you are an Analyst I expect you to have the knowledge described here: Official Google Search Engine Optimization (SEO) Starter Guide.]

3. Be different.

I covered this a bit in #5 above. But wanted to share more context with you.

In their analysis some people tried to be different. That is always a good thing.

Instead of sharing a site and three things one person shared three things they would change about the state of Texas!

Made me smile (and I sent him a free copy of Web Analytics 2.0 :)).

On a serious note… you know the obvious things people will say in these situations, and so do the HiPPO's (they have heard it all before). Try to be different (though not Texas different!).

4. Be sweet.

Without exception everyone was very sweet. Most people tried really hard to send me the best submission they could. I got special graphs, images, wonderfully formatted word documents… so much.

It was so nice. I feel profoundly grateful.

Life is short. Be sweet to those around you. They'll reflect it back. One person at a time we can make the world a better and less bitter place.

Closing Thoughts.

I recognize that you won't do all of the above for an "impromptu analysis", else there would be nothing impromptu about it.

I hope that you'll take the principles outlined in this blog post and make them a part of your DNA. When you are asked to do some quick analysis that you'll activate these principles, even without thinking about them too much.

When I have to analyze a site I quickly make a note of the two or three objectives of the site (and one of those falls in the non-obvious category). I log into Compete and Trends and get some data about clickstream. I see if there are clues in Insights for Search and Ad Planner about the site's business. Then I write down two of three things recommendations / fixes that I can back up with data, or in case of no data formulate and preset a couple hypotheses for testing.

It takes me between 30 mins to an hour. I won't change the website's trajectory in a massive way, but I'll definitely give them some concrete things that will have a short term noticeable positive impact.

And you can too!

Ok now it's your turn.

What is your approach when put on the spot and asked for some analysis of a site you don't own? What are one or two techniques that work for you? Thoughts on the above nine principles?

Please share your critique / approaches / feedback in comments below.

Thank you.