The post on questions to ask WA Vendors started with a observation that it is perhaps more important that you ask yourself a few questions before you get into the Web Analytics tools selection process. These questions would help you understand your company needs in a very unique way which would help you pick the right tool.
The core point of that argument is simple: Most tools selection processes have very little self reflection built in, they might even be cookie cutter processes that might be from some book (or worse from some random blogger!! :)). This is not optimal.
Each company is unique and your approach to picking the right tool should be preceded with a lot of self reflection to ensure you understand who you really are and what you really need. (The “radical” selection process recommended in a earlier post forces this self reflection.)
This post covers three questions that should prompt the kind of critical self reflection that should increase the possibility that you’ll pick the right web analytics tool for yourself.
Summary:
# 1: Do you want reporting or analysis?
# 2: Do you have IT strength or Business strength? Or Both?
# 3: Are you solving for the Trinity or Clickstream reporting / analysis?
Now the questions, context and an attempt at providing some guidance…..
Do you want reporting or analysis?
- Decentralized decision making.
- Company cultures (consensus, “cover your back”, layers of management, matrixed, paperwork driven and so on and so forth).
- Availability of tools / features. History.
- Propensity of risk (as in no propensity to take risk and risk taking actually harmful for career).
- Distribution of knowledge in people / teams.
- Availability of raw analytical brain power.
- And more….. (suggest your own in comments below).
This is a very difficult question to answer because most organizations have a very hard time being honest about what they need, humans are also poor at being self critical.
The other reason this is hard is that its like asking if you like a $0.99 Hershey chocolate bar or a piece of L'Artisan du Chocolat. It would be a sin not to say L'Artisan du Chocolat.
Everyone wants analysis, yet few organizations (especially ones that are greater than 100 people in total size) actually want analysis. They want reporting.
There are numerous reasons for this, including:
The bottom line is that while everyone actually says that they want analysis. Admitting that you simply want reporting would be a sacrilege. Yet most companies simply want reporting. They want a web analytics tool and a web analytics team that simply facilitates reporting that is requested from Marketers, Business Leaders, VP’s, CMO’s and others.
The web analytics team might be told that they should provide analysis (remember given a free choice no one will opt for the Hershey’s bar), but they are not set up to have time or knowledge to provide analysis and if by their free will they do provide analysis then it rarely translates into acceptance and action. Success of the web analytics tool implementation is measured by the number of reports it provides, the number of KPI's it is pushing out, the number of Marketers who say they are getting reports.
If you are deciding on what web analytics tool to choose you should take a really hard self critical look at your company, its decision making structure, its needs and then be honest and decide if you want reporting or analysis. Then decide on what tools in your selection criteria are good at reporting and what tools are good at analysis. Ignore the category you are not interested in (usually analysis).
This is a really hard choice to make, even harder to justify (because everyone thinks they want analysis). But having the tough conversation will ensure that your company will be happy with the choice that it makes, it will in all probability save money (reporting tools are much cheaper) and it is highly likely that on the long run you will be successful.
You choose the wrong tool (say a true analysis tool), it will only turn people off the tool and using data (because it will suck at reporting) and on the long run hurt you a lot more. You choose the right tool for your company (say that is really powerful at reporting) and you will glean a lot more traction since atleast they will look at something and your web analytics people will know what they are getting into. Over time your company, if the culture and org structure and risk taking are all lining up, will get smarter and maybe you’ll be ready to move to a analytical tool.
I am going out on a limb here but currently there are potentially only two true analysis tools: ClickTracks and VisualSciences. They come at massively different price points and have a very differentiated set of features and performance (speed, complexity, depth and breadth, sexiness). They are both right for the right kind of organizations, in case you decide you are it, and both need a deep self reflection on your part before you send a chq.
Other tools are also making great progress getting there, empowering true analysis out of the box to complement their far superior strength in being able to do awesome reporting.
IMPORTANT: Hopefully you’ll see that there is no judgment being made above if reporting is better or analysis is better. Each serves its purpose. The emphasis is on figuring out what you really need and then buying what you really need.
Do you have IT strength or Business strength? Or Both?
Some companies are good at IT (technology), others are good at the Business side of things (marketing, analysis, strategic decisions etc etc). A rare few are good at both, or have environments where the two are the same when it comes to Web Analytics.
Pulling off a successful web analytics implementation is complicated and it is often easy to get it wrong. A professional services company recently shared with me that seven out of ten times they find the tool implementation is wrong (and the client has not known about it for a while, and have been using wrong data).
If you have solid IT (technology) and Business strength in your company then go at it all by yourself and you’ll be fine (here are Web Analytics Technical Implementation Best Practices).
If you don’t have strong IT strength (by that I mean IT who knows and gets Web Analytics and not just standard IT) then go with a partner, say my buddy Justin at EpikOne (Justin let’s talk about my cut of this deal! – all joking aside I have not financial or other ties with EpikOne).
If you don’t have IT strength and you don’t want (or can’t afford a consultant) please consider following this process: How to Choose a Web Analytics Tool: A Radical Alternative.
Here is one last reason to assess IT strength: if you want to do in-house and not asp. If you want to consider hosting the data collection and analysis in-house (say with WebTrends, ClickTracks or Unica NetInsight) then you need some serious IT/Technology strengths (in atleast one person) to pull it all off. Ensure that you are covering this important consideration.
If you don’t have strong Business strength (realize that just by figuring this out honestly and you are already ahead of your peers) consider following this process: Web Analysis: In-house or Out-sourced or Something Else. It provides a great framework for the evolution of a effective and efficient web analytics program.
Are you solving for the Trinity or Clickstream reporting / analysis?
This is a mind-set question. It is a question that tries to judge what you are solving for. It tries to get to help you understand the level at which you are approaching the solution set. It is all about trying to know if you need to buy a tool that will help you “understand clicks” (which is ok) or do a lot more than that. It tries to help you crystallize what your short term goals are and what your long term goals are.
The Trinity mind-set & strategy calls for having a robust Qualitative and Quantitative analysis in your web analytics approach with the goal of: Understanding the customer experience explicitly (via Research), to then influence customer behavior on your site (via Clickstream analysis), leading to win-win Outcomes (via outcomes analysis) for your company.
Your consideration criteria will be vastly different depending on where you are and what your own point of view and approach is in this context. In one case a simple log parser is fine, in another you need a tool that integrates with other data sources and in yet another case you need a tool will pay ball with your data warehouse.
See this post: Web Analytics Tool Selection: 10 Questions to ask Vendors. The questions you’ll ask and what you’ll stress with change with the answer to the question above.
That’s it. Three questions.
A bit of my own self reflection & guidance:
I am on record saying “don’t spend too much time on worrying about web analytics implementation, turn on Google Analytics on your website and follow a simple, but “radical”, process”. Given that I feel a bit embarrassed at publishing two consecutive posts on Web Analytics Tool selection. Both posts combined I suspect highlight how complex the process of selecting a tool can be. To some of you it might even look scary as to how much thought you need to put into selecting the right tool for your company.
To put a nice bow around all this (and to make it easy for all of you), here is how to think about web analytics tool selection & implementation:
# 1) If you are considering a web analytics tool this is the best process to follow: How to Choose a Web Analytics Tool: A Radical Alternative.
# 2) If you reach Step 5 of that process (honest and deeply self critical review) then this post shares questions you should ask yourself / your internal customers / your bosses to do that review.
# 3) After you are done with that step and you are ready to call the Web Analytics Vendors then ask them these questions: Web Analytics Tool Selection: 10 Questions to ask Vendors.
# 4) When you do implement a tool be sure to follow these implementation tips: Web Analytics Technical Implementation Best Practices.
It’s that simple.
My apologies if there is any inadvertent confusion. On the bright side regardless of your size and what tools you own now you have a end to end overview of how to go from nothing to something to something awesome. Good Luck!!
What would be your own self-reflection questions? Will the above questions work for you? Am I missing anything obvious? Is this a good use of time? Vendors what else can we ask ourselves before we ask you to bare your soul? Please share your feedback via comments.
[Like this post? For more posts like this please click here.]
Via 
Avinash you have done a great job here of bringing your full practitioner experience to provide a unique perspective, it is not often that people on the outside can provide the depth and clarity that you have provided here.
"Do you want Reporting or Analysis" can be a full post all by itself. I am sure there will be a lot of debate about that. Thanks for forcing the conversation.
The checks in the mail! I bumped your cut up this week :)
Only ClickTracks and VS? C'mon!
That's like saying deploy Google Analytics at a billion dollar company and don't do an RFP!
Kidding!
Awesome post, arguable… What good thoughts aren't?
Judah
Judah: Ok I'll bite (and I know you are kidding but I am sure others might have the same thought).
(I am open to be proven wrong about this, if there is someone who would like help me with that please email me, blog at kaushik dot net and I will be happy to meet with you or have a phone / email discussion.)
It is my understanding, from my humble experience, that these are the only two tools that will do "post-facto segmentation". I.E. you don't have to know all your segmentation possibilities upfront and remember to pass the information to the javascript tag (log file) to be able to answer those questions after data has been collected.
Most tools will need to be told up front and as you know it is very rare that you would know all the questions you'll want to ask after you get the data. Both of these tools allow you to go back and segment on either Customer Type or Customer Behavior and do so on the fly and fast with lots of possibilities (regular expressions, joins, intersections etc).
The second, and equally important, fact is that you can go back and traverse history and do all of the above for any time period you want. So you can manipulate history and learn and optimize your decisions.
As you evaluate your vendors deeply stress test both of these things to know if they can truly do this. The pat answer will be "of course we can", then it will be your job to understand what that "we can" really means (you can call me if you want help!).
I do point out in the above post that both of these tools might not be right for everyone, they come with their own unique pro's and con's and they each differ from the other in significant ways. End of caveat. :)
Ok that's a fair shot. Stated that way! :)
Here is my context:
If you are going to choose a web analytics tool you should do so from a place of intelligence and not non-intelligence. For the most part you will remain non-intelligent about all the massive challenges (that you in your unique company face) unless you actually have web analytics. Actually having a tool makes you smart about a lot of things (tools, processes, problematic IT, crappy sites, problematic Marketers, missing Analysis brain power etc etc).
When you do a RFP you are doing it from a non-intelligence. So why not get a free tool, implement it in ten minutes on your website (don't use Google Analytics, that's ok, there are other choices) and start becoming intelligent.
Then do a RFP. You will do a smarter RFP, ask smarter questions, have something (GA or alternative) on the ground on your website to compare your new tools to, and you will make a smarter (and I promise much cheaper) choice.
Does this help? Am I explaining myself?
I know you are only kidding but you asked two really good questions. I welcome your argument and I welcome disagreement and I thank you for caring enough to argue with me. :) Thanks so much.
-Avinash.
Avinash,
I'm glad you got my humor. Thank you for the personal email, and I'm glad to exchange knowledge with you in a reply on your blog. And may I reiterate, imho: your aggregate post was a wonderful contribution of very valuable, hard-learned knowledge. :)
1) As for ClickTracks and VS and my opinion on the vendors. They are all wonderful in their unique ways. And some love NDA's! But, my opinion, is that one other vendor you named can do what you describe if I read you correctly (i.e. temporal analytics using lookups, bridges, joins). The challenge is how we define "segmentation," and I've been on the 'net too long to light the flames on that one. Let's defer to Kotler. But over tea sometime, let's chat.
2) Re: RFP. I understand and tend to agree with your clarification regarding the perspective of non-intelligence and intelligence. Very well said, but there are many types of intelligence(s) one needs to consider in WA. I knew years ago some programmers who could malloc() in their sleep but tended to look for corners in round rooms (a band once said "going out to find blind spots…") Your somewhat disparaging past comments about RFP's were somewhat disconcerting to me, which caused this long time lurker to post. I will agree that RFP's can be bloated and filled with misinformed questions and could obfuscate understanding of functionality or technology. They also wonderfully irritate vendors. :)
Here's my quick take (it's your blog! :) and NOT specific to the web analytics industry or any vendors: RFP's, like SLA's, are useful artifacts from the annals of software development. Besides the data and "answers" you receive in responses, the savvy decision-maker looks beyond the data reported by the "yes-only salespeople" and analyzes the response from an anthropological and behavioral perspective. *Directly observed* past and current performance is indeed a predictor of future behavior. The RFP, to me, enables you to observe vendor behavior to answer two important questions to any technology selection process: 1) Can they do the job? and 2) Can I work with them? Anything learned beyond that (and RFP's are informative when done right) is gravy. Your judgment on the appropriateness and applicability of the response to your business requirements and the data collected sets the stage for the deep technical dives, and my favorite dives–the ones about the usage of statistical methods in the WA tools. And it is great to use the RFP as a documented basis for explaining to internal folks how vendors differ–in the vendors own "vendorspeak."
If one crafts an RFP question set that meets vendor expectations but with a crafty strategy behind it (i.e not really being so, but appearing to be another uninformed person asking misguided questions from a perspective of "non-intelligence"), you start the relationship "one leg up" because you are underestimated. Thus, you are able to gauge how your future potential partner will treat someone who they think isn't quite as savvy (like entry-level WA "hires" in the future). And I say "partner" because that's what you need in Web Analytics. One who returns your phone calls and will work with the personalities of you and your team.
So I say play up that angle if you have the power in your organization to do so; be super, unrelentingly nice; try to come off as naive and ask very simple questions and amazingly complex ones; use the rfp opportunity to ask the same questions in different ways to which you know the answers and so on… to see simply observe vendor behavior. I know that may sound a bit odd, but it works again and again and again across industries, time, economic cycles (if you have the money to spend, authority/time to do so) Look for boilerplate responses, which indicates a lack of careful thought. Which vendor actually took the time to read your response and cared enough to delegate a real human to craft a real response? Perhaps a quality team like VS would even have the Great Peterson guide the response. The side-benefit to all this anthropological/behavioral analysis is that you get real data to analyze about almost everything you want to know and a quasi-“relationship” to proctor into the vendor’s organization to ask deeper questions (on perhaps omitted or glanced over answers). The benefit of the RFP done well, imho, is not only the selection of the best technology match for your requirement. For big companies who will spend lots with the vendor over the long run, the RFP enables the savvy manager to select the best partner to match the organizational culture.
I love Google and have been using it since my days working in what used to be called "information retrieval" when I replaced Northernlight.com with it (back when it did results clustering in the "daze" of talk like "NASDAQ 10000.") I wholeheartedly agree that if you don't know anything about WA, then GA is a great place to start before you craft the real RFP you recommend to do the deeper analysis we both require to do our jobs.
ps: I do think GOOG has a very smart cookie that behaves uniquely.
Warm regards and thanks for engaging me,
Judah
Judah: Thanks for providing more context on a effective RFP strategy. I have updated my "How to Select a Web Analytics Vendor" post and cross linked to your comment here. I am sure people reading my perspective on RFP's will benefit greatly from your comment.
Thanks again,
Avinash.
Thanks Avinash. Glad to positively contribute to the great wealth of valuable knowledge you've created here.
Looking forward to your forthcoming book!
Best,
Judah
Very well presented steps Avinash! Thanks for sharing. I've also a link to your article in my blog for reference. Thanks!
To me there is only one real Analytics solution and that is Omniture. It is a rock solid solution with SLA's on their data unlike Google. That is my biggest challenge with Google, if your data matters then why would you use Google? You don't own your data and because there are no SLA's they don't guarantee the accuracy or availability of that data. Omniture does all of the above and so much more. Their new Viral Video tracking is unbelievable.