Unfortunately (and somewhat intentionally), Google has made Insights for Search a bit of a black box, with the data stripped of any meaningful scale aside from relative to what’s being compared. That’s not to say that you can’t draw some meaningful conclusions, so long as you have a thorough understanding of how the tool works. This post is the first in a series aimed at giving you such an understanding. Nowhere else have I seen this much detail on Insights for Search in one location; I put this guide together by piecing information from a variety of sources.

Last, if you find something you disagree with, or know is outright false, I’d love to hear about it in the comments.

So here we go!

Part 1 – What is Google Insights for Search Anyway?

In short, it’s a way to glimpse into the mind of the average internet user. Specifically, the average internet Google user. What’s that good for? I’ll get into that later, but for now, trust me, it’s good for something.

It provides near-real-time data (2 day lag) on keywords that have high enough volume. What is “high enough”? That’s not that clear, but you have to be getting into some fairly obscure terms before Insights will stop giving you data. The data you are able to get is “aggregated over millions of users without personally identifiable information,” and “is powered by HAL 9000 computer algorithms.”

You are able to compare up to five terms (or groupings of terms) in one analysis. Although, with some work, you can extend beyond 5 terms (more on that later). For any given comparison, you have the ability to look at the following outputs*:

• A histogram with the search volume, indicating interest over time, plotted on a normalized scale from 0 to 100
• A breakdown of the categories that the terms are classified into (provides context to terms, e.g. how many people are searching for Ford, the car company vs. Ford, the former president)
• Lists of the top searches and rising searches
• A heat map graphically displaying the search volume index with defined regions, cities, and metros (based on IP address)

But before looking at the results, let’s first take a detailed look at the interface.

There are three basic sections to the interface – “Compare By”, “Search Terms”, and “Filter”

Compare By:

Search Terms:

Locations:

Search Terms

Filter

Finally, in the filter section, you can adjust the parameters that set the context for your comparison. In other words, do you want to compare three terms for the last 30 days or the last 2 years? In the United States, or in Canada? Specifically:

• Web searches – google.com
• Image searches – images.google.com
• News searches – news.google.com
• Product searches – products.google.com

*Source: http://www.google.com/support/insights/bin/answer.py?hl=en&answer=87276

**Source: http://www.google.com/support/insights/bin/answer.py?hl=en&answer=96542

Much has been written about what happened in that month, and at some level it’s safe to say that Brown’s campaign embracing the digital space above and beyond the traditional political campaign played a major role in his success. But what is up for debate is whether or not his tactics should have other campaigns rushing to imitate. After all, there’s only so many hours in the day, and so many days in the week.

If every election had similar conditions to the one between Scott Brown & Martha Coakley, the answer would be pretty straightforward: candidates should be investing heavily in social media, leveraging rapid, cost-effective platforms for interacting with broad bases of support.

But the Scott Brown election was special, and while other elections might share some similarities, the following conditions combined into the perfect storm for Scott Brown’s digital revolution:

Overconfident opponent who ran a lackluster, mistake-ridden campaign

Perhaps the single most important component to Brown’s victory was the abject failure of a campaign run by Martha Coakley. Beyond the stunning lack of digital tactics, Coakley stumbled over and over after winning what was assumed to be the more competitive Democratic primary. With even a semi-competent opponent, the groundswell behind Brown would have been caught and caught early, extinguishing the flame before it was too much to control. This campaign is a classic example of the value in keeping tabs on your competition.

Impending healthcare debate raised regional race to national significance

Without healthcare reform, Brown wouldn’t have had the finances to compete. Most local elections don’t have the benefit of a looming national issue hinging on it. By figuring out that he was in a highly unique position, Brown watched the money pour in from concerned citizens across the country. Think that would happen if Brown would have been vote 39 or 45 instead of 41?

Special election had nation’s undivided political attention

In addition to having a controversial issue to tap into, Brown didn’t even have to compete for coverage at a national level, making it easier for advertising and PR to reach peak effectiveness. Think about if this had been a normal election cycle? If there were 60 other Senate seats up for grabs, how could Brown have ever defined himself as vote 41?

Increased apathy amongst Democrats (Obama letdown)

Further compounding Coakley’s shoddy campaign, internet-saavy Democrats who had made such a difference in Obama’s campaign have experienced a post-election let down. This points to perhaps the largest concern with heavily utilizing social media – when you really everyone around your underdog image, and then you win, you’re no longer the underdog.

Growing Tea Party Movement a motivated, pre-organized grassroots base

Of course, social networking becomes a heck of a lot easier when you have an existing network ready and waiting. Then it’s more of a communication channel than a recruiting tool, and your critical mass has instantly been developed. The Tea Party movement has been heavily leveraging free social media tools as cost effective, efficient means for self organizing. Scott Brown merely had to flip the switch.

So?

Good question. While I think social media can be a tremendously effective tool, and should absolutely be a part of any campaign (be it political or commercial), the level and depth to which social media is employed needs to be carefully thought out. If you’re an up and comer looking to slay Goliath, social media should probably figure much more heavily in your strategy – it’s a highly effective means of organizing and rallying supporters behind your cause, generating momentum without breaking your budget. You can quickly become the poster boy for change, the rallying cry against the status quo, and the hip and trendy insurgent.

But if you’re the category leader or the incumbent, you’re simply not going to see the same empassioned response. Your social media presence becomes much more of a defensive play against any potential Davids lurking around your ankles. Your use of social media becomes much more PR-focused, monitoring and correcting the record where needed, and gleaning as much consumer/voter insight as you can.

In other words, the decision is not whether or not to use social media, but how.

Or is it?

What did you look at to determine if your traffic was up? Hits? Assuming even a mild sense of internet sophistication, hopefully not. (If you did, quickly acclimate yourself to How Idiots Track Success, and then grab a cup of coffee with good old Ted).

Maybe you looked at visitors? Unique page views possibly? Time on site? Bounce Rate? Pages per visit? These metrics are getting warmer, but still not quite there…

Evaluating whats working and not working in your little corner of the internet requires a deeper, more thoughtful look beyond raw numbers increasing or decreasing. Avinash Kaushik offers some great insight:

Click-stream has become a lot more sophisticated. What you should measure is quality.

Who’s coming to your site? Is everyone in purchase mode? Lets say you’re running a small business that relies heavily on staffing, and you are constantly recruiting employees. Are your metrics improving because customers are finding your site easier to use, or because your visitor demographics are changing? Did you bump up your search marketing only to be more easily found by job seekers? If attracting talent isn’t a goal for your site, job seeking-visitors are muddying up your numbers.

Conversely, your metrics could be in decline not because things have taken a turn for the worse, but rather that you’ve improved the quality of your clickers. If you run a commerce site, your pages per visit and time on site declining could certainly signal lower customer engagement and interest, but could it also signal that those changes to your check out process you made are working? Or that your new product organization is making it so much faster and quicker for customers to find what they’re looking for?

Scenarios similar to this one highlight the need for a clear understanding of the following questions:

1. What are you trying to accomplish with your website?
2. What are the relevant metrics to track progress against those goals?
3. Should you be segmenting your visitors before tracking these metrics?

Without answers to these, that low bounce rate really can’t be entirely trusted.

I’m not referring to their opinions on your color pallet or whether or not they like drop down vs. static menus. Think bigger picture. When someone steps into your website, what mindset are they in? Do they know what they’re looking for? Are they ready to purchase as soon as possible? Have they even heard of your company before?

You might be thinking that the average visitor is in ready-to-purchase mode when they hit your site, and that could actually be reality. But I would like to pose that you consider changing the average mindset of your visitor in the name of increased revenue (and profits, of course!). Travel upstream in a customer’s journey to their purchase and drive them to your site during their discovery/exploratory phase. Why? More after the jump…

Let’s put ourselves in the mindset of a customer. Let’s say we’re going to buy a new flat screen TV. What would be your first step once you’ve decided you’re in the market? Would you start price shopping immediately? Would you look up what brands made such a TV? Maybe you’d look for reviews from consumer reports or CNET or something similar. Or would you immediately go to Best Buy and buy the first TV you see?  Maybe, but you’d probably agree that you would be in the minority of what most people would do.

A 2007 study done by Accenture revealed that almost 70% of consumers prefer to make purchases in person, but almost 70% first go to the internet research product features, compare prices, or generally locate items. A mere 13 percent of consumers responded that the Internet added no value to their shopping experiences. If you’re targeting your online efforts purely towards people who are ready to buy now, you’re missing out on a huge chunk of consumers. Of course, it helps to have a physical presence to drive consumers who just won’t buy online to, but even purely online retailers can stand to capture a customer who’s in research mode.

Get consumers to your site when they first begin looking. Drive your brand’s top-of-mind awareness by showing up early and often during the information gathering phase. Target research-oriented (in addition, or even as opposed, to purchase-oriented) keywords. Add comparison-shopping tools. Do whatever you can think of to make your site a place consumers can come to get informed and then buy, instead of just to buy.

Think of your site just like an old-school hardware store; staff it with knowledgable resources who know about every product in the store, and can tell you why you should choose this hose clamp over the other one for your home improvement project. Start thinking about more than just clicks to your “check out now” button.

Less than 10% of online retailers’ web traffic, on average, comes from search engines…

…the majority of retailers’ web traffic (61%, on average) comes from people going directly to a retail site — consumers typing, say, Amazon.com into a browser address bar.

…many people use search for product research, which influences their behavior down the line. He illustrated an example: You search for a red cashmere sweater, looking for who has the best deal. You see J. Crew has it for $119 and Neiman Marcus for $329. You don’t buy it then but later head over to jcrew.com and make the purchase.

So what’s the answer? Are consumers really steering away from non-branded paid search because retailers like Amazon have such massive brand presence? Is branded search ROI significantly higher than general search terms? Should you adjust your marketing investment away from general search terms, or away from paid search completely? In short, no.

Perhaps one of the greatest aspects of on-line marketing is the ability to track results better than almost every other channel. And while all that data is great, it’s only useful when it’s interpreted correctly. Worse still, data interpreted incorrectly can be severely detrimental, sending you spiraling down misinformed decision after misinformed decision into major losses. In the case of paid search traffic for retailers, and really traffic in general, it is critical to understand how visitors are discovering and interacting with your site, and the supplementing that knowledge with your data.

Are your customers making quick, impulse purchases, or are you selling something that requires a lot of comparison shopping or education? Are your customers making repeat purchases, or is most of what you sell a once every 2-3 years type purchase? Are you selling to consumers, or b2b? These are just some of the questions that should be used as a lens into that precious traffic data you need to draw insight from.

Let’s look at an example: My company is Acme Thumb Tacks, and I’m selling thumb tacks. Lots of thumb tacks.

I look at my SEM dashboard and see the following results for the three keywords in my campaign:

Now, I can draw a lot of instant analysis from this data. I’m converting 1 out of 3 visitors searching for “Thumb Tacks,” 1 out of 2 visitors searching for “Red Thumb Tacks,” and every visitor searching for “Acme Thumb Tacks.” Additionally, my cpc for “Acme Thumb Tacks” is a fifth the cost compared to the generic search, and a third of the cost compared to the color search. For simplicity, let’s assume that every transaction is the same value: $0.65. In this case, the generic search term appears to not be profitable. But is that really the case?

If each customer only buys from you once, then it is, assuming that customers aren’t doing much comparison shopping or need to be educated on the finer qualities of a good thumb tack. If they are comparison shopping, the may discover your site searching for thumb tacks, but not actually make a purchase until they have browsed your competitors. When they’re ready to make the purchase, they might just type your URL in directly, search again for “Thumb Tacks” and click on your add, or (most likely) search for “Acme Thumb Tacks” and click on your ad. And now things are muddied; should that sale be attributed to your branded keyword, or your generic one? In a perfect world, you’d create a group of both keywords (because it took both to get the customer to your site) and assess their combined profitability. But when you’re dealing with hundreds of keywords, that’s just not feasible.

So it goes back to knowing how your customers shop, and making educated decisions on how you allocate your budget. If you have the time and resources, track your spending choices over time and regress the data against overall performance. Just remember, a keyword may have a negative ROI on paper, but it could just be because the metrics that are tracking it aren’t sophisticated enough to capture the true impact.

Never make decisions in a vacuum.

If you’re using Google Analytics, your paid search efforts will not automatically show up correctly in your traffic stats. Google Adwords is the easiest to fix by simply clicking a check box in your AdWords account settings to link to your Google Analytics account. Yahoo Search Marketing and Microsoft adCenter are a bit more complicated. Both offer placement in search results and on their respective content networks. In Google Analytics, search results placements will be recorded incorrectly as organic traffic. This is a lose-lose – you gain no insight into the actions of pay-per-click visits and muddle your understanding of your organic search performance. Similarly, content network placements will show up incorrectly as referring sites. Not quite as detrimental, but equally useless in determining cost effectiveness. The good news is, with a little bit of work, these problems can be overcome.

Step by step guide and a before and after example after the jump…

Step 1: Get Ready to Change Your URL’s

Google’s URL Builder will become your best friend for this excercise. If you’re not already familiar with the tool, take a quick read through here as well as here. You will be using the URL builder to change the way Google Analytics interprets visitors from your pay-per-click efforts.

Step 2: Set Your Campaign Variables

The 5 campaign variables used by Google Analytics are fairly straightforward. Two of the five are almost always static:

Campaign Source:

For Yahoo Search Marketing, use “yahoo”. For adCenter use “msn” (both without the quotes, of course). You may consider using “bing” instead of “msn” as Microsoft’s organic results now are labeled as “bing.” Really a matter of personal preference.

Campaign Medium:

For both Yahoo & adCenter, use “cpc”. It’s important you use cpc in all lowercase because this is how AdWords will appear in Google Analytics. So, if you want to take a look at the overall effect of your pay-per-click efforts across platforms, you’ll need to have them all reporting with the same labeling convention.

Campaign Term:

This is the keyword that was searched on and resulted in your ad being displayed and clicked. For this campaign variable, both Yahoo and adCenter provide you with dynamic identifiers to automatically insert keyword information:

Yahoo: Use {OVKEY}

adCenter: Use {keyword}

WARNING: When using the URL Builder, be careful. Placing these dynamic identifiers in the builder will result in them showing up in the URL as “%7BOVKEY%7D”. This is because the URL builder is thinking you actually want to record the keyword not as a dynamic identifier, but as “{OVKEY}”. As a result, it replaces the brackets with an html-friendly substitute. Anywhere you use a dynamic identifier, be sure to adjust your URL’s after you generate them.

Campaign Content:

This is typically the headline of the ad, but you can also use it to track actual search queries (this data can help you mine for new keyword ideas. Again, both Yahoo and adCenter provide dynamic identifiers:

Yahoo: Use {OVADID} if you want to track campaigns by their ad text id number (I don’t find this very helpful) or manually type in the headline for an ad (more useful, but can be time consuming depending upon the size and complexity of your campaigns)

Use {OVRAW} if you want to track the actual search query that resulted in your ad being triggered. (I find this to be the best use of this variable)

adCenter: Use {AdID} for tracking by the text id number (again not all that useful) or manually type in the headline for the ad (again, more useful, more time consuming)

Use {QueryString} if you want to track the actual search query that resulted in your ad being triggered. (Again, I find this most useful)

Campaign Name:

This is just what is says it is, the name (or ID #) of your campaign. Similar to Campaign Content, Yahoo and adCenter provide dynamic identifiers, but they may not be all that useful for your analysis:

Yahoo: Use {OVCAMPGID} or manually enter in the name of your campaign

adCenter: Use {OrderItemID} or manually enter in the name of your campaign

For both I find manually entering in campaign names to be much more useful. Otherwise your traffic sources report will be filled with a bunch of meaningless ID numbers.

Step 3: Build and Insert Your New URL’s

Now that you’ve set all of your campaign ID’s, use the URL Builder to create new URL’s to replace your existing destination URL’s. I find it’s easiest to do this en masse using an exported version of your accounts, as opposed to replacing URL’s one by one, but that’s just me.

Step 4: Test, Verify, Refine

As always, test and adjust until you’re getting the data that is the most useful for your business.

Keep in Mind:

Any time you change your destination URL’s, your data in Google Analytics will have a break in it. For example, if you accidentally entered your campaign source as “Yahooo” and then correct it 2 weeks later to “yahoo”, your cpc data will show 2 weeks of cpc’s coming from Yahooo and then stopping, with yahoo cpc being zeroed out for the first 2 weeks and then beginning. Changes to your tracking URL’s will not modify old data in Analytics.

Example:

Let’s say I run an Ostrich farm and want to advertise to people looking for Ostriches. I’ll put up accounts on Yahoo Search Marketing and Microsoft adCenter with one campaign, one adgroup, one ad pointed at “http://www.ostrichesRus.com”, and one keyword for each:

Campaign Name: Ostrich Campaign

AdGroup Name: Ostrich AdGroup

Ad Headline: Need an Ostrich?

Keyword: Ostrich (Broad Match)

Let’s say someone searches for “I want to hunt an Ostrich” and my ad is triggered. Before adjusting my destination URL’s, visits from these ads would have shown up in Analytics as follows:

To get my clicks recorded correctly, however, I am going to use the following campaign variables in Google’s URL builder:

My new URL’s will then look as follows:

For Yahoo: http://www.ostrichesRus.com/?utm_source=yahoo&utm_medium=cpc&utm_term=%7BOVKEY%7D&utm_content=%7BOVRAW%7D&utm_campaign=Ostrich%2BCampaign

For MSN: http://www.ostrichesRus.com/?utm_source=yahoo&utm_medium=cpc&utm_term=%7Bkeyword%7D&utm_content=%7BQueryString%7D&utm_campaign=Ostrich%2BCampaign

Unfortunately, I’m not quite done. Google has screwed up my dynamic identifiers (because it thinks I want to actually call my keyword “{OVKEY}”), so I’ll need to manually edit those in each link. After fixing that, the links will look as follows:

For Yahoo: http://www.ostrichesRus.com/?utm_source=yahoo&utm_medium=cpc&utm_term={OVKEY}&utm_content={OVRAW}&utm_campaign=Ostrich%2BCampaign

For MSN: http://www.ostrichesRus.com/?utm_source=yahoo&utm_medium=cpc&utm_term={keyword}&utm_content={QueryString}&utm_campaign=Ostrich%2BCampaign

I am finally all set to completely understand what kind of impact my Yahoo Search Marketing and Microsoft adCenter accounts will have on my Ostrich business.

Know of a way to improve this method? Let’s hear about it.