This is part 1 in a two part series about tracking on site search with Google Analytics. In this post I outline the setup and configuration you need to perform to create accurate, usable data in the new reports. If you’d like to get a look at the reports, you can skip to part 2, where I talk about the reports and how to use them.
How to Set It Up
The new search reports are based on a query string parameter that contains the search term that the visitor entered. All you need to do is specify the name of the parameter in the profile settings. As GA processes the data it identifies the search term and builds the reports.
It is important to note that on site search processing happens BEFORE filters are applied. This means that we have no way to modify the way that search data appears in our reports. I personally believe that this is a mistake and it limits our ability as practitioners to generate the data that we need for analysis. I’m hoping that Google can somehow change this.
Basic Setup
The first thing you need to do is turn on site search reporting. This is easy, it’s an option in the profile settings (please see the image below).
Next, you need to tell Google Analytics which query string parameter contains your search term. The best way to identify the query string parameter is to do a search on your site. If I do a search on my site for ‘world series’, and the search results page URL looks like this:
http://www.website.com/search_results.php?q=world+series
then I would add ‘q’ to the search term field. Pretty simple, huh? The query string parameter goes in the ‘Query Parameter’ field (again, located in the profile settings).
You’ll notice that I’ve added two query string parameters in my example. The epikone.com website has two different search engines. You can define up to 5 parameters. This gives you a lot of flexibility if you are using multiple on site search engines.
Category Setup
Many search engines allow users to refine their searches using categories. For example, if I search for ‘lavender’ the search engine might let me focus my search in the ‘soaps’ category or ‘tea’ category. This configuration drives a specific report that show the search terms for each category.
Setting up the categories is the same as setting up the search terms. All you need to do is identify the query string parameter that identifies the category. In the following example I would enter ‘cat’ in the Category Parameter field.
http://www.website.com/search_results.php?q=lavendar&cat=soap
Other Options
There is one final setting that is very important. You may have noticed the radio buttons for strip query parameters out of URL. This setting will remove the on site search query string parameter from the URL after on site search processing is complete. This means that the query string parameter, and the search term, will be removed from ALL other reports. It’s very much like the exclude query string parameters setting in the profile settings.
If you choose yes, GA will consolidate pageviews for the search results page. This can be good and bad.
Here’s an example. If you do a search on my blog the URL will look like this:
http://cutroni.wpengine.com/?s=world+series
Now, if I choose to remove the query string parameters GA will process the site search and then merge the resulting URI with all other instances of
/blog/
As you can see this is going to INCREASE the pageviews for /blog/. Is that good or bad? In the case of my blog it’s a bad thing because /blog/ is also the homepage. Removing the query string parameter will artificially inflate pageviews for the homepage.
However, if your on site search engine has a specific results page, like:
http://www.site.com/seach-results.php?q=world+series
Then removing the query string parameters is a good idea. The reason is you’ll have a single line item for your search results page in your reports and you’ll still be able to analyze on site search data using the On Site Search reports.
Common Problems
What if your on site search results page does not contain the search term in the URL? This is usually the case if your search form is using the POST method.
The workaround is to alter the way the search results pageview is created. This means manually creating a pageview for the search results page, that contains the search term. This is done by passing the tracking code on the search results page a value. Make sure that value includes a query string parameter containing the search term. Then specify the query string parameter in the on site search setup.
Here’s how you can modify the tracking code on the search results page to resolve this issue:
Old urchin.js implementation:
urchinTracker('/blog/search?q=
New ga.js implementation:
pageTracker._trackPageview('/blog/search?q=
Where
NOTE: If you haven’t heard, there is a new GA tracking code. You can read more about it in this post.
Optimizing the Setup
One thing that I’ve noticed is that the reports do not massage the data in any way. They report the raw keywords that the visitor entered. It’s great that we can see raw data, but it’s very easy to have duplicate data based on capitalization or spelling. Here’s an example. The following terms would all appear as different line items in the search reports:
Red Sox
red sox
redsox
red socks
From an analysis standpoint, the above terms are all the same. Visitors are searching for information about ‘red sox’. A certain amount of normalization would help, but we don’t want to eliminate too much data. Unfortunately the site search processing happens BEFORE filters, which means that we can not use filters to modify the data. I had initially thought we could use filters, but that is not the case.
Any data normalization MUST be programatic, it can not be done with filters! So, if you want to force all of your search terms to lowercase you must modify the page tag on your search results page. Specifically, you need to change how the pageview is created in GA. Here’s a sample piece of code that will force the query string parameters to lower case. You’ll need to update the tracking code on your search results page.
New ga.js code:
pageTracker.trackPageview(document.location.pathname + document.location.search.toLowerCase());
Old urchin.js code:
urchinTracker(document.location.pathname + document.location.search.toLowerCase());
This code change will eliminate all duplicates caused by capitalization.
Unfortunately there is no easy way to remove duplicates caused by misspelling. If we could only use filters… :)
Other Uses
What I find interesting is that this framework can be used to track other things. For example, let’s say you have an online ticketing service and the search feature passes certain search characteristics via the query string. I perform a search for tickets to see the Red Sox play at Jacob’s field. I specified the price range of the tickets and the date of the game. The URL might look like this:
http://www.tickets.com/search?price=100-200&date=20071014-20071016
What if I configure the new search reports to use price as the search field and data as a category? The reports should show me some really cool information. I haven’t tried this yet, but it could be very useful.
Hi Justin,
Sorry, but I really don’t understand where to “add ‘q’ to the search term field”… or “All you need to do is specify the name of the parameter in the profile settings.”. Exactly where on Profile settings?
Till now I measured on site search like this:
Content > Top Content > Find URL containing: s=
(this search term field is for WordPress blogs, but for others it can be any search term field)
Thanks in advance for reply!
Nebojsa
Hi Nebojsa,
This feature has not been released yet. When it is released you’ll see a filed in your profile settings where you can enter the parameter that identifies your on-site search parameter.
Your current approach is the way that most people measure on site search. But the new reports are a HUGE improvement, you’ll be amazed.
Thanks for reading,
Justin
When should we anticipate this feature to release? And can you please write a post on how to enter the parameter that identifies the on-site search parameter, I don’t think I’m getting the picture yet. Thanks so much!
These features should be out in the next few weeks. Once the admin interface is updated I will update this post.
Justin
Nice post Justin,
But i have a doubt.
How to implement the “Site Search” feature when the search is being performed on a different server(i.e different GA profile) that the Website itself.
Let me elaborate a bit.
What if we have created a GA profiles for the site http://mysite.org. And there is a searchbox on each page. When a user types in a search query it is sent to http://search.mysite.org?query=searchquery&start=0&end=10. A seperate GA profile has been created for http://search.mysite.org.
Now in situations like this what workaround can we do to be able to use the benefits of the “Site Search” functionality??
Any insights would be helpful.
Thanks,
Jahangir
So if we want to know what search term someone is using, we enter just the query parameter “q”, yes? GA also does not explain what it means to either strip the query parameters out of the URL or leave them in. Is that for the ‘cx’, ‘cof’, and ‘sa’ parameters, which appear to stay the same?
Hi Justin and thanks for this post!
A quick note to those of us who started sith search URLs including %20 characters to replace spaces in search phrases, you definitely need those spaces to be replaced by plus ‘+’ signs instead so show up nicely as spaces in the keywords report!
Cheers,
Julien
Hi Jahangir,
In that situation you should configure GA for sub domain tracking. Then you’ll be able to create a profile for all mysite.org traffic (including search). You’ll also have the ability to create profiles for the various sub domains you may have set up.
I’ll write a post about third party domain tracking this weekend.
Justin
Julien,
Thanks for the great tip!
Justin
Hi Scott B,
Yes, all you need to do is enter the query string paramter that contains the on site search term in the Query Paramter field.
I re-wrote the post to better clarify the setup. I think the new post will answer both of your questions.
Thanks for the question and thanks for reading!
Justin
Juliens tip to replace “%20” by ‘+’ signs indeed is a good one – but how do you do that, since filters do not have any effect on the on site search reports ? Can you do it also programatic, just as you explained us in the post for forcing all search terms to lowercase ? And what is the code that you have to use for doing so ?
By the way Justin: a very good post on the poorly documented new on site search reporting of GA ! I hope that you will include this in the next update of your O’Reilly Shortcut – which is I believe the best reference book you can buy if you want to digg into GA seriously !
Hi Luc,
First, thanks for the kind words. I am including this content in GA Short Cut along with a section about event tracking.
Regarding the character replacement, yes, the character replacement must be done programatically in a similar manner to forcing the keyword case. %20 is encoding for a single space. All %20 and white spaces should be replaced using a plus sign.
As for the exact code, try:
I hope that helps and thanks again for reading the blog.
Justin
Man, the use of filters would really be useful. I actually spent time on Friday writing a filter for site search, only to come in this morning and see your blog post. Doh! Back to the drawing board, I suppose.
What could be the reason for pre-processing the site search, but post-processing goals?
By the way, what is that animal on your ebook and how was that one chosen?? :)
I hope you’ll forgive me if I go off at a tangent, please… The information about the application of filters and the order in which Google processes information prompts me to ask if e-commerce information is also processed before filters are applied.
I’ve experimented with using filters to set campaign source, medium and name based on the existence of a single parameter in situations when it is not possible to tag inbound links properly.
This seems to work for the traffic reports, but not for e-commerce.
Do you know if perhaps the filters are applied too late for them to affect e-commerce reports?
Hi Justin,
I have made the set up of the site search and it is now working, however key words used on my site is in arabic language , and it appear in the report as something like “%C7%E1%E4%E5%D1%ED%E4”
How can I fix that?
Thank you
Dalia
Hey Jenny,
All of the SHort Cuts have the same cover animal, I had no say in the choice. In fact, I’m not sure what the animal is. I’ll ask at some point.
Thanks for reading!
Justin
Hi Tim,
I believe that the conversion is attributed to the campaign information that is present in the cookies at the time of the transaction. I’m pretty sure that you can not create campaign information, but you can modify existing campaign information.
Sorry for the lack of info on this one,
Justin
Hi Dalla,
Great question. I’m not sure there is a solution because GA is not available in arabic. I would write to Google. If enough arabic speakers complain they may add an arabic version.
Thanks for reading the blog.
Justin
Hi Justin,
Couldn’t agree more on the pre-filetring processing!!
What exactly is the limitation?
Does it work with exclude/include filters?
Quick example: If I have two profiles one for the public traffic an another for public, Can I watch only the search from the public traffic on the right profile or are they will be mixed?
I know it doesn’t work with rewriting Request URLs to include the query strings…
Great post by the way!!
Andres,
Processing is done on a profile by profile basis. The settings for one profile will not interfere with the settings for another profile.
Hope that helps,
Justin
Hi Justin,
Following the instruction in this post as well as in http://www.epikone.com/blog/2007/11/19/tracking-sub-domains-with-google-analytics/ I was able to implement the Site Search feature. But there is one problem that im facing. When I go to Site Search>Start Pages all i get is (entrance) and when go to Site Search>Destination Pages all i get is (exit). It doesnt show me the pages on which the searches was actually made.
Have I missed something that has caused this problem ?? Any ideas ??
Thanks,
Jahangir
Hi Jahangir,
I have not seen that issue before. My guess is that the sub domain tracking is somehow interfering with the search navigation.
My suggestion would be to create a profile specifically for the sub-domain with your on site search and check the data. If you can eliminate the sub-domain setup as the cause, then you can start to look for other causes.
Hope that helps and thanks for reading,
Justin
Justin,
I created a “main” profile that didnt contain any sud-domain filters and now i am able to see the Start and Destination pages.
Thanks for your tutorial and keep up the good work.
-Jahangir
Hi Justin,
Thanks for this Blog, it is a huge help.
In the Site Search configuration, you can list five variables that may contain search strings, but it appears that you can only have one of those variables in the URI at any given time. Are you familiar with the method for utilising multiple parameters (5 in this case) and can you confirm or dismiss the above assertion?
Many thanks,
Robert
Hi there,
I’m trying to track internal searches across sub domains. I have a main profile and internal search works. Although it shows [exit] and [/cgi_bin/] pages only.
My sub profiles are all working but are not tracking internal search at all. I have them all set up the same as the main profile (with sub domain filter) but they aren’t working. Any advice appreciated.
Thanks,
Steve P
Hi Steve,
Is the site search located on a different sub domain that is included in the sub domain profile? I’m wondering if you’re excluding the search data by accident with the sub domain include filter.
Let me know if that helps,
Justin
Hey Justin,
I’ve been trying to do some research on how to get GA to track my searches and have been fairly unsuccessful. Even after reading this entire post I’m still not exactly sure what I’m supposed to put as the query parameter. Here is what I get when I search for “tubing.”
http://www.mysite.com/search/search.cgi?p=1&lang=en&include=&exclude=&penalty=&sort=&q=tubing&mode=all&submit.x=40&submit.y=16
I haven’t seen anything in GA when I use “q” as the parameter so what am I missing? My IT guy has way to much on his plate to give this any attention so any help would be greatly appreciated.
Jason,
Sorry if my post was not clear. Based on your URL you want to enter q into the ‘Query Parameter’ field in the site search settings. You can find Site Search settings in the Profile settings. You’ll need to enable site search first. Then you should see the ‘Query Parameter’ field.
Hope that helps,
Justin
I found the site search setup and entered q as the query parameter, however I haven’t seen any tracking in Google analytics. I’ve used the search box on my website numerous times hoping to see some tracking but still nothing. Should I select strip query parameters out of URL? Or is it possible that I am using categories based on the URL I gave you? I figured that q was my parameter and had it set before I posted the first time, but apparently I’m still missing something. Thanks
Hi Jason,
Sorry, but without digging into your site and looking at the stricture I’m at a dead end. The most common problem is that the search term is not in the search results page (ie the search form is using a post to pass the search term to the server).
My lsat guess is that you’re passing a value to _trackPageview() that’s overwriting the q parameter in the URL.
Hope that helps,
Justin
Hi Justin,
Have a client with a site that does not pass the search term as a url parameter. It is an online classifieds site with subdomains for each city and every page has a search box.
We want to implement site search and also ensure that sub-domain tracking is unaffected.
Is this how the customized GATC would look?
var gaJsHost = ((“https:” == document.location.protocol) ? “https://ssl.” : “http://www.”);
document.write(unescape(“%3Cscript src='” + gaJsHost + “google-analytics.com/ga.js’ type=’text/javascript’%3E%3C/script%3E”));
/script>
var pageTracker = _gat._getTracker(“UA-xxxxxxx-x”);
pageTracker._setDomainName(“.domain.com”);
pageTracker._initData();
pageTracker._trackPageview(‘/search?q=’);
Any help would be much appreciated.
Thanks,
Raina
Hi Raina,
That looks good.
Thanks for the question,
Justin