A while back I wrote a post about how you can integrate Google Analytics with a CRM system. That posts referenced the old urchin.js code and is in need of an update. Plus a lot of people have been asking me about different integration options and I thought it would be a good time to revisit the topic.
So here it is, an update. I’m not going to re-do the entire post, just a few key points.
Why Do This?
This entire hack configuration is based on one simple idea: to build a more robust view of customers/prospects/leads/whatever-you-call-them. The more information we know about people that we sell to the more we can adjust our marketing. This technique let’s us combine where the visitor came from (referral, campaign, direct, search engine) and how often they’ve been to the site with CRM data.
How It Works
We’re going to extract data from the GA tracking cookies and add it to a lead generation form. The GA data will be stored in hidden form elements that the visitor can not see. When the visitor submits the form the GA will be connected to the other information that the visitor entered into the form (usually name, contact information, etc.).
Here’s the data that we’re going to pull from the GA cookies:
* Current referral information (i.e. where the visitor came from)
* Custom segment value (if it exists)
* How many times the visitor has been to the site
You’re probably wondering why we’re not extracting the data via the GA API. More on that in a minute.
The Code
Here it is, the updated code:
The new code differs from the original code in three ways:
First, the inclusion of ga.js. No explanation needed there.
The second change is the addition of the _uGC() function. This function was in the original urchin.js, and because I don’t include that file I need to include the code in the script.
The function is used to parse pieces of data and return certain parts. For example, I use to to parse the cookies (which are just a string) and return the value of the campaign cookie (__utmz), the custom segment cookie (_utmv) and the visitor identification cookie (__utma).
Finally, I added some new functionality to include this visitor’s number of visits. This data is pulled from the __utma cookie (which identifies the visitor). The last integer in __utma is actually a visit counter. I think it is interesting to understand behavior based on frequency so I added this nugget of data.
If you want to see the code in action try this:
In my previous post I included a reference section that documents the format of the campaign cookie and the _uGC() function. Check it out if you’re looking for more technical information.
An Alternate Approach
Remember, all the data we’re pulling is in first party cookies. This means that you could extract the info at the server level rather than the browser level (first party cookies are available to server side code).
You don’t need to do this in JavaScript, as I did. You could use ColdFusion, PHP, .NET or some other server side language.
I’m just trying to provide some inspiration here :)
What About the Google Analytics API?
I’m sure there are a lot of people out there wondering about the GA API and why I don’t just use it to pull data out from GA. The reason is the data in GA is all anonymous, aggregated data. There’s no way to reach into GA and say, “tell me where John Doe came from.” That’s why we pull data from the cookie, because we can combine it with a visitor’s info when they supply it.
Sure, it is possible to pull out GA data and place it in a CRM (like traffic from an email campaign, bounce rate from paid search, etc.) but it’s tough to identify individuals in that data.
Is it impossible? No, not from a technical standpoint. You would need to place a unique identifier in GA to link the data.
But we’re getting into a risky area here.
Remember, it is against the GA terms of service to capture data in GA that identifies an individual. This could be an email address or database ID number. Not my rule, Google’s, and they will crack down.
I’d be interested to hear people’s thoughts on this. Feel free to comment!
Thanks for the updated code Justin. We’ll be sure to use it when we switch to the new tracking code. (We use the old version with great sucess right now!).
Closing the sales loop on marketing is the holy grail of results-driven marketing. Great, informative post.
Hello,
Thanks a lot for this post! One of my clients had some concerns about implementing this kind of script on their website, but I would love them to do so because it would enable me to link leads ans actual sales, providing an ROI for every marketing initiative.
Their concern is:
As a publicly traded company, they are worried about privacy issues and they have raised some questions about the fact that we would be collecting data about a user without him/her knowing it. Every Analytics tool does it, but as you mentionned in this article, they do not relate keywords or sources to specific individuals. This script goes a step further.
My question is: are large organizations using that level of precision when tracking online forms? If so, is there any privacy issue, or should it be mentionned somewhere that this kind of data can be collected by the website?
Since I would really like to convince them to use this script, any argument “in favor” or examples of large organizations using this kind of method would be very appreciated!
Thanks again,
Antoine
Thanks for the feedback everyone, glad you like the post and thanks for reading!
Antoine, to address your concern. This script only executes and collects data when a visitor submits a form. By submitting the form they’re sharing data with you, this form just collects a bit more.
As for examples, I can’t talk about specific clients and what they do or don’t do :)
Thanks again everyone.
Justin
Hi Justin
Thanks for all the great posts.
If I knew that a customer bought a certain product on a certain day, I could use that product ID to link the web analytics data to back end data, often to individual level. So in many cases the capacity is there already to link to PII. I personally do not have strong feelings about companies tracking my behaviour but I acknowledge and abide by the fact that many people are bothered by this. However I find the GA privacy policy vague and contradictory, even without storing down personal details with my analytics tool I often have the capacity to link behaviour to back end demographic data at individual level anyways.
Best wishes
Niamh
Thanks. Great info.
Was looking for a solution to track the quality of leads the sales team is receiving via online channels.
Your solutions is exactly doing that.
Justin, great article. Love reading all these techniques/hacks.
Would really be neat to see more writing on CRM and Google Analytics. Maybe a list of some really popular CRM solutions and how the matching is being done. I assume it’s based on logged-in/registered users so it’s first name/last name or a unique key identified to them.
Would also be neat to see a table showing the pros’s and cons of server-side extraction vs. browser level in-terms of accuracy, how complicated, etc.
Finally, would like to see some articles showing how the Google Analytics data is being fed into the CRM for metrics such as engagement, etc.
Great updated post, Justin! And it was a pleasure meeting you, if ever so briefly, at the Google offices.
Justin as always your posts rock man and I am going to totally suck for doing this but I need some help on this one and cant find the expert advices elsewhere.
Google Website Optimizer Technical Question
Situation is this:
Have a site with product pages that dont see enough traffic to each individual product page for a multivariate test (Google website optimizer) without waiting months for results. For example lets say I have 100 product pages each seeing about 200 views a week. I want to setup a simple multivariate test with different checkout buttons and locations. For this example lets use total of 10 combination’s. If I run that multivariate test on even my most popular product page with only 200 views it will take forever to get solid results and to me Google was just never have enough solid data to calculate. Now if i could aggregate all the product pages into a test 100×200 so 20k views a week that is the kind of data i need GWO to see. So my questions is this:
Does the Javascript really care about the Test Page URL or even Conversion Page URL for that matter as long as the scripts are referenced. In other words I want to make 100 product pages look like one test page they will all have the same scripts on top of page and at bottom of page and then the checkout will always have the same conversion script. Has anyone tried something like this or any expert in JavaScript out there know if this will work? Any insight would be much appreciated.
-parky
I would love to see a comment from Google here re their T&Cs and how to work around it for a CRM.
Hi Justin and Everyone,
My 2 cents for GA API ! If you put unique identifier , lets say the value of a cookie (or part of it), does it uniquely identify individuals ? Also, fromt a privacy standpoint,with cookie or unique value , isnt it just hard to trace down individuals (unless they did a transaction on site?
does it really uniquely identify human attached to that cookie ??!!
On second thought, a good revenue stream for adwords is their API,as every server call makes money for google. If Google Analytics to go that path, it would be great for Google to make money out of the API server calls from Analytics ! However to make that happen this privacy thing really needs a great clarification i believe.
Great work on code as expected !!
Hi Justin,
This is a great article. I thought it was interesting because with that integration, you’ll be able to tie number of visits to desirable action and other info attached to it.
This definitely helps CRM to segment various communications based to user traffic behavior to a certain level.
Great stuff!
Thanks K. I was thinking the same thing about the number of visits. Glad you liked this integration and thanks for reading the blog.
Justin
Thanks for the great blog! Just a comment about the GA TOS. I would say the reason GA has that TOS is that it is against the privacy law (at least here in the Netherlands but I assume in most countries) to exchange data that identifies an individual between two organizations, without the expliciet consent from the end user. As the statistics data from GA is owned by Google, not by the original site, this means this is an exchange of privacy data, hence not allowed, and will never be. So it’s not Google’s wish to have the TOS like that, they really have no choice.
I think to (legally) have real identifiable data (detailed profiles of users for example), the only way to do that is to have software in your own datacenter, not in a SaaS/hosted environment where you don’t own the data.
I would think that if you DO create(from a technological viewpoint) identifiable data in GA, you could get in trouble not only from Google itself (violating the TOS), but also from visitors of your website (if they find out), as you would be tresspassing the privacy law of exchange privacy data with a third party without explicit consent of the enduser (visitor).
I am no legal expert by any means, just my interpretation of how privacy legislation works.
Thanks Martijn for the thoughtful analysis! I’m not sure, but I believe that the TOS for various geographic regions is different and the European TOS is more strict. I could not agree more that adding personal visitor data to GA is a bad idea. Plus, in my opinion, it’s just not actionable. That’s why I like this method a bit better.
I would assume that submitting a form, that has a disclaimer, would constitute “explicit consent”.
Thanks again for the comment and reading the blog.
Justin
The long term issue from my point of view is that the industry (at least the industry I work in: Web Content Management) is working towards a complete customer profile, to build up as much knowledge about the visitor as possible. Only then you can provide the visitor the content/products/information he/she needs, or in the case of CRM, provide sales with the best information about their customers. But with the current IT trend to ‘outsource’ parts of your applications to SaaS solutions like Google Analytics, you cannot legally put identifiable data to these solutions. I am curious how the IT world will solve this the coming years. Asking for expliciet permission is one way to solve it.
But this is all a bit offtopic for this blog entry of course ;)
Hi Justin,
Just read your article. Why don’t you just pull the cookie data out of ga.js with _getLinkerUrl(), and pass back through the utm.gif parameters, so the CRM can just make image requests to GA?
That way you can send back pageviews (conversions) as well as transactions and products.
At least that the way I’d do it :)
Hey Nick,
That’s another way to do it. I’d say it’s about the same complexity :) That’s the thing I love about GA, there’s always more than one way to get something done.
Justin
Justin,
Thanks for the great “how to”. I have converted part of the script to asp.net vb. Hopefully someone can benefit. Code snippets are below.
Ethan
Dim strCookieValue As String = “”
Dim strKeyword As String = “”
Dim strStart As String = “utmctr=”
Dim strEnd As String = “|”
Try
strCookieValue = ReadCookie(“__utmz”)
strKeyword = getValue(strCookieValue, strStart, strEnd)
Catch ex As Exception
‘just swallow exception and move on
strKeyword = “”
End Try
Private Function getValue(ByVal strSearch As String, ByVal strStart As String, ByVal strEnd As String) As String
Dim strRtn As String = “”
Dim strWorking As String = “”
Dim intStart1 As Int32 = 0
Dim intStart2 As Int32 = 0
intStart1 = strSearch.IndexOf(strStart) + Len(strStart) + 1
strWorking = Mid(strSearch, intStart1)
intStart2 = strWorking.IndexOf(strEnd)
‘if no end found take the whole length
If intStart2 = -1 Then intStart2 = Len(strWorking)
strWorking = Left(strWorking, intStart2)
‘replace HTML encoded spaces
strRtn = Replace(strWorking, “%20″, ” “)
Return strRtn
End Function
Private Function ReadCookie(ByVal strName As String) As String
Dim strCookieValue As String = “”
Dim ga_cookie As HttpCookie = Request.Cookies(strName)
If Not ga_cookie Is Nothing Then
strCookieValue = ga_cookie.Value
End If
Return strCookieValue
End Function
Wow, thanks Ethan! I appreciate it!
Justin
Hey everyone!
Has this script been configured to PHP yet? I don’t have the skillset to do it myself, but I am told that’s how I need to have it by a friend of mine.
Thanks!
Hey Adrian,
Unfortunately no. But if you know a PHP developer they should be able to figure out he to translate the functionality out of JavaScript and into PHP.
Thanks for the question,
Justin
Hi Adrian, I have created a solution for this and am happy to help you implement it.
Reach out @arshammm
Justin, you’re the man. I”m following you all over as I love the stuff you post.
Has the script been updated for the new custom segments, rather than Set_Var code. Thanks
Thanks for the great script! I’m testing it at one of my websites and can see the organic/cpc breakdown. However I can’t see any keywords for cpc traffic, just got “-” all over the place; same is for Campaigns. Any ideas how I can improve this?
After 5 hours of trying to figure out how to get this done on other sites. I stumbled upon this one.
Just 20 minutes later and IT WORKS!
This is awesome! (Added your site too Google Reader.)
Hi Justin, I know this is an old post, but it was the only thing I could find right now! As of the other day (6/22/2010) or so, this script (as you’ve detailed here) stopped working on all clients (using ga.js tracking code). Very weird, since we have been using it successfully to pull the info into form submittals for a while (tks!). I mention this here, because I notice that the results of your form here (in the “The Code” section) are now looking like mine (i.e. pretty much blank)
GATC Source: –
GATC Medium: –
GATC Term: –
GATC Content: –
GATC Campaign: –
GATC Segment: (not set)
GATC Number of Visits: 1
I didn’t see much out there on the web about this yet, but I’m guessing the changed some GA configuration, since a bunch of different sites stopped working at the same time!
Anyway, just wondering if anyone has any thoughts/experience with this issue… am I the only one?
Tks!
Guys,
Does this Violate any of Google’s TOS? As I understand it, it does not. Can anyone confirm this?
Anyone try Ruby Script like this:
def set_analytics_session_variables
#utm_source, utm_medium, utm_term, utm_content, utm_campaign
session[:utm_source] = params[:utm_source]
session[:utm_medium] = params[:utm_medium]
session[:utm_term] = params[:utm_term]
session[:utm_content] = params[:utm_content]
session[:utm_campaign] = params[:utm_campaign]
end
Follow up question to my PHP one from above; how would one go about translating this to the new Async code? Thanks for any help!
Hi,
I believe this code is different from the code you put on your GA Analytics book.
I have one qustion about tracking the campaign name. Does it mean that we wont be able to get the campaign name if we have Auto tagging enabled? If we are not able to do so do you know if there is a solution for tracking the campaign name as well. I believe campaign name is an important information.
@kaan – Correct, if you have auto-tagging enabled you won’t be able to get the campaign name. That info is obscured. The only thing we can infer from auto tagging is that the visitor came via Google AdWords.
@Adrian – I’ll try to get an async version out soon.
@Taylor – No, this does not violate the TOS. Google prohibits you from tracking any PII in Google Analytics. We’re just taking the cookie data and sending it to a CRM when a visitor tells us who they are.
@Justin – how about async version? You wrote in last comment : @Adrian – I’ll try to get an async version out soon.
@MarekTus – Things have been busy! With the merger, and the new version of GA, there has not been a lot of time to work on an async version of the CRM code. But it’s still on the to-do list. Sry for any delays!
Can you use this method to extract Google Analytics Campaign names? We recently started using the multi-channel funnels feature of the analytics. Ideally, what we would like to do is have a hidden form field that shows all of the traffic sources that have caused the visitor to fill out the lead form. For example, they might have initially visited the website from a campaign called 2012 Expo. They didn’t fill out a form at the time. But a week later, they came back to the website after clicking on another campaign called 2012 Join Expo Magazine Ad. At this point they fill out the inquiry form. With multi-channel funnels, you can see the full history of campaigns / sources that lead up to someone fill out the inquiry form (at least for the past 30 days). Can the hidden field contain both “2012 Expo” and “2012 Join Expo Magazine Ad”? Or can it only extract one campaign / source? If it can only extract one (definitely not ideal), which one does it use? The last campaign that brought them BACK to the website is important but the initial campaign they had clicked on is equally important. I’d love to know how much is actually stored in the cookies and how much can be pulled out. Thanks!
@Russell: Yes, you could include more than one piece of campaign information. However you would need to store that information in a cookie on the visitor’s machine and then send it along with the form submision. The code would need to do the following:
1. When someone visits the site copy the utmz information into a new cookie
2. On every subsequent visit append the utmz information to the existing cookie value
3. When a form is submitted send the information stored in your custom cookie to your CRM
Hope that helps.
Is it possible to parse the info stored in “Session / Page” level custom variables, just like the one you mentioned in custom segment cookie (_utmv) ? Say I want to know if the visitors viewed certain page/content before posting a form?
I’m looking for an update as well. This would be extremely helpful for me. I’ve tried to implement, but the best that I can do so far is for my form to return “undefined” for all variables. Do I have to set “UA-1-1” to be the tracking ID of my google analytics account?
@Dan – Update coming, soon I hope… As for the tracking ID, you should use the tracking ID for your web property.
@Vinny – No, you can not pull out visit or page level custom variables. Those values are stored on the GA servers, not in cookies.
How would someone go about extracting the user-device and OS and storing it in a form variable?
@chris: That Google Analytics data is not available in the browser environment. It’s only determined after processing the data.
Did anyone ever post a list of data that could be extracted or an updated version for the asynchronous tracking code?