Properly Analyze A/B Test results with Advanced Segments

After my last post on GTMtesting.com, I received a couple of questions, mainly dealing with: How can I find out how each variation is performing?

For optimizers and analysts, this is something we do on a daily (or at least weekly) basis.

For incidental A/B testers, this can be quite a task because it requires a lot of clicks and you don’t want to make a mistake.

In this post, I will explain how to report results on an A/B test using Google Analytics. You can use the logic in this post and apply it to any advanced analytics tool.

 Measure which variation is seen

Firstly, you need to track which variation is seen by the visitor.

My favorite way is to fire an event. This is also the way GTMtesting does it. When the variation is loaded correctly, a Google Analytics event is fired, with in it the test name and variation name in it.

You can do that by publishing the event in your variation code.

There are alternative ways, but events are the most flexible.

Alternative 1: use a custom dimension

• pro: if you use a user-based custom dim, you can easily use this to drill-down in reports (set it as primary or secondary dimension)
• con: you have to be very careful not to use the same custom dimension twice for the same user, or else you will “miss” users because the dimension gets overwritten

Alternative 2: use page names (e.g. for split-url tests)

• pro: it might be slightly more accurate, since it doesn’t require an extra event call
• con: use with extreme caution. You want to be absolutely sure that you redirect ALL visitors.
  E.g. When your control-page is /default.html, redirect 50% to /varationA.html (which is a copy of /default.html) and 50% to /variationB.html (which is your actual variation). If you don’t understand why to do this, ask me why in the comments :-)
• con: it will mess up some other metrics, like pageviews-per-visit, bounce rate, time on site. Not a showstopper, but worth mentioning.

So, you now have an event with the test+variation name in it. Excellent, let’s get to work.

Use advanced segments – the proper way

For most tests, you will evaluate your test based on unique users (or: visitors) and only 1 binary outcome (conversion y/n). So, you want the following conditions to be met:

• count each user just once
• each user should be in only 1 condition (variation A or B)
• discard users who neither A nor B
• discard users who are in both A and B (check how many there are. If you see lots of users, start questioning your experiment design)
• count each conversion just once per user (you don’t want 1 call-center agent to skew your test results with loads of conversions, I’ve seen it happen)

These rules look simple (and they are), but it’s easy to make mistakes in segmenting correctly.

The correct way is to use sequences in your segments.

For each variation, create 2 segments:

1. A conversion segment
   1. For Variation A: exclude users with the event ‘test 001 – variation B‘ (and similarly for other variations)
   2. include this sequence:
      1. users with the event-action ‘test 001 – variation A‘
      2. followed by: users that visited page /thankyou.html (or whatever your conversion is)
2. A non-conversion segment
   1. For Variation A: exclude users with the event-action ‘test 001 – variation B‘
   2. Exclude users that visited page /thankyou.html (your conversion)
   3. Include users with the event-action ‘test 001 – variation A’

Now you have segments that are mutually exclusive: 2 for each variation.

Now, you can set your time period, and just get the total number of visitors for each segment, and do the math.

• Users for variation A = ( Users of Conversion Segment A + Users of Non-Conversion Segment A)
• Conversions for variation A = Users of Conversion Segment A

And the same for variation B, of course.

Now put your newfound knowledge into Analytics action..

[bctt tweet=”TIL: you can track A/B test visitors in Google Analytics with advanced segments (with sequences)” username=”zjuul”]

Questions, additions or comments? Send me a tweet, or leave a reply, thanks!
Photo Credit: https://flic.kr/p/6ATEuo

Related posts:

Review and demo of GTMtesting.com Multi Channel Funnel Conversion Rate Conversion Attribution: a Markov model (using Google Analytics and the R ChannelAttribution package) GTM Monitoring in Google Analytics