Lean Your Marketing: Control The Conditions With Split Tests


Once you’ve got a hypothesis in hand, it’s time to get testing.

Split testing, aka A/B testing, means testing two options against each other at the same time and against the same randomized population.  This is pretty easy to do in a lot of digital marketing:

  • The major paid search providers allow for different ads to be rotated evenly against your keywords
  • Many banner ad networks also provide for testing, although direct banner buys may not
  • If your email marketing provider doesn’t give an automatic way to test emails, you can pretty easily split your list yourself and send out two different messages
  • Services like Optimizely or Unbounce make landing page testing easy

Split testing is a great way for you to feel comfortable that the result you observe is a result of the thing you are testing and not some other random factor.  For example, if you ran one search ad for two weeks with a conversion rate of 3%, then another for two weeks with a conversion rate of 5%, could you really be sure that the second ad won because it was better?  Maybe it was because it was two weeks closer to Christmas.  Or maybe it was because you had a site outage the first two weeks. Or you had some good PR the second two weeks.

But if you test the two ads against each other in a split test, alternating them evenly to a randomized population, then all those other factors don’t matter, and you feel comfortable saying the second ad was just better.

Keep It Under Control

So setting up split tests helps you minimize extraneous factors; you can minimize them even further by paying careful attention to how you set up your tests. Make sure you limit the differences in your two marketing assets to just the thing you want to test.

Say you want to test whether having a call to action in your search ad improves response.  What would be wrong with these two ads as your test subjects?


The problem is, although one ad has a call to action (“Try It Now!”) and one ad doesn’t, there are also other differences between them.  The headlines are different, the key messages are different, one is longer than the other.  If what you really want to learn about is how effective a call to action is, you’re not going to get at it this way.  You might see a difference in performance, but you won’t be able to confidently attribute it to the factor you’re looking at.

And then you’ve gone and wasted time and effort by setting up a test that didn’t teach you anything.  Those happy scientists in the last slide?  That makes them cry.  EVERY TIME.

Don’t make the scientists cry.

Instead, make your ads largely the same except for what you want to test:


The first ad wins?  We say, “Yay! Calls to action work!  I will try this in other ads!”

This doesn’t mean, though, that all split tests must involve tiny little tweaks— it depends on what you’re testing.  If you’re designing banner ads and you really don’t have a sense for whether a playful overall effect works better than a serious one, you’ll be putting out two pretty different ads.  Once that larger question is answered you might move on to smaller changes of the concept, until eventually you’re changing background color or testing different pictures.  And then you might just want to test the whole concept again against another concept.

The key is to control non-test factors as much as possible, so that you feel confident that you are learning something real.

Next up: what to do when split tests aren’t possible.

(Photo credit: shar ka via Flickr)


Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20



Lean Your Marketing: Create A Hypothesis


Did you ever do science fair in school?  Creating that yearly experiment taught me the scientific method:

  1. Create a hypothesis
  2. Test it in a controlled way
  3. Gather results
  4. Draw conclusions that either confirm or invalidate the hypothesis

You’ll hear Lean Startup aficionados talk a lot about validated learning, and a key step in that is to first have something to validate– a hypothesis.  In the context of developing product, a hypothesis is basically that customers will like this service or that feature and will be willing to pay for it.  To test the hypothesis you develop and launch the feature and record whether it succeeds or not.

Test Why, Not Just Which

In the context of marketing, most people recognize that they should do some kind of testing of search and banner ads and the like.  However, often their method for doing this is to throw up two different search ads, promote the winner, and stop the loser– basically, they go right for steps 2 and 3 of the scientific method. This is fine, and better than doing nothing at all, but your testing program will be even better if you add steps 1 and 4 into the mix.  Then you’re not answering the question “Which ad is better,” but more importantly, “Why is it better?”  You’re testing motivations for behavior that can enhance all your marketing.

For example, say you want to test the background color of two different banner ads.  If you just randomly pick blue and red, and blue wins, you’ve really just learned that blue is better.  Instead, if you were testing whether warm or cool colors work best, and blue (as a representative of cool) wins, you can further test different cool colors against each other, or test another warm/cool pair to see if it was the shade or the intensity.

With search ads, you don’t always have ad groups big enough to support a robust testing program– if you have 12 tightly-organized ad groups, for example, only 2 or 3 of them might have enough traffic to return results quickly.  If you just throw up different ads in those groups, you learn what works for those groups.  But if you test a theory such as “A call to action will increase response,” then if you validate that hypothesis you can add calls to action to lower-volume groups.  You can also test more different things at once– you can test three different hypotheses in three different ad groups and learn in parallel.  You might find that a dynamic headline, a call to action, and at least one exclamation mark works better in your search ads and then combine those into one ad to see if that’s even better.

In short, creating a hypothesis before you test is a key part of learning, not just optimizing.

(Photo credit: peaceplusone via Flickr)


Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20


Lean Your Marketing: Everything’s Trackable


Some things lend themselves to easy tracking– paid search, for example, generates lots of data and, at least in the case of Google, runs it through a nice dashboard with lots of reports.  But what about other kinds of marketing?  Some may be harder to wrestle into shape, for sure.  But everything can at least be approached with a bit more discipline.

Tracking Digital Campaigns With Google Analytics

Google tracks a lot of stuff for you automatically– referral traffic vs. paid search vs. organic search, for example.  But you can take this even further and track a wide range of digital efforts by creating custom campaigns.

Now, when you say, “create a custom campaign” people tend to picture that there is somewhere in Google Analytics where they will click a “+” icon and enter in details.  But in reality, creating a custom campaign in Google is both easier and harder than that.  Basically, all you need to do is append every URL you want to track with some parameters, and it will be tracked for your automatically.

Um, what’s a parameter?

If you already know what a URL parameter is, you can skip to the next section (sort of like a Choose Your Own Adventure).  If you don’t, read on!

A parameter is a part of a URL that passes information to the browser or to code within the page for tracking purposes.parts-of-a-url

Parameters are separated from the first part of the URL by a question mark, and each parameter has both a name and value.  Different parameters are connected with an ampersand.  In this example, the page “example.html” would have some sort of code on it that would tell it to scan all incoming URLs for parameters called “source” and “type,” then store the value it finds– in this case, “google” for “source” and “banner” for “type.”

What Parameters Does Google Analytics Track?

Any page that you have coded with your GA code will track up to five parameters.  From GA’s custom campaign page:

“We recommend you always use utm_source,utm_medium, and utm_campaign for every link you own to keep track of your referral traffic.  utm_term and utm_content can be used for tracking additional information:

  • utm_source: Identify the advertiser, site, publication, etc. that is sending traffic to your property, e.g. google, citysearch, newsletter4, billboard.
  • utm_medium: The advertising or marketing medium, e.g.: cpc, banner, email newsletter.
  • utm_campaign: The individual campaign name, slogan, promo code, etc. for a product.
  • utm_term: Identify paid search keywords. If you’re manually tagging paid keyword campaigns, you should also use utm_term to specify the keyword.
  • utm_content: Used to differentiate similar content, or links within the same ad. For example, if you have two call-to-action links within the same email message, you can use utm_content and set different values for each so you can tell which version is more effective”

So to track up to five different things for any one link, just add the variable name and a value to the end of your URL, making sure to separate the parameters with a “?” before and a “&” between each one.  Here’s how that might look:


You can use these customized links on anything you might like to track– links in an outbound email, links to your site that you put on Twitter or Facebook, links on Slideshare presentations; anything that links back to your site can have custom campaign variables on it so that you can better see how people are getting to your site– and how your marketing is performing.

At one of the places I worked, our outbound email program didn’t track specifically what people clicked on. So we tagged each link in our email with a different custom link so we could see what content got a better response.  That told us there was one section of our newsletter that no one ever clicked on; we dropped that section and added more content people liked, driving up overall response.

Of course putting all those variables on a link can make it quite long; use a shortener like Bit.ly to make your links more Twitter-friendly.

How do you see results?

In GA’s standard reporting, look at the Traffic Sources report.  “All Traffic” gives you a report that defaults to a view of source/medium; you can also select source, medium, or “other,” which allows you to select campaign or content.Screen Shot 2013-01-08 at 11.10.29 AM

But of course, who likes to stick with standard reporting when custom reports are at your fingertips?  You can set up a custom report in Google Analytics to look at any numbers you want (visitors, conversions, costs, etc) and to drill down by the factors you’re tracking: source, medium, etc.  This gives you the power to analyze exactly the data you wish.  If you’ve never used custom reports, log into your Google Analytics account and download this sample report I’ve prepared: GA Custom Report Parameter Tester.

This has only one number, unique visitors, but five levels of drill down.  Once you’ve copied it to your own GA account you can change it and play around with it to see how it works.

Track Your Tracking

The reason that creating custom campaigns by just sticking something on the URL is easier than creating some sort of campaign in Google Analytics is that you can do anything you want, on the fly; but, on the other hand, unless you are pretty careful it can start to be hard to remember what custom variables you have out there, because Google doesn’t track what you WANT to track, only what it actually gets in.  Also, it can be a little tedious to build all these URLs by hand.

I usually handle this by creating a spreadsheet that tracks all the different custom URLs we are using; that way when we look at our Google report, if there is a source or a campaign we don’t know right away we can just check the spreadsheet as a reminder.  As an added benefit, the spreadsheet can be coded to create the links for you, which cuts down on user error, and you can also store the matching shortened link instead of creating it fresh each time.

I’ve created a sample tracking spreadsheet in Google Docs that you can copy and use: Google Analytics Custom Campaign Tracker

Okay, but what about stuff that you don’t control?  What about PR?

This is all well and good for links we are putting out into the universe on our own; but what about something like PR or blog outreach, where we just hope for a mention at all and can’t really ask them to use our custom-tagged URL?

I’ve worked at a couple of different places that were managing PR efforts, and let me tell you one metric that isn’t super useful: number of placements. This, to me, falls squarely in the realm of bullshit metrics, because it doesn’t help you learn anything about your marketing efforts.  Instead, you should see what placements are driving traffic.  Here custom reports will help you again.  When you know you’ve gotten a placement, check out your source/medium report to see what referring URLs from the placement looks like.  Then create a custom report just for tracking your PR by using another key custom report feature: filters.

You can set up custom reports to filter for a whole range of things, but in this case we’ll have it filter by just the sources of our PR traffic by using regular expressions, aka regex. (Don’t be scared, it’s easier than it sounds.)  Regular expressions mean that it won’t look for things that exactly match what we give it; it will look for things that are similar, too.  So if we tell it to look for traffic from wsj.com, it will also give us the traffic from m.wsj.com as well.  Set it up like this:Screen Shot 2013-01-08 at 11.48.43 AM

Now, here’s where it gets a little tricky.  Next time you get a placement, you’ll want to add it to the report so you’re looking at your overall PR efforts.  However, for some reason the filters in GA are set up to only be AND, so if you try to add another filter it will look for traffic from wsj.com AND forbes.com.  Then your report will return nothing, because that  doesn’t make sense. You want it to look for traffic from wsj.com OR forbes.com.

The answer is to use a regex character, the pipe, which means the same thing as OR.  In other words, set up your filter like this:

Screen Shot 2013-01-08 at 11.55.19 AM

Any time a new source comes in, add another pipe and throw it on the report.

(Here is a good explanation of regex expressions you can use with Google Analytics.)

Tracking PR like this may give you some surprises– you may be amazed at how much traffic you get from someplace you’ve never really heard of, while disappointed at how little you get from a Big Name Placement.  But surprises are the stuff of learning.  Form some hypotheses about why some placements are working better, then test it by going after more like that.  Soon your PR efforts will be humming.

Huh, long post. I could have done a whole presentation on just this slide, apparently!  Next up: Learning.

(Photo credit: Michael Kappel via Flickr)


Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20


Lean Your Marketing: Make Tracking Part Of The Product Plan


I’ve got to say, everything I’ve recommended on these slides so far has been something that, even if clients weren’t doing it right away, they got there eventually– deciding on a key metric, figuring CPA, etc.  This, though?  This one’s more like a cri de couer for a perfect that world that might exist if we all hold hands and dream together.  This is something that I’ve never really seen happen, but that I’d hope all new companies do, and then when they are wise, grey-whiskered 5-year-old companies we’ll all look back and say, “Can you believe that people didn’t used to make tracking part of their product plan right from the get-go? Wow, what a pain in the butt that was.”

But wait– let me back up and explain what I’m talking about.

Pretty much every place I’ve ever worked or consulted, tracking metrics has involved placing a snippet of code on every page you want tracked.  Google Analytics works that way, KISSMetrics works that way, Mixpanel works that way, etc.  Further, to do the sort of tracking we smart lean marketers are going to do, where we are tracking conversion events deep in the funnel and not just surface stuff like site traffic, you need to tell your metrics tool what counts as a conversion– often by citing a page that comes up after a success event occurs (such as a thank-you page after a signup or a purchase).  There are other ways to track successes, but that is by far the most common one– basically telling your tracking tool that “when you see Page X, count it as a conversion.”

At the last couple of places where I set up new metrics regimens, though, this wasn’t possible. The user flow wasn’t set up to deliver such a unique page; however, at both places, we didn’t even realize this until we were validating the numbers against what was in the database and just couldn’t figure out why they wouldn’t come out correctly.  In both cases we eventually realized that the page we had marked as coming up only for new users came up any time a user created a new page, account, whatever.  So we were tracking *a* metric, just not *the* metric we wanted.  This then resulted in having to create a klugey work-around, put a change in user flow into the next product plan, and put a little asterisk on all the reports to explain why the number we’d be using was wrong and shouldn’t really be compared to future numbers, and… ugh.

How much better would it have been if the person designing the user flow and the person in charge of tracking results had a quick chat before the product was even launched to coordinate on what was going to be tracked and how to track it?  And how lovely would it be if when site changes are planned, those two people meet again to make sure that tracking that’s in place will be maintained, that funnels won’t be wiped out, etc?

In an organization where numbers-driven decision-making is front and center, this is what will happen.  If we all just hold hands and dream hard enough.

Next up: how to track things that don’t really seem trackable.

(Photo credit: theiry49)


Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20


Lean Your Marketing: Have A Goal And A Target Cost Per Acquisition


Okay, so you’ve figured out your key performance indicator— you’re all set, right?  Almost.  But to really make sure you understand whether your marketing is performing the way you want it to, you need to create a goal for your KPI that is specific and time-bound.  Instead of saying that you just want “more leads,” for example, say that you want 100,000 leads in a year’s time. Sure, this seems pretty obvious, but you’d be surprised how many clients I talk to don’t have specific goals in mind.  You’ll learn pretty quickly whether your target goal is realistic, but if you don’t at least put something out there then you don’t have a real yardstick to measure results against.


Basically everything I’m recommending here comes from asking clients what they’re trying to achieve and having them answer,”????” Not only do they often not know what metrics are most important to them, they don’t have a goal in mind and they definitely don’t have a target Cost Per Acquisition, or CPA.  But this is a very important boundary condition to know, because it does a lot to dictate what marketing methods you’re willing to try and helps you figure out which ones are more effective.  If you are willing to pay $100 per lead, there are a lot more programs you can try than if you only want to pay $10 per lead.

Calculating Your Target Cost Per Acquisition

There are a couple of ways to go about this.  In one, you take the pot of money you have available for marketing, divide it by how many (we’ll just call them leads, but substitute your own success event) you want, and there’s your target cost per lead:

Marketing Budget $100,000
Lead Goal 25,000
Cost Per Lead $4

The problem with this method is that it doesn’t tell you if you’re losing money on every acquisition.  If you’re spending $100,000 to get $10,000 your business isn’t going to be one for the ages.  So the better method is to figure out how much value you get from each new lead:

Leads 10,000
% Convert to Customer 1,000
Conv % 10%
Lifetime Rev / Customer $150
Lifetime Profit  / Customer $50
Target Cost / Lead
Lifetime Profit * Conv % $5

Once you’ve used a method like this to figure out how much you should be paying to get people to your site, it will be easier to evaluate each new method to see whether it is working.

Next up: making tracking a part of your product plan.


Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20


Lean Your Marketing: Find The Metric That Pays Your Bills


The customer funnel. We know what this is, right? It’s the progression of people through stages of your marketing, from (for example) searching on an ad, clicking on the ad, visiting your site, and signing up for or buying your wonderful product. All of those stages produce opportunities for tracking.  Any one-horse ad product is going to give you impressions, clicks, that kind of thing.  And the most basic GA implementation will give you site visits.  And because those numbers are easy to get and also large and easily juice-able… well, you’d be surprised at how many people consider those to be very important metrics.

Well, no less an authority than Mark Andreessen has a message for you: your metrics are bullshit.

Eric Ries, author of Lean Startup, is more genteel, so he calls them vanity metrics.  If you pay attention only to things at the top of the funnel– ad clicks, site traffic, page views– you can feel like you are making progress when in truth your growth is stalled or worse.


Instead, smart organizations look deep into the funnel to find the one KPI, the key performance indicator, that really shows the health of their business.  What is it? Well, it’s going to be different for different organizations.  The way I like to approach it with different clients is to ask them: what pays your bills?  If there is revenue coming into the organization, what is its main source?  Is it subscriptions? Product sales? Donations?  Then that’s the KPI.  Even if the company is pre-revenue you are likely to still have a business plan in place for how money will come in eventually; is it by acquiring active users? Is it by showing ads to a big user base?

The exciting thing about identifying this core metric is that it gives everyone in the organization the same focus.  Once you’ve agreed what the single most important number is for your organization, everyone should be adjusting their performance to making that KPI grow.  You won’t have your marketing people off looking at ad clicks and while your product people are looking at user engagement.  If your organization has agreed that Daily Active Users is your KPI, then that’s what everyone should be trying to produce.  Your product people will be tuning the site to increase this number, and your marketing people will judge their projects solely on the number of daily active users they generate.  Your customer service people might have targets for how quickly they handle tickets and amount of positive feedback from customers, but the overall goal of their group will be helping to maintain daily active users.  Everyone in the organization will be rowing the same direction.

Is it easy to track deep into the funnel?  Generally no, and that’s why so many companies don’t bother.  But is it worth it?  Absolutely.  Having a real understanding of what’s going on in your organization will lead to smarter, more confident decisions.

Once you’ve identified your KPI you’ll want to turn it into a goal; that’s the topic of my next couple of slides.


Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20


Lean Your Marketing: Start Early And Stay Focused


Aaaah, the Measure portion of the presentation. *cracks knuckles, gets ready*

So, it was kind of funny trying to figure out how to organize this presentation, because Measure kept wanting to come first, despite being clearly second in the Build-Measure-Learn cycle.  And the reason it kept wanting to come first was because of this recommendation: start early.  Many’s the company that literally tackles things in that order: build it first, then figure out how to measure it.  But you should think about measuring and metrics very early in the game because, people: Google Analytics is a great tool, especially considering its price, but it isn’t magic.  Not a few people have asked, once I got some tracking set up on GA, if it could look at past results.  Yes, let me just drape a little blue cape on it and make it fly around the world until time flows backwards.  You want a baseline of your early performance?  You’ve got to set up performance tracking early.  It’s pretty simple.

As for what you track, it can be surprisingly hard for companies to be disciplined when it comes to looking at numbers.  When I’m talking to potential clients I’ll ask them if they’re interested in more traffic, more leads, more sales, etc, and they’ll generally answer yes to everything.  But it’s not that helpful to try to optimize 15 different stats at once– instead you should focus on the numbers that really matter, which is the subject of the next couple of slides.


Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20