Growth Hacking: The Results Are In. Kind Of.

Image

Wah, what happened?  How is it July, and not even early July but mid July?  How have I not posted in so long?

Well, here’s the truth: I’ve been posting every week, but the damn zombie post has been eating them.  Because they were smart posts, and thus full of brains.

(You: “Doesn’t that mean this post is dumb?” Me: “Probably. Let’s move on.”)

Anyways, people mostly ignore everything I write, but I have gotten questions about the growth hacking experiment.  “How did it turn out?” they want to know.  “Have you disappeared off the face of the earth because you’ve been so very busy hacking away?”

Kind of?

Unfortunately for a data-driven marketer, I can only bring anecdata to the table, and the reason why illustrates the difficulties of testing things without true split conditions.  I quieted down on the blog for a couple of weeks, unlinked the blog from the LinkedIn profile, and generally tried to control things so that changes in views and incoming requests could be attributed to the title change and nothing else.

But at the same time, I had been working on a piece on the KISSMetrics blog about landing page optimization, which was a great opportunity, and one that I didn’t have that much control over. They ran the piece when they ran the piece, and I couldn’t really say, “Hey, could you hold off for six weeks?  I’m doing an experiment.”  So the piece ran right around the same time I Growth Hacked my job title.

So– yes, I did get a lot of traffic to my profile, and I did get a lot of inbound requests from people wanting hacking.  But did it come from the title change?  Did it come from the KISSMetrics piece?  Did it come from putting my Lean slides on SlideShare, which also produced traffic?

It’s hard to say.  My gut– which even a data-driven marketer has– says that yes, the title change did produce some traffic and some inbound opportunities.  I admit, this is not a resounding answer.

Now if you’ll excuse me, I’m off to carefully cull Andrew Chen’s posts and Twitter so I can get ahead of the curve on the next buzzword opportunity.

Growth Hacking: Does It Bring All The Boys To The Yard?

I'm really not sure how this particular shot of Willie Wonka became a meme. He actually looks pretty friendly here.The first time I encountered the term “Growth Hacking” was on the LinkedIn profile of my Simplee colleague, Aaron Ginn.  “Ha-ha, that is some buzzword BS,” I thought dismissively, because buzzwords give me hives.  But soon enough, Aaron was writing a series on growth hacking in Tech Crunch.  Apparently it is a thing now. (And apparently using cliches doesn’t give me hives.)

“Get with the program,” Aaron told me. “Growth hacking is the new black.”

So what is growth hacking?  Allow entrepreneur and blogger Andrew Chen to explain:

Growth hackers are a hybrid of marketer and coder, one who looks at the traditional question of “How do I get customers for my product?” and answers with A/B tests, landing pages, viral factor, email deliverability, and Open Graph. On top of this, they layer the discipline of direct marketing, with its emphasis on quantitative measurement, scenario modeling via spreadsheets, and a lot of database queries.

Huh, I thought when I read that.  That just sounds like the sort of smart, scrappy marketing every startup should be doing.  But whatevs, I guess if we need a new title to take its place alongside “Social Media Guru” (15,157 results on LinkedIn) and “Viral Ninja” (312 results, a growth opportunity! Although sounding somewhat like an aggressive case of Japanese encephalitis), then “Growth Hacker” works fine.

Cut to today; I was chatting with another marketing nerd and he mentioned that after he finally broke down and put the term “Growth Hacker” on his LinkedIn profile, the opportunities came pouring in like gravy at a Southern buffet.

“Seriously,” he said. “Give in. Change your title.  Belly up to the trough.”

Okay, I thought, I’ll do it.  BUT I’LL ONLY DO IT FOR SCIENCE.  I realized this is an exciting chance to– say it with me– test!  I’m going to change my title and see how it affects how many people look at my profile.

This is a good example of the kind of experiment that can’t really be tackled with a split test.  That means I have to try to control what other factors I can.  And pretty much the only factor I can control is that for the last month I’ve been running all my blog posts through LinkedIn, as well as participating in some groups there.  So I disconnected my LinkedIn from the blog and– I’m sorry to do this to you, Lean Startup Group– I’ll also refrain from commenting in groups.  I’ll let the control conditions stand for a couple of weeks, and then I’ll change the title.

And then I’ll sit back with a napkin around my neck and a piece of white bread in each hand, waiting to sop up that sweet, sweet gravy.

It’s exciting, I know.  Try sleep well at night, regardless.  I will keep all The World apprised of results.

Significance

two coconuts

This is what came up when I searched for “compare.” This and lots of pictures of a meerkat which is apparently named Compare.

So you’re running a marketing campaign, because you are awesome and know that testing is your path to improved performance and general hosannahs.  You’ve got a couple of different banner ads with conversion rates of 4% and 5%.  (Hahaha, I know no banner ad ever in the history of the Internet has had that kind of conversion rate, but stay with me for a minute.)  Time to declare Mr. 5% the winner and move on, right?

Not necessarily.

You’ve probably noticed that your conversion rates don’t stay very stable– one week they’re down, one week they’re up, even if you haven’t done a thing to your campaign.  So how do you know that the one that looks to be the winner today won’t take a nosedive tomorrow?

By testing the significance of the difference between the two ads.

Results are considered statistically significant if it is unlikely that they occurred by chance.  Statistical significance is also sometimes referred to as a confidence level— the higher the significance, the more confident you are that differences between two results aren’t due to random chance.  Often a confidence level of 95% is considered the threshold to declaring a winner, but you may choose to do less if you’re trying to move through testing options quickly.

If you’re hungry for the stats– and who isn’t?– you can take a look here to see the specifics of how you can compare two different proportions (which is what a conversion rate is) to see if the difference between them is significant enough.

If you’d just like to skip to the part where you check your results, there are a couple of online tools you can use (here or here).

But if you’re like me you’ll quickly find that using the online tool gets tedious.  That’s why I created a spreadsheet that lets you input the impressions and conversions of the winning and losing ads and from that calculates the degree of confidence in the result.  You can find it here:

Split Testing Results Calculator

Having it in spreadsheet form makes it easier to use it for your own glorious purposes– for example, I created a different version that lets me paste in downloaded AdWords results and mark the winner and loser, and it automatically picks out the right stats and throws up my results.  Magic.

Quick note on the inputs

I mostly use this for PPC ad tests, although you can use it for banners, emails, and any old thing with a response rate.  You need two stats:

  • Population stat: this is going to be something like impressions, opened emails, etc. Basically, it’s how many people saw your thing.
  • Success stat: this is the thing you wanted to happen. God willing you’ll make it a conversion event and not clicks.

For PPC ads some people just use conversion rate, meaning conversions over clicks.  However, there could easily be a situation where an ad converted better but had a lower click through rate so that you end up getting proportionally fewer of the people who originally saw the ad.  Therefore I prefer to take it all the way from impressions.  Which of course means always having to calculate test results by hand, because Google doesn’t even offer conversions over impressions as a stat.

Now go be all scientific.

Photo by thienzieyung via Flickr.

Split Testing In A World Of Tiny Traffic

As you know, I think split tests rock and you should definitely do them.  However, over at TechCrunch Robert J. Moore brings up a great point about A/B testing:

…What if, like most start-ups, you only have a few thousand visitors per month? In these cases, testing small changes can invoke a form of analysis paralysis that prevents you from acting quickly.

Consider a site that has 10,000 visitors per month and has a 5 percent conversion rate. The table below shows how long it will take to run a “conclusive” test (95 percent confidence) based on how much the change impacts conversion rate.

A/B Testing Populations

If you’re a startup with low traffic, is his point, you don’t have as much opportunity to cycle through tests as might a site with more visitor flow, so you want to make sure the tests you do run will have a big impact.  Change only something small about the home page, and you may find yourself needing to let the test run for weeks before you reach significance.  Some implications of this:

  • Take traffic into account when designing a test plan.  If you’re doing a banner ad campaign with several different segments, it probably only makes sense to run tests in the segments big enough to get reasonable traffic.  If you’ll only get a few conversions from a segment, you likely won’t have enough volume to generate significant results.  In the principle of minimizing the amount of resources expended on projects, only spend time preparing and tracking tests if you are likely to see results.
  • Start big, go small.  If you’re in early days of testing start at the concept level– different tones, different layouts, different messages.  Test things that are likely to have a bigger impact, and refine from there.  That being said…
  • Small changes can have big impacts.  Once I tested search ads that were completely identical except that one had a period after the text and one didn’t.  The period generated a surprisingly big lift. I just heard from a friend that changing one word in the call to action on their home page lifted response by 20%.  On the other hand I’ve run tests where two ads were completely different and didn’t really get a significantly different result.

If you’ve got tiny traffic you will have fewer test cycles available to you.  You don’t always know ahead of time what’s going to move the needle, so check your results regularly and end tests that don’t seem to be going anywhere.

Lean Your Marketing: Everything’s Trackable


beth-morgan-lean-your-marketing-everythings-trackable

Some things lend themselves to easy tracking– paid search, for example, generates lots of data and, at least in the case of Google, runs it through a nice dashboard with lots of reports.  But what about other kinds of marketing?  Some may be harder to wrestle into shape, for sure.  But everything can at least be approached with a bit more discipline.

Tracking Digital Campaigns With Google Analytics

Google tracks a lot of stuff for you automatically– referral traffic vs. paid search vs. organic search, for example.  But you can take this even further and track a wide range of digital efforts by creating custom campaigns.

Now, when you say, “create a custom campaign” people tend to picture that there is somewhere in Google Analytics where they will click a “+” icon and enter in details.  But in reality, creating a custom campaign in Google is both easier and harder than that.  Basically, all you need to do is append every URL you want to track with some parameters, and it will be tracked for your automatically.

Um, what’s a parameter?

If you already know what a URL parameter is, you can skip to the next section (sort of like a Choose Your Own Adventure).  If you don’t, read on!

A parameter is a part of a URL that passes information to the browser or to code within the page for tracking purposes.parts-of-a-url

Parameters are separated from the first part of the URL by a question mark, and each parameter has both a name and value.  Different parameters are connected with an ampersand.  In this example, the page “example.html” would have some sort of code on it that would tell it to scan all incoming URLs for parameters called “source” and “type,” then store the value it finds– in this case, “google” for “source” and “banner” for “type.”

What Parameters Does Google Analytics Track?

Any page that you have coded with your GA code will track up to five parameters.  From GA’s custom campaign page:

“We recommend you always use utm_source,utm_medium, and utm_campaign for every link you own to keep track of your referral traffic.  utm_term and utm_content can be used for tracking additional information:

  • utm_source: Identify the advertiser, site, publication, etc. that is sending traffic to your property, e.g. google, citysearch, newsletter4, billboard.
  • utm_medium: The advertising or marketing medium, e.g.: cpc, banner, email newsletter.
  • utm_campaign: The individual campaign name, slogan, promo code, etc. for a product.
  • utm_term: Identify paid search keywords. If you’re manually tagging paid keyword campaigns, you should also use utm_term to specify the keyword.
  • utm_content: Used to differentiate similar content, or links within the same ad. For example, if you have two call-to-action links within the same email message, you can use utm_content and set different values for each so you can tell which version is more effective”

So to track up to five different things for any one link, just add the variable name and a value to the end of your URL, making sure to separate the parameters with a “?” before and a “&” between each one.  Here’s how that might look:

http://www.marketingnerdistry.com/?utm_source=linkedin&utm_medium=outreach&utm_content=slides&utm_campaign=midwinter

You can use these customized links on anything you might like to track– links in an outbound email, links to your site that you put on Twitter or Facebook, links on Slideshare presentations; anything that links back to your site can have custom campaign variables on it so that you can better see how people are getting to your site– and how your marketing is performing.

At one of the places I worked, our outbound email program didn’t track specifically what people clicked on. So we tagged each link in our email with a different custom link so we could see what content got a better response.  That told us there was one section of our newsletter that no one ever clicked on; we dropped that section and added more content people liked, driving up overall response.

Of course putting all those variables on a link can make it quite long; use a shortener like Bit.ly to make your links more Twitter-friendly.

How do you see results?

In GA’s standard reporting, look at the Traffic Sources report.  “All Traffic” gives you a report that defaults to a view of source/medium; you can also select source, medium, or “other,” which allows you to select campaign or content.Screen Shot 2013-01-08 at 11.10.29 AM

But of course, who likes to stick with standard reporting when custom reports are at your fingertips?  You can set up a custom report in Google Analytics to look at any numbers you want (visitors, conversions, costs, etc) and to drill down by the factors you’re tracking: source, medium, etc.  This gives you the power to analyze exactly the data you wish.  If you’ve never used custom reports, log into your Google Analytics account and download this sample report I’ve prepared: GA Custom Report Parameter Tester.

This has only one number, unique visitors, but five levels of drill down.  Once you’ve copied it to your own GA account you can change it and play around with it to see how it works.

Track Your Tracking

The reason that creating custom campaigns by just sticking something on the URL is easier than creating some sort of campaign in Google Analytics is that you can do anything you want, on the fly; but, on the other hand, unless you are pretty careful it can start to be hard to remember what custom variables you have out there, because Google doesn’t track what you WANT to track, only what it actually gets in.  Also, it can be a little tedious to build all these URLs by hand.

I usually handle this by creating a spreadsheet that tracks all the different custom URLs we are using; that way when we look at our Google report, if there is a source or a campaign we don’t know right away we can just check the spreadsheet as a reminder.  As an added benefit, the spreadsheet can be coded to create the links for you, which cuts down on user error, and you can also store the matching shortened link instead of creating it fresh each time.

I’ve created a sample tracking spreadsheet in Google Docs that you can copy and use: Google Analytics Custom Campaign Tracker

Okay, but what about stuff that you don’t control?  What about PR?

This is all well and good for links we are putting out into the universe on our own; but what about something like PR or blog outreach, where we just hope for a mention at all and can’t really ask them to use our custom-tagged URL?

I’ve worked at a couple of different places that were managing PR efforts, and let me tell you one metric that isn’t super useful: number of placements. This, to me, falls squarely in the realm of bullshit metrics, because it doesn’t help you learn anything about your marketing efforts.  Instead, you should see what placements are driving traffic.  Here custom reports will help you again.  When you know you’ve gotten a placement, check out your source/medium report to see what referring URLs from the placement looks like.  Then create a custom report just for tracking your PR by using another key custom report feature: filters.

You can set up custom reports to filter for a whole range of things, but in this case we’ll have it filter by just the sources of our PR traffic by using regular expressions, aka regex. (Don’t be scared, it’s easier than it sounds.)  Regular expressions mean that it won’t look for things that exactly match what we give it; it will look for things that are similar, too.  So if we tell it to look for traffic from wsj.com, it will also give us the traffic from m.wsj.com as well.  Set it up like this:Screen Shot 2013-01-08 at 11.48.43 AM

Now, here’s where it gets a little tricky.  Next time you get a placement, you’ll want to add it to the report so you’re looking at your overall PR efforts.  However, for some reason the filters in GA are set up to only be AND, so if you try to add another filter it will look for traffic from wsj.com AND forbes.com.  Then your report will return nothing, because that  doesn’t make sense. You want it to look for traffic from wsj.com OR forbes.com.

The answer is to use a regex character, the pipe, which means the same thing as OR.  In other words, set up your filter like this:

Screen Shot 2013-01-08 at 11.55.19 AM

Any time a new source comes in, add another pipe and throw it on the report.

(Here is a good explanation of regex expressions you can use with Google Analytics.)

Tracking PR like this may give you some surprises– you may be amazed at how much traffic you get from someplace you’ve never really heard of, while disappointed at how little you get from a Big Name Placement.  But surprises are the stuff of learning.  Form some hypotheses about why some placements are working better, then test it by going after more like that.  Soon your PR efforts will be humming.

Huh, long post. I could have done a whole presentation on just this slide, apparently!  Next up: Learning.

(Photo credit: Michael Kappel via Flickr)

Related:

Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20

 

Lean Your Marketing: Make Tracking Part Of The Product Plan


beth-morgan-lean-your-marketing-make-tracking-part-of-product-plan

I’ve got to say, everything I’ve recommended on these slides so far has been something that, even if clients weren’t doing it right away, they got there eventually– deciding on a key metric, figuring CPA, etc.  This, though?  This one’s more like a cri de couer for a perfect that world that might exist if we all hold hands and dream together.  This is something that I’ve never really seen happen, but that I’d hope all new companies do, and then when they are wise, grey-whiskered 5-year-old companies we’ll all look back and say, “Can you believe that people didn’t used to make tracking part of their product plan right from the get-go? Wow, what a pain in the butt that was.”

But wait– let me back up and explain what I’m talking about.

Pretty much every place I’ve ever worked or consulted, tracking metrics has involved placing a snippet of code on every page you want tracked.  Google Analytics works that way, KISSMetrics works that way, Mixpanel works that way, etc.  Further, to do the sort of tracking we smart lean marketers are going to do, where we are tracking conversion events deep in the funnel and not just surface stuff like site traffic, you need to tell your metrics tool what counts as a conversion– often by citing a page that comes up after a success event occurs (such as a thank-you page after a signup or a purchase).  There are other ways to track successes, but that is by far the most common one– basically telling your tracking tool that “when you see Page X, count it as a conversion.”

At the last couple of places where I set up new metrics regimens, though, this wasn’t possible. The user flow wasn’t set up to deliver such a unique page; however, at both places, we didn’t even realize this until we were validating the numbers against what was in the database and just couldn’t figure out why they wouldn’t come out correctly.  In both cases we eventually realized that the page we had marked as coming up only for new users came up any time a user created a new page, account, whatever.  So we were tracking *a* metric, just not *the* metric we wanted.  This then resulted in having to create a klugey work-around, put a change in user flow into the next product plan, and put a little asterisk on all the reports to explain why the number we’d be using was wrong and shouldn’t really be compared to future numbers, and… ugh.

How much better would it have been if the person designing the user flow and the person in charge of tracking results had a quick chat before the product was even launched to coordinate on what was going to be tracked and how to track it?  And how lovely would it be if when site changes are planned, those two people meet again to make sure that tracking that’s in place will be maintained, that funnels won’t be wiped out, etc?

In an organization where numbers-driven decision-making is front and center, this is what will happen.  If we all just hold hands and dream hard enough.

Next up: how to track things that don’t really seem trackable.

(Photo credit: theiry49)

Related:

Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20

 

Lean Your Marketing: Have A Goal And A Target Cost Per Acquisition


beth-morgan-lean-your-marketing-specific-and-time-bound

Okay, so you’ve figured out your key performance indicator— you’re all set, right?  Almost.  But to really make sure you understand whether your marketing is performing the way you want it to, you need to create a goal for your KPI that is specific and time-bound.  Instead of saying that you just want “more leads,” for example, say that you want 100,000 leads in a year’s time. Sure, this seems pretty obvious, but you’d be surprised how many clients I talk to don’t have specific goals in mind.  You’ll learn pretty quickly whether your target goal is realistic, but if you don’t at least put something out there then you don’t have a real yardstick to measure results against.

beth-morgan-lean-your-marketing-know-your-cpa

Basically everything I’m recommending here comes from asking clients what they’re trying to achieve and having them answer,”????” Not only do they often not know what metrics are most important to them, they don’t have a goal in mind and they definitely don’t have a target Cost Per Acquisition, or CPA.  But this is a very important boundary condition to know, because it does a lot to dictate what marketing methods you’re willing to try and helps you figure out which ones are more effective.  If you are willing to pay $100 per lead, there are a lot more programs you can try than if you only want to pay $10 per lead.

Calculating Your Target Cost Per Acquisition

There are a couple of ways to go about this.  In one, you take the pot of money you have available for marketing, divide it by how many (we’ll just call them leads, but substitute your own success event) you want, and there’s your target cost per lead:

Marketing Budget $100,000
Lead Goal 25,000
Cost Per Lead $4

The problem with this method is that it doesn’t tell you if you’re losing money on every acquisition.  If you’re spending $100,000 to get $10,000 your business isn’t going to be one for the ages.  So the better method is to figure out how much value you get from each new lead:

Leads 10,000
% Convert to Customer 1,000
Conv % 10%
Lifetime Rev / Customer $150
Lifetime Profit  / Customer $50
Target Cost / Lead
Lifetime Profit * Conv % $5

Once you’ve used a method like this to figure out how much you should be paying to get people to your site, it will be easier to evaluate each new method to see whether it is working.

Next up: making tracking a part of your product plan.

Related:

Lean Your Marketing: The Slide Deck

Slide 1 | Slide 2 | Slide 3 | Slides 4, 5, 6 | Slide 7 | Slide 8 | Slides 9 & 10 | Slides 11 & 12 | Slide 13 | Slide 14 | Slide 15 | Slide 16 | Slide 17 | Slides 18, 19 & 20