Five Split Tests We Did at Time Doctor and What We Learned

Time Doctor split test

We’re constantly running multiple split tests on the Time Doctor website, with the intent of increasing conversion rates. We always have at least one split test of the home page running (usually more than one), and also split test other pages of the signup process, the pricing page, etc.

Here are the results of 5 split tests that we’ve run in that past:

1 – Home Page Style – Normal VS Minimalist

In this test we decided to try running a minimalist version of our home page. Here are the winning & losing versions of the page:

(Click each image to see the larger version)

Time Doctor split testing version 1

Version 1 – Winner

Time Doctor split testing version 2

Version 2 – Loser

Notice that Version 2 has a white background, minimal text (including the reduced menu at the bottom), and blue lines around the form and testimonial.

For this split test we had more than two versions of the page running, so that we could track the effect of each the various changes that we made to the page. The two versions you see above are the worst and best converting versions. Version 1 had a 59% conversion rate increase over Version 2, at a 99% confidence level.

2 – Pricing Page Headline

In this test we split tested the headline on our pricing page. We tested 2 different versions.

(Click each image to see the larger version)

Pricing Page Headline 1

Version 1 – Loser
Conversion rate: 10.69%

Pricing Page Headline 2

Version 2 – Winner
Conversion rate: 14.12%

Version 2 resulted in a 32% increase in conversion rate over the original (Version 1). This test reached 98.9% confidence. A conversion was counted when a visitor to the pricing page submitted the first form in our signup process.

We have to thank 37 Signals (creators of Basecamp) for the inspiration to run this split test. We decided to run it after reading their blog post about a similar split test that they’d run.

3 – Explaining Features on Home Page

Here we decided to try adding the content of our “Features” page to the bottom of our home page, thinking that people would be more likely to sign up if they were more easily able to see the features of the software. The two pages looked like this:

(Click each image to see the larger version)

Features on homepage version 2

Version 1 – Winner

Features on homepage version 2

Version 2 – Loser

Version 1 beat Version 2 by 38% at a 95% confidence level. To provide the full context of this split test, I should explain that there were two issues with Version 2 (maybe if we fix these issues and re-run the split test again, we’ll have a different result):

  • Since we simply copied & pasted the content of our “Features” page onto the home page, the page didn’t “flow” as well as it could have. I think if we run a similar test again we could have a better transition into the features section, and maybe also simplify the features explanation so that it is more suited to a home page.
  • The features explanation wasn’t written as well as it is now (we’ve re-written our Features page since running this split test). Maybe the fact that the previous text was mediocre contributed to its poor performance in this split test.

4 – Home Page Form Button

This is one of our older split tests, but it had a significant result that we thought was worth sharing. For this test we changed both the text and functionality of the button on the form on the Time Doctor home page. Here are the two versions:

(Click each image to see the larger version)

Home Page Form Button 1

Version 1 – Winner

homepage form 2

Version 2 – Loser

Version 1 was the original page in this test. Our theory was that with the “Download Time Doctor” button on Version 2, we were offering visitors instant gratification (i.e. “You can download Time Doctor instantly, rather than simply “getting started”) which we hoped would convert better.

In reality, Version 2 turned out to be one of our most spectacular failures. The original beat Version 2 by 283% with 100% confidence.

In hindsight, we can only speculate why Version 2 performed so poorly. I think the reason is probably that people are wary of installing new software on their computer. With Version 1, visitors who are signing up take a few smaller/easier steps before we ask them to take the bigger step of downloading some software.

5 – Pricing Page Plans

We’ve always had a non-prominent “Free” and “Solo” (cheap) plan listed on our pricing page beneath our main plan. For this test, we tried removing them. Here are the two pages:

(Click each image to see the larger version)

Pricing Page Plans 1

Version 1 – Winner

Pricing Page Plans 2

Version 2 – Loser

Version 1 beat Version 2 by 43% with a confidence level of 99.7%. It turns out that a lot of our visitors sign up initially on the free or solo plans, even if they switch to a more expensive plan later on, and a lot of those visitors won’t sign up at all if all they see is the expensive plan up front.

Wrapping Up

If you visit the Time Doctor site today, you’ll notice that some of the pages look significantly different even than the winners of the split tests above. That’s either because:

  • We’ve since tested new versions that convert even better, OR
  • You’re seeing a new version of a page that we’re currently split testing and are still waiting for statistically significant results.

Always be testing!

Listen how Time Doctor got 283% more trial users from these split tests in this episode of ConversionCast.


Try Time Doctor

Justin Clark

Justin Clark

Justin Clark takes care of paid advertising & conversion rate optimization at Time Doctor - a time monitoring and productivity monitoring tool that is designed to help businesses reduce wasted hours at work and improve employee productivity. He is from Ottawa, Canada.


Leave a Reply

Your email address will not be published. Required fields are marked *