Now that we have some holiday-specific benchmark data to work with here at Justuno, we decided that it was time to put some of the key indicators from that study to the test.
Given the broad differences between promotions with timers and promotions without timers in our larger industry benchmark study, I wanted to get some more specific data. I also wanted to apply what we learned from that large data set to make an informed decision about what to test and see what the results were.
The results were, to say the least, interesting:
We tested our own cart abandonment promotion here at Justuno to see what difference a timer would have on performance. For the sake of accuracy, it is important to keep in mind that here at Justuno we are a SAAS (software as a service) and as such will have different performance compared to other verticals (e-commerce, local services, etc).
Let’s take a look at the actual designs:
What we tested
With timer
No timer
The difference is not major, but according to the benchmark data just adding a timer to an open-type promotion can, on average, have much higher rates of engagement.
Let’s look at that data now as well as the data from this 3 week case study.
Benchmark data
No Timer (ONT):
Engagement Rate: 23.33%
Conversion Rate: 5.6%
With Timer (OT):
Engagement Rate: 49.79%
Conversion Rate: 3%
Our test results
No Timer (ONT):
Engagement Rate: 47.3%
Conversion Rate: 8.8%
With Timer (OT):
Engagement Rate: 50%
Conversion Rate: 7.14%
These numbers are incredibly interesting because of the rather large differences between our benchmark data and this specific case study. There are, I believe, some important factors that come into play that explain the differences.
Suspected weighing factors
- Placement: Our test focused specifically on cart abandonment using both the OT and ONT promotion types whereas our benchmark data ignored placement. The reasons for this were twofold:
- We wanted to give broad benchmarks for each promotion type to serve as a starting point for novices in the field to have some metrics to work from.
- Given the fairly limited window that our industry benchmark looked at we expected and controlled for outliers as much as possible. It was believed that by clustering each promotion type without regard to placement would create a smoothing effect on the data as a whole.
- Industry Type: Justuno is specifically a SAAS whereas our benchmark data was industry agnostic. Based on other industry papers it is obvious that different industries will have different baselines.
- Offer: our discount offer was generally higher than most discount offers and that may have had an impact on overall numbers.
It’s difficult to say with any certainty which of these three factors has the most significant weight to it, but I suspect that placement is the key determining factor here. Cart abandonment pop-ups tend to be very effective and our data in this test confirms that trend.
Engagement rate
Seeing the results of this test was interesting because it gave us some interesting insights into our benchmark data.
Our promotion with no timer had nearly double the engagement rate of our benchmark for the same type while our promotion with a timer had a nearly identical rate to the benchmark for the same type. This is the strongest indicator in our data that placement is the largest contributing factor at play.
Remember that any open promotion can be used for almost anything and in almost any place. What this tells us is that in our benchmark data open promotions with no timer are most likely being used for cart abandonment much less often as compared to open promotions with a timer are.
Conversion rate
This was perhaps the most interesting and certainly the most surprising revelation from this test. Not only did both of our tests significantly outperform the benchmark data, but when you control for fluctuations in impression counts in our variants the conversion rate is not what we expected.
Overall, the variant with no timer had a fairly substantial higher conversion rate when compared to the variant with a timer. This seems to fly in the face of the notion that adding a timer and therefore adding a real sense of urgency to promotion will outperform promotions without that added sense of urgency.
However, look back at our benchmark data and you’ll see that promotions with no timer also outperformed promotions with a timer in that much larger data set as well.
When we control for the differences in sample sizes (our benchmark data had millions of data points) our study still outperforms the benchmark data.
Study conclusions
- Open promotions without a timer are likely used less often for cart abandonment so any promotion that falls into this category used for cart abandonment should outperform our benchmark data.
- Timers do make an impact but are less critical than commonly thought.
- Cart abandonment promotions perform incredibly well. This is backed by both our benchmark data and now our own targeted case study. If you are not using cart abandonment for your site, you should be.
How to add a cart abandonment promotion
Implementing a cart abandonment promotion is quite simple on the Justuno platform. To get started, you’ll need to login to your Justuno account. If you don’t have a Justuno account, get a free account here! Follow the steps in the video below to add your own exit offer with a timer to your site.
