The Jones Road Test: How to Measure and Prove Incrementality in SMS

We know SMS makes you money—but how much more money exactly?

Merchants have wanted an answer to this question for a long time. When Cody Plofker, Chief Marketing and Revenue Officer of Jones Road, wanted to make sure his monthly spend was providing incremental revenue, we decided to run a test to gather some real ROI.

We turned to our Data Science team and Joanne Coffey, Jones Road’s Retention Marketing Manager, and constructed an easy experiment that we could conduct as a starting point. 

Here’s what happened:

The entire Jones Road retention team, along with their Postscript Customer Success Manager and our data team, landed on an experiment they’d run for one month.

The subscriber list was grouped into 3 different segments:

• 50% of SMS subscribers would keep getting their normal cadence

• 25% of SMS subscribers would not receive any messages

• 25% of SMS subscribers would get 2x more—receiving 2 texts per week instead of their usual single weekly send

Subscribers were randomly assigned to the separate campaign treatment groups and we measured attributed revenue among each of cohort. The one exception to this was new subscribers, who were all kept within the standard send group.

Each subscriber was assigned a subscriber property within Postscript to make sure they were in one of the groups.

"Helping set up the experiment was surprisingly straightforward. With the support of Postscript's data science team and our CSM, we were able to get it off the ground quickly. It's a testament to how responsive our marketing strategies can become when we're focused on real, actionable data,” said Coffey.

Throughout June, all campaigns from Jones Road were filtered according to the assigned treatment group (normal, holdout, and 2x). Once the month was complete, the data team ran an analytics query to look at the number of total purchases and purchase revenue per subscriber in each of those 3 groups.

The test also looked at the added cost that Jones Road spent on marketing per subscriber in each of those 3 treatment groups to make sure the revenue was truly incremental. 

“We set a clear plan from the beginning and had the right tools in place. With our Postscript integration, assigning subscribers to different groups was seamless, allowing us to focus on the strategy rather than get bogged down by logistics," said Coffey.

Without burying the lede—more sending produced a higher ROI and 98% certainty that the group getting twice as many sends was yielding Jones Road a significant lift in revenue. 

Here’s a closer look at how these results were measured:

In the chart above, the red line represents the revenue per subscriber during the month. This is for the group that got the 2x campaign treatment.

The green line is the standard campaign frequency, and then the blue line is the holdout group.

Based on this experiment, the group that received 0 campaigns during the month spent the least! And the group that received 2X amount of texts did purchase more—a true look at incremental revenue earned per subscriber.

It’s important to note that when we are looking at incremental sends, that also means incremental revenue—the R in ROI.

When Jones Road began sending 2X per week—double their normal amount of sends—they generated 33 cents per subscriber. 

Of course, with these additional sends, we also considered the additional cost of sending when we used a formula for measuring true incrementality. 

To look at incremental ROI, we divide the extra amount earned against the extra amount spent, which is an incremental ROI of 11% for the standard versus holdout group.

"Discovering that increasing our SMS frequency could bring in more incremental revenue without leading to diminishing returns was a game-changer for us. It's one thing to theorize about the potential of ramping up our marketing efforts, but seeing the tangible results has been incredibly affirming,” said Coffey.

The test also saw a statistically significant lift in revenue per subscriber of roughly 10% for the group that received 2x the number of messages compared to the standard group.

"The results spoke volumes,” said Coffey, “Knowing that we can actually send more to earn more, without adversely affecting our customer engagement, was an eye-opener."

At Postscript, we’re focused on making our merchants more money and coming up with tailored experiments for brands to help prove the incrementality of their SMS programs. Get in touch with your Customer Success Manager or schedule some time to chat with us to see how our in-house data science team can help.