May 28

Reducing churn with Econometrics

If you run a recurring revenue web app, churn is one of your biggest enemies. Right now our churn at Treehouse hovers around 9% and we’d like to get that to 5% or less.

We just built a tool that should decrease cancellations or pauses by up to 50% so I thought I’d share how we did it.*

The idea is pretty simple: Create a list of users who look like they’re about to cancel, based on the past behavior of users who cancelled. Then contact this list and ask how you can help them or give them ideas on how to better utilize the service. Ideally, this should discourage them from cancelling.

How we built our tool

We hired an very talented Statistician who focuses on Econometrics (I found him by tweeting). We asked him to build a tool that would output a list of email addresses and first names from people who looked like they were going to cancel or pause. I said it was probably related to these factors:

Number of videos watched in the last X days
Number of logins in the last X days
Total Badges unlocked
Number of Badges unlocked in the last X days
Number of days since signup
Days until their next billing cycle
Days until the 1st day of the next month
Plan ID (We have two plans: Gold $49/mo and Silver $25/mo)

Our freelance Statistician built the tool with Stata, R, SAS, JMP and SQL.

The input to R pulls from an exact copy of the SQL Treehouse database. All data organization, cleaning, modeling, output, etc. is done in R. The tool outputs a CSV of emails for contact, minus those that have been contacted in the past X days (based on some pre-existing output files).

The tool takes about 45 minutes to run on a quad-core i7 2.2 with 8GB of RAM.

There is a parameter that we can adjust with the model to increase/decrease sensitivity with respect to false positives. Right now, two tested examples are:

The model successfully identified 37% of cancellations within the next 7 days with an 8% false positive rate.
The model successfully identified 42% with a 12% false positive rate.

There is a trade off between length of time that any model will be able to predict ahead and accuracy. In a short enough time horizon nobody cancels, in a long enough horizon everyone cancels. We believe 7 days was a good medium, but this time horizon could be changed with probably <2 hours of coding.

The code can be run on a daily basis and will automatically improve with new data.

Now that we have the tool, we’re going to start contacting Treehouse Members who look like they’re about to cancel to see if we can help them, thus discouraging them to cancel. I’ll report back on the results.

* This figure is based on a discussion I had with a SaaS business that’s been operating for 11 years with over 1,000 employees. They use a similar tool and they’ve been able to reduce their churn by 50%.