We just built a tool that should decrease cancellations or pauses by up to 50% so I thought I’d share how we did it.*
The idea is pretty simple: Create a list of users who look like they’re about to cancel, based on the past behavior of users who cancelled. Then contact this list and ask how you can help them or give them ideas on how to better utilize the service. Ideally, this should discourage them from cancelling.
How we built our tool
We hired an very talented Statistician who focuses on Econometrics (I found him by tweeting). We asked him to build a tool that would output a list of email addresses and first names from people who looked like they were going to cancel or pause. I said it was probably related to these factors:
- Number of videos watched in the last X days
- Number of logins in the last X days
- Total Badges unlocked
- Number of Badges unlocked in the last X days
- Number of days since signup
- Days until their next billing cycle
- Days until the 1st day of the next month
- Plan ID (We have two plans: Gold $49/mo and Silver $25/mo)
The input to R pulls from an exact copy of the SQL Treehouse database. All data organization, cleaning, modeling, output, etc. is done in R. The tool outputs a CSV of emails for contact, minus those that have been contacted in the past X days (based on some pre-existing output files).
The tool takes about 45 minutes to run on a quad-core i7 2.2 with 8GB of RAM.
There is a parameter that we can adjust with the model to increase/decrease sensitivity with respect to false positives. Right now, two tested examples are:
- The model successfully identified 37% of cancellations within the next 7 days with an 8% false positive rate.
- The model successfully identified 42% with a 12% false positive rate.
There is a trade off between length of time that any model will be able to predict ahead and accuracy. In a short enough time horizon nobody cancels, in a long enough horizon everyone cancels. We believe 7 days was a good medium, but this time horizon could be changed with probably <2 hours of coding.
The code can be run on a daily basis and will automatically improve with new data.
Now that we have the tool, we’re going to start contacting Treehouse Members who look like they’re about to cancel to see if we can help them, thus discouraging them to cancel. I’ll report back on the results.
* This figure is based on a discussion I had with a SaaS business that’s been operating for 11 years with over 1,000 employees. They use a similar tool and they’ve been able to reduce their churn by 50%.
Recent commentsBlog comments powered by Disqus
- startupsbunch likes this
- illbelieveinanything reblogged this from ryanleecarson and added:
- netonomics likes this
- growhackweb reblogged this from ryanleecarson
- jackholmes likes this
- fredstevenssmith likes this
- mytangent likes this
- lex1 likes this
- joshlong likes this
- shalinguyen reblogged this from ryanleecarson
- themattyg likes this
- kirklove likes this
- swimmingundersoundwaves likes this
- zackgilbert likes this
- ryanleecarson posted this
Stuff I Like
Jay-Z connecting the dots between Russell Simmons and Mark Zuckerberg
“People in the record business had always made a lot of money. Not the artists,...
We work a 4-day week and just raised $4.75m
I think there’s something messed up about the startup culture in the USA. The belief...
@ryancarson on TwitterFollow @ryancarson