How to Time Reactivation Campaigns for Maximum Impact — Jackson Shuttleworth, Duolingo

David Barnard:

Welcome to the Sub Club Podcast, a show dedicated to the best practices for building and growing app businesses. We sit down with the entrepreneurs, investors and builders behind the most successful apps in the world to learn from their successes and failures. Sub Club is brought to you by Revenue Cat. Thousands of the world's best apps trust Revenue Cat to power in-app purchases, manage customers and grow revenue across iOS, Android and the web. You can learn more at revenuecat.com. Let's get into the show.

Hello. I'm your host, David Barnard. Today's conversation is shorter than usual and will be featured in Revenue Cat's State of Subscription apps report. Each episode in this series will explore one crucial metric and share actionable insights from top subscription app operators. With me today, Jackson Shuttleworth, group product manager, retention team lead at Duolingo. On the podcast, I talk with Jackson about Duolingo's data-driven approach to retention, how social connections dramatically improve reactivation and why letting users set the preferred notification time backfires. Hey, Jackson, thanks so much for joining me on the podcast today.

Jackson Shuttleworth:

Yeah, thank you for having me, dude.

David Barnard:

So, I'm really looking forward to talking about reactivation and keeping users in the app and getting it back when they've left. But, for context, Duolingo is such a huge company. I did want to give listeners just a little insight into what you work on day to day.

Jackson Shuttleworth:

So, Duolingo, I manage our retention efforts. This sits within the growth org at Duolingo. So, this is focusing on how do we grow our user base, specifically focusing on DAUs. So, within retention, a lot of this focusing on the most habit-forming features in our apps. So, you've probably heard of the Duolingo streak. This is what I spend a lot of time every day thinking about how to better optimize with some of the other teams under my umbrella, focusing on things like notifications, push notifications, emails, things that are both helping keep the user in the app, and that's what we focus most on is how do we get you to never churn. But, then once we do churn, or once the user does churn, how do we get them back into the app as quickly as possible, because you learn that the longer the user has been away, the harder it's to get them to come home.

David Barnard:

So, in the report, we're not going to share usage metrics. We are going to be sharing subscriber metrics and subscriber reactivation metrics. But, what's going to be fun chatting with you is you're not as focused on the subscriber reactivation, but you're focused on the user reactivation, and, ultimately, that's upstream of the subscriber reactivation. If you ever have a hope to win somebody back to your subscription, you first need to win them back to the app. So, let's chat about that. What's your philosophy around how to get people back into the app?

Jackson Shuttleworth:

The easiest way to retain a user is to not let them churn to begin with. So, a lot of the features that you'll see on Duolingo are really focused on building that daily habit. We've run so many experiments on our streak. Even things that have users spending one, two days away from the app make it significantly more difficult to get them to come back in two or three days later. So, a lot of our focus is getting users, if they can use the app each day, to use it every day. I will say that, again, this team does manage things like notifications, emails. This is a lot of our power users have taken a few days off, getting them back in the app, and we send a series of seven notifications for each day that you don't use the app.

So, you take a day off, we send you a notification. Second day off, we send you a notification. At seven days, we turn off our practice reminder notifications, which are our most powerful notification because the additional value we get of continuing to reach out to you after seven days is significantly lower. Honestly, even by day seven, the efficacy has dropped significantly. So, that is an insight into you can't wait until day seven to get that user to come back into the app. You probably can't wait until day five. If they're not coming back by day three or four, again depending upon the use case, for us, that's for a daily use app, that's definitely the case. Man, it is just going to be tough sledding to get them to come back.

David Barnard:

For a lot of apps that aren't as focused on daily use, which I think is a good portion of the folks who will be listening, is a golfing app where you're going to go golfing two or three times a month or even a fitness app, which maybe you're doing really well to be working out three times a week. For the folks who have past that seven-day mark who are more challenging, I feel like maybe that's more relatable to a lot of the folks who are going to be listening to this. How do you think about those people who have been out of the app for seven days who that harder problem?

Jackson Shuttleworth:

We think about it across a few different timescales. I'll say the first one is the first few weeks after we've lost you. So, we have a progress report type email that goes out to users. The fact that it's an email is nice because it gives us significantly more space for us to highlight elements of your progress report versus just to push a notification where we're really having to rely heavily on you tapping into this to go back to an app that maybe you've uninstalled. So, here actually channel is important.

So, this is nice because it is a weekly progress comparison where, week to week we say, "Hey, how did you do this week versus how you did last week?" If you've taken a week or two off, obviously, that's framed somewhat negatively, but what you could do is ... and there's all sorts of different copy that we've tested. There's ways to draw attention to like, "Hey, have you taken a break? Come back and jump in." Or, "Hey, man, it seems like a slow week. How much are you going to learn this week?"

There's different ways to make that copy and messaging to users either a little bit passive-aggressive or hopeful or a challenge and this is really important. What I'd say is you got to test all of these different copy variants. One of the things at Duolingo, we test so much copy, like a disgusting amount of copy, particularly with a lot of our notification type work. Different users are different, and you can have as many hypotheses as you want about what works. You just got to test it. A lot of it is just engagement metrics that we're showing you. How much XP did you earn? How much time did you spend in there? Some of that's because it is more complex for us to show you more like, "Hey, you learned this word or the feasibility of that is higher."

As we think about ROI, what we've realized is that just showing a user how much they've done and allowing them to infer and using copy to support it, like, "Hey, look at all the stuff you've done." It's much easier and honestly gets a lot of the bang for the buck. For Duolingo, where you're obviously using Duolingo to learn a language, make better use of your free time, it's easier for us to tell this narrative of you use Duolingo, but isn't that obviously a good thing, this time you spent? But, I think there's a lot that you can do as for me, the product manager, to just draw attention to here's the activity in the app. It is necessarily a good thing. Don't you want to come back and get more of that?

You can frame data lots of different ways to make it seem motivating, and I would challenge your listeners to really look out because, again, these users, sure, if they're not compelled by your message, they're not going to come back.

David Barnard:

So, then moving on to the logger time span. I guess it was the seven- to 30-day and then after 30 day is, I guess, the next bucket?

Jackson Shuttleworth:

Yeah, yeah, yeah. So, a few different things. I'll say one thing that's hot on our mind is we do a big year in review-type, similar to a Spotify or Wrapped where, at the end of the year, we're showing you stats and information about your learning, and it's super fun. I recommend anybody who's been on Duolingo this year, go check it out. Even if you were a user who used it for a few days in April, we can still show you something and give you value and give you value pretty much immediately after reopening the app that is awesome and exciting and makes you laugh and makes you think like, "Oh actually hey, I could use that app again."

So, I think the more you can frame some of these outreach moments, even months later, in a way that when the user comes back, they're getting something delightful, they're getting a little endorphin rush. It is something fun and particularly if you can tie it to them. Again, the user that's used the app for a few days in April isn't going to have a lot of really high-value analysis we can give them, but we could still make it a super delightful experience, and that might be what it takes to get them over [inaudible 00:08:30] and using the app again. So, I think the more you can tie it to ... tie these outreach moments to either specific times of year, specific information, what can you deliver to the user in terms of value and not just like, here's a product update, much better performance, though, when we can actually attach it to something relatively personal.

David Barnard:

Speaking of performance, are there any campaigns that stick out in your mind that's especially effectively? Leading up to the summer after COVID, we said, "Hey, are you traveling this summer?" Was there any, just knocked it out of the park kind of thing that you recall?

Jackson Shuttleworth:

Specific campaigns, less so than there are specific times of year that are always really, really important to us. It's exactly as you said. None of this should probably be all too surprising because we try to build our campaigns around the mental model that users have around using our product. So, back to school, always very big. People are going back to school. You want to start learning again. Even people who are not in school, you get into that mindset of going to class. So, that's a really big time for us, and we make sure to target that with messaging that resonates.

Same thing beginning of the year, new year's resolutions. Duolingo's often learning language, learning anything, definitely a new year's resolution that folks have. It's funny because that actually works quite well with our year in review-type program where we're getting users back at the end of the year. Hopefully, some of those users are sticking around at the beginning of the year where then they're making that commitment to, "Hey, I'm going to learn a language. I'm back on Duolingo. I'm just going to keep it up." But, again, for us it's less the actual ... to me anyway, less the actual creative and more how are we targeting the right times of the year with the right message.

David Barnard:

How do you think about data and experimentation in relation to all of this? I know Duolingo famously does a ton of testing. So, when you're thinking about these reactivating users and winning users back, how do you think about it in terms of data and making it more data informed?

Jackson Shuttleworth:

For Duolingo, notifications, again, be it email or push, are a critical part of our feature set and a critical driver of growth for us. So, we have machine learning-driven systems that choose the right notification copy for the right user. They optimize on their own. For a PM, it's really nice. You just stick copy variants in there. With AI, we probably won't even have to do that soon enough. Yeah, we'll take and then it optimizes, which is great. Once you have the size and scale to do that, building a system like that is awesome. Never sleep on copy. With notifications, it is infinitely more important to constantly be testing because all it can rely on is that subject line and maybe some preview copy.

Your push notifications, all it can rely on is what's in the title and what's in the body. You don't have the whole feature set that it's supporting it or the fact that the user's in the app already. It's just all copy, it's all testing. So, again, depends on the size of your population, but you can look at open or click through open rates. Open rates or click through rates, again, be it email or a notification. You can get, I would guess, static data pretty quickly. But again, I just see a lot of companies, I don't know, looking at notifications. It's clear that there's some that are just like, "All right, cool, this notification worked. Let's move on." Our practice reminder notification, we've been experimenting on that notification for six years. For a PM that loves experimentation, it's beautiful to see.

So, really honing that message, because again, I said this earlier, these users have churned. You don't know what they about their experience. There's a lot that you don't know about this user. This is the user you need to experiment on almost more than any because it's just less likely that you have a good handle on what it is that they need. All of our scheduling is smart. Although one of the funny things about our scheduling that we realized is that I think there's this desire from companies when you onboard and you say, "Hey, when do you want us to remind you? In the morning, in the evening, at this time?" What we realized is that as soon as you set a time, the likelihood that you were going to be free at that time for perpetuity, which is what a lot of apps assume, is just zero. We all have busy lives. We all use these apps at different times of the day. What was always a winner for us was scheduling the next day's notification for 23.5 hours after you practiced for the first time that day.

So, we basically just were always targeting, when did you practice the day before the next day? So, there is a lot of ... and this doesn't seem smart, although we've tested a lot of different strategies here, and this is the one that continues to win.

David Barnard:

Any other top tips for getting people back?

Jackson Shuttleworth:

One that's been an interesting thread for us to pull on is social in the app as it pertains to win backs. Duolingo has been investing in our in-app social network now for many years. I think we're at 55 or 60% of our users have a friend on the app, and these aren't all ... my mom for instance, uses Duolingo. We're friends in the app.

David Barnard:

Oh, that's a great win back, man. Mama in the app. Like, "Hey Jackson."

Jackson Shuttleworth:

Yeah, well, whenever my mom lives in the street, it cuts particularly deep. So, one of the things that we've, again, threads we've been pulling on is getting users to resurrect each other. So, if you have a friend who's been gone for months and months, prompting the user to, "Hey," get them to come back. This is one of our best-performing resurrection campaigns, or win-back campaigns, just because it feels more personal. It's not Duolingo saying, "Hey, David, come back." It's Jackson saying, "Hey, come back and learn with me."

Then, what's cool is now we have other features in the app. So, one feature we recently launched is friend streaks. It's a way for you to learn with somebody else. Now, we have a social feature set that when you can get this person back into the app, now that there's been a personal appeal made, you can ease them into a more social experience with that person. So, it's a nice way to pair the out-of-app communication with the in-app experience, but even for folks that it's like, again, the random person that you find on your leaderboard still a much higher click-through rate for ... there's people that you've never met in real life. Even messages, win-back messages. From those folks still perform better.

David Barnard:

As we're wrapping up, anything else you wanted to share?

Jackson Shuttleworth:

Yes. If any of this sounds interesting for you to work on, we are hiring, actually. We're hiring both for my team, looking for a senior to lead PM who wants to work in Pittsburgh, helping drive a lot of our retention efforts. So, again, if this streak excites you, do that professionally. Then, the other area is we're starting to build a notifications team. So, looking for a senior to lead PM in New York who wants us to help build out an even smarter, more robust and thoughtful notifications platform. So, this was a very relevant discussion for us today, and if you find it relevant to you, to what you like to think about, definitely reach out.

David Barnard:

Awesome. Well, thanks so much for joining me today. It is so insightful. A lot of really fun stuff. Thank you.

Jackson Shuttleworth:

Yeah, of course.

David Barnard:

Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community.