Lessons from 121 A/B Tests - Kenneth Schlenker, Opal

David Barnard:

Welcome to the Sub Club Podcast, a show dedicated to the best practices for building and growing app businesses. We sit down with the entrepreneurs, investors, and builders behind the most successful apps in the world to learn from their successes and failures.

Sub Club is brought to you by RevenueCat. Thousands of the world's best apps trust RevenueCat to power in-app purchases, manage customers, and grow revenue across iOS, Android, and the web. You can learn more at revenuecat.com.

Let's get into the show.

Hello, I'm your host, David Barnard, and with me today RevenueCat CEO, Jacob Eiting. Our guest today is Kenneth Schlenker, founder and CEO at Opal, a screen time app with a mission to empower humans to focus better every day.

On the podcast, we talk with Kenneth about scaling to five million in ARR on paid ads, positive and negative results from 121 A/B tests, and why they still haven't built an Android app.

Hey, Kenneth, thanks so much for joining us on the podcast today.

Kenneth Schlenker:

Thanks so much for having me, David. Very excited to be here.

David Barnard:

And, Jacob, nice to have you joining us today as well.

Jacob Eiting:

I'm excited to talk about screen time and limiting it with Opal today. So let's go.

David Barnard:

Kenneth, I wanted to kick things off talking about Opal. I'd love for you to share just what the app is and why you created it.

Kenneth Schlenker:

Sure.

Yeah, Opal is a screen time app available on the App Store. We today are the leading application for screen time management. And yeah, it's an app that basically saves you an average of an hour and 20 minutes per day and less screen time and helps you be more balanced in how you use your phone.

David Barnard:

Awesome.

So I know you've been thinking about building Opal in this three-phase approach, and I think that this approach is something a lot of apps should consider. It's that when you first launch there's certain things you should be thinking about and then as you move along you change focus and grow.

So let's talk through those different phases of how you think about Opal and how to grow it as a subscription app business. What's the first phase and how you kicked off launching and finding product market fit with Opal?

Kenneth Schlenker:

We've been at this for three years now and started off with just thinking that there's an opportunity to build better screen time software than what's available by default on every single iPhone and actually Android today. But we did this in phases that you said.

The first phase was really about, okay, let's prove the opportunity. Not only that people are willing to use a better screen time software, but also that they're willing to pay for it. And we've been in that phase for over two and a half years, so it's a long journey.

In this first phase, the main metric of success for us has been D8 return ad spend. So essentially, what we try to figure out is not only are people willing to use an app, but are they willing to pay for it, and then are we able to acquire customers in a way that makes sense economically with just immediate cash payback.

David Barnard:

And you shared with me before that you all hit five million in ARR, which is an incredible milestone for a subscription app. And I would've thought three years ago in your founding like, "Oh, we're going to build a screen time app. It's built into iOS. Do people really need it? Are people really going to pay for it?"

Jacob Eiting:

I'm sure you never heard that Kenneth in your [inaudible 00:03:33]-

Kenneth Schlenker:

Never. It's first time.

David Barnard:

In that first phase, how did you think about proving out whether or not people actually care enough to use and pay for and can you build an app that is a better version of what comes for free in the system?

Kenneth Schlenker:

I think something amazing about subscriptions, and obviously this is music to your ears at RevenueCat, but something amazing in this early stage of the business is that essentially if you get people to pay for your products as a consumer, that's a pretty strong signal that you have something that's pretty valuable.

And so there used to be a trend of consumer apps that were like, "Okay, let's build a use case. Let's get people to use this and then we'll figure out how to monetize later."

But actually we took it the other way around, which is let's build something that's good enough for people to pay for it. First, let's figure out who these people are. Let's figure out how we can actually acquire people like these people specifically. And that's the original product market fit.

You mentioned we recently passed five million in ARR. We basically did this in a year, starting from almost zero a year ago to five million just recently.

And this has been done with basically focusing relentlessly on how can we build something that people are willing to pay for and then figuring out who those people are and figuring out how to acquire more of these people.

And specifically for us, what's interesting is we actually initially launched a business thinking, "Well, this app will be used mostly by, I would say Gen Z, younger, maybe students," but it turned out that the people willing to pay for the app are more knowledge workers, 25 to 40 who care about this issue so much they're willing to pay for it, and they also see direct ROI of the time they're saving, how they can actually be more productive at work, spend more time outside of work with their family, for example.

And so it's been a really interesting way to essentially use subscription as Discovery Engine as a product market fit engine, just focusing on this one metric, which is D8 return ad spend.

Jacob Eiting:

I'm curious. I don't stare at everybody's stuff, but once in a while I'll see somebody pop to the top of the leaderboard. And last year I've seen Opal be like, "Oh, you guys figured something out."

In those early days as you were iterating on ROAS, were you dumping money in from the very beginning and how much? How did that practically work? Because you need some amount of critical mass to make those measurements. And did you do that literally on day one? Was there no period of no monetization, just experimentation? Or how did that play out?

Kenneth Schlenker:

There was an MVP phase, which was just getting the product to work, which mostly test flight only a few hundred people. But immediately once we were on the App Store, we immediately started testing paid marketing, and so basically with a paywall, and I think we hooked up RevenueCat from the very first week that we were-

Jacob Eiting:

I think that's when we met, yeah.

Kenneth Schlenker:

Yeah. On the App Store. And yeah, it gave us a lot of signal really fast, gave us volume. I think in the early days what you want is you want a hundred downloads consistently per day, more than a drip of a thousand downloads on a waitlist. So having a drip of a hundred people per day paid marketing can actually deliver that quite well. And then you're able to push a new release every single week.

One thing we've done also from the very first week that we were on the App Store until today is we've released one app version every single week on Monday. And that's really a discipline in the team that creates a rhythm where you can actually experiment things and-

Jacob Eiting:

By Thursday, you're like, "Let's get something in because I want to see the number change the next week." Right? Yeah.

Kenneth Schlenker:

Right. And then sometimes, it's a very small thing. Sometimes, it's much bigger, but that allows us to organize our work around this release cycle. And it aligns really well with...

We look at day eight ROAS because we have a one-week free trial, and the big advantage of that is you get very short iteration cycles. So if you make something new on Monday, eight days later you get a sense of did you get an increase or a decrease in ROAS and you can keep iterating with very short cycles. So yeah, I think it's a really good way to discover the true value of what you're building.

I also think the landscape's changed so much in last few years. What was true of three or four years ago is not the case today. And today it's much harder to break through organically as an app maker and many people do it. I don't know necessarily how to do it. So for me it was, okay, let's figure out something that people willing to pay for that we can actually acquire customers for.

And now we're thinking... We're going the opposite way of thinking, "Okay, how do we actually from here now build a lot more organic traction?"

David Barnard:

Did you have specific milestones in those early days like, "We're going to launch and we're okay for now as a learning at 50% day eight ROAS or 25%?"

And was that the case? When you first started spending on ads, was it like 20% ROAS at day eight, so you knew you were going to spend money to learn and then it just inched toward that 100% or how did that go early on?

Kenneth Schlenker:

We spent a very small amount of money in the early days as a test, but very soon we reached 100% ROAS.

Part of the reason for that is we priced quite high from the beginning. It's a hundred dollars subscription on annual basis. When we started off, actually, we're at 69 and we ended up rising the price. But throughout this past year, we went from zero to five million in ARR. We've been operating at over a 100% day eight ROAS.

So basically, we don't have a marketing budget. We're only reinvesting in eight-day cycle.

Jacob Eiting:

I think the point you make about using eight day how much money you realize revenue in eight days over ad spend is really smart.

The subscription model has enough inputs and variables that it can tend to get confusing even I think for the smartest people to totally reason about. And so I think it's really helpful to cut and slice something that you believe is going to be indicative and optimize. You were saying better for that iteration cycle than total correctness, right?

You're going to be wrong sometimes. You're probably going to overspend, probably some of those cohorts weren't fully positive based on measurements and whatever, but as long as you have some number that you believe in, the grand arc is going to relate to your goal, reducing that measurement time is the most important.

And then you couple that with a rapid iteration cycle, that's rocket fuel, right? You just keep hitting it and hitting it and hitting it and hitting it until you're where you want to be.

Kenneth Schlenker:

But it's interesting. Definitely, it's super confusing. The main value for me is having one metric that you look at is like...

Jacob Eiting:

It's hard enough for the CEO and then you got to communicate it to a bunch of people that aren't as familiar with the model and all that stuff. But honestly, it's enough even just for you, right?

Kenneth Schlenker:

Yeah, yeah.

Jacob Eiting:

The product's complicated enough, don't make the business model super complicated too.

Kenneth Schlenker:

Yeah. That was a good learning of just pick one metric that you can iterate against. It just simplified so much.

I wouldn't say... We didn't get there right away. By the way, what I'm talking about right now is this clear one phase where we only did eight ROAS, it didn't actually happen like that from the beginning. It evolved into that and decided to drop everything else and just focus on this single metric. But once we did, that's really when we were able to grow.

Jacob Eiting:

Yeah. I think folks think they have to build a whole company first. You don't. You just got to get one little engine of something going. And forget about all the other things. Forget about tackling too much.

And the reason that that's okay is you will need to do all those other things, but it's much easier at five million in ARR than it is at zero. You just have a little more resources, you have a little more attention, you have a little more experience.

So did you just stick with the tiny little drips of ad spend and then one day you're like, "Okay, let's go," and then just turn the money on? How did you decide? And then what was that like to trigger the inflection?

Kenneth Schlenker:

We tested until we got close to... We didn't get to 100 right away when we start scaling, but we felt like we had a path to get to a 100% and that's when we started to scale ad spend.

And then basically, we pilot ad spend based on that. Every week, we look at what was the ROAS the previous week, are we able to expand budgets or have to lower them. If we're below 100%, we just basically cut back to get back to 100%. And yeah, week over week basically driving that evolution. As long as we're above 100%, we scale. So that's how we started spending a lot more and then we're able to grow.

There's no grove goal that comes from just, "I want to get to five million ARRs." It just happens as a consequence of setting the bar in terms of ROAS.

David Barnard:

Tactically, how were you measuring that? So were you just doing blended return on ad spend from a weekly cohort? So you would just say everybody who opened the app during this week counts towards our spend that week? What's the actual calculation of day eight ROAS?

Kenneth Schlenker:

Yeah. So now we look at cohorted based on install, so someone who installs the app within that week, and then we look at the revenue that generate eight days from install. We also look at day 30.

When we started out, it was much simpler than that. We just looked at how much are we spending this week and how much are we making a week later with an eight-day delay. That worked at the beginning. And then you end up at a point where you have a lot of late conversion. So people download the app in January and then end up paying in May, and then so the numbers don't line up, and so you need to have a coordinated view. But we started off in the most basic way possible.

The other thing we do is we do attribution in a very simple way, which is we ask people in the onboarding where they came from and we found that to be actually the most reliable way of doing it. So we also have all the fancy attribution software and at the end of the day, just asking people where have you heard about us. And if they say Facebook, it's one thing and they say word of mouth, something else, and then basically you segment the cohorts based on that answer. And then we look at unblended ROAS based on those cohorts.

Jacob Eiting:

Another good example of keeping it simple. You could tie yourself into knot being like, "Well, not everybody that installed that week probably saw an ad that week. They probably saw it the week before or whatever." And then suddenly, you're tied up all in your cord and you're falling over or you just go, "Yeah, I don't know. We'll just say everybody who installed this week saw an ad this week." Right?

Kenneth Schlenker:

Yeah. Yeah.

Jacob Eiting:

It's like is it correct? No. But is it useful? Probably. It lets you make decisions faster.

Kenneth Schlenker:

It's funny.

I should probably talk to you guys more about this as you probably have a ton of experience working with other apps, but attribution is one of the things I talk to other app founders most about like how do you guys do this, what are the best practices. Everybody has a different view. There's not like one single fully approved method for doing this, so everybody's doing their own.

Jacob Eiting:

Yeah.

Kenneth Schlenker:

But yeah, I find it's just asking people where they come from is just the dumbest, but also the probably most reliable way of doing it.

Jacob Eiting:

Even if you had perfect instrumentation on every single campaign and every single creative and all that stuff, you probably as a team, your size don't have the time to action on all of that data anyway, right? So you have to think in broad terms how much are we spending on this channel, on this broad creative strategy.

I guess falling off of that, I sent you an email the other week because I got an ad on Instagram for Opal that was so good, which... Maybe talk about that. You talk about spend and campaigns and stuff, but there's the whole building the creative and all of that stuff. How are you splitting your time between app development, the performance, actual marketing, and putting those things out there, and then the creative development part?

Kenneth Schlenker:

Thank you for that feedback. I loved receiving your email.

Yeah. It's basically for people who haven't seen it, maybe you have, but we did release an ad actually a few weeks ago that took over our entire campaigns and basically this ad, it's much more of a organic type video that talks about our mission in general more than the... It doesn't even show the product.

Jacob Eiting:

Yeah. So good though. I was amped up. It was like 1984, the famous Apple ad. It was great. We'll link it in the show notes.

Kenneth Schlenker:

There's a fun story also about this.

What happened is someone made this video before we ever reached out to them, and it wasn't about Opal, it was just about the space, but we like to talk to creators who are sensitive to the mission that we have, and it was very aligned with our mission.

So we reached out to them and they wrote back and said, "Actually, I use Opal and I love it." We're like, "Well, can we use this video as an ad?" And they said, "Yeah, sure. You should just take it." And we made a deal with them.

We tweaked it slightly, but really it was made as organic content and it worked really, really well. And actually now we have more. It was the second time we're doing this, it wasn't the first time, but now we're doing this a lot more where we're finding that basically whenever we talk about the mission, it's working quite well.

So it's probably 80% of the work is creative, 20% is campaign management, performance. So definitely creative is a big part.

We've tried everything. We have tried several agencies where... Including fully outsourced creative agencies that basically make ads, we'll run it for you and you pay a percentage of spend. We've tried working just with creative agencies that just send us concepts and we're more and more doing things fully in-house with... Literally, right now it's team of one and then a little bit of contracting.

What we're finding works right now is finding creators in our space that care about what we're doing. Ideally, they're using the app and they love it and then producing stuff with them.

But in terms of spending my time, I actually have someone great in the team right now that's really specifically creative partnerships manager who's working just on the creative side, and we're looking to bring in someone else with more the typical use acquisition. I would say more technical side of campaign management and data.

We're a team of seven. We have one person that just does creatives basically.

Jacob Eiting:

Speaking from CEO position, the folks that come in and they're most excited about RevenueCat and our mission, they bring a certain something magic that is hard to find-

Kenneth Schlenker:

Totally.

Jacob Eiting:

... so you can bring that in in your creative work.

And that's one of the reasons, with the exception of a few fringe activities. I don't love working with contractors, especially for core things. And for an app business, the marketing is very core. It maybe is marketing, it's maybe not part of the product technically, but as a founder and a builder, it is, right? This is-

Kenneth Schlenker:

Yeah.

Jacob Eiting:

... probably half of what you're focused on all the time is the acquisition aspect.

Kenneth Schlenker:

Definitely. I think it's probably half products, half marketing would be a good split of my time.

And it's interesting because, yeah, marketing informs product, product informs marketing. It really works together. We have to understand why people are willing to pay inside the app, what features...

For example, one of the things that people love is the ability to essentially lock yourself out of distracting apps without any way of canceling or bypassing, the normal-

Jacob Eiting:

Oh. I know. I dismiss screen time every day. I'm like, "Goodbye, computer. I have power over you. Ignore all day."

Kenneth Schlenker:

So we for you, Jacob, have built this special feature, it's called deep focus, and basically, there's no way to cancel or bypass. That's just simple value proposition. This is one of the reasons why people like this app and like to pay for it. And so of course, you want to build creative around that and that tends to work well.

Just understanding why are people using this for work during the day, are they using it in the evening to disconnect and unplug with family or with friends. What's the use case that you can highlight in ads?

And then also when you see an ad or a creative that works that talks about, I don't know, sleep for example, you're like, "Well, actually maybe we should make a bigger deal of sleep inside the product and prove the product." So it works both ways and it's I would say probably 50/50 right now of my time and attention between product and marketing.

David Barnard:

Well, we could talk about this for another hour. The zero to one step is so fascinating and all the ads up, but I did want to move on to the second phase that you're now moving into after getting this dialed in and that's the freemium strategy. So tell me about why you decided that this was a needed second phase to move into as a company.

Kenneth Schlenker:

So we believe that there needs to be great screen time software available for everyone. We think focus is very important in the 21st century. And what that means is we don't really have, I think, the luxury of being a smaller niche product. By niche, it could be a very profitable business with a few hundred thousand people, they're subscriber, but we really think that this product we're building needs to be used by millions of people.

One of the things we see is students, for example. We get a huge amount of organic traction with students, but the business model we have right now is focused on converting people to pretty expensive yearly plan that doesn't fit at all this audience. Very few students at scale I think will pay $99 a year, but there's a huge amount of traction. They really, really need it.

And I want to say students, I mean college, but also it goes down to junior high school, high school.

The mission for us is very important, and so we'd like to get in the hands of people who need it and there's probably millions of them. So we don't think we have a choice to stay just smaller.

And the second phase that we're entering is about building the best screen time app for all knowledge workers, all students, and in a way that allows them to use it daily for free. And that means we need to build a better freemium funnel, smarter upsells.

Most of what we've done in the first phase where we focused on day eight ROAS was about converting people right away onto a paywall where they pay an annual plan, mostly I would say based on intention.

I would call this the gym membership effect where you're going in, you want to change, and so you're paying for an expensive subscription, not only because you want to access the gym, but also because you want change and make a commitment to yourself. So that's worked really well in the first phase.

But in the second phase, we're moving away from this and building a more freemium funnel where you can start to use the product for free, do a lot of things, and then there's different ways in which you pay for the app subscriptions and also in-app payments, I would say like smarter upsells.

So we're doing maybe the opposite of, for example, a Duolingo did where they started off completely free, focusing on retention, and then figured out, okay, how do we actually get people to pay.

We're doing the opposite, which is we're figured out how to get people to pay a smaller number, but now we're looking at how do we expand this to many, many more people, including people using it for free.

Right now, the paid penetration, so if you look at the monthly actives of Opal, 17% are paid subscribers. And I think if you compare that to Duolingo for example, they're 78% paid penetration.

So what that means is we can probably have a lot more people using the product for free and convert a smaller percentage of people while still having a great business.

Why do I think it's necessary also to have a freemium funnel to grow much faster? I think it comes down to it's just a lot easier to talk to your friends about an app if it's a free app than if it's a paid app.

I think Duolingo's onboarding says learn a language for free forever. That's on the onboarding for a screen.

I think that the perception, even though actually their app isn't really free, there's a lot of paid features, but the perception as a free app is so important in organic referrals. And so yeah, that's what we're working on in the second phase.

And the other aspect is when I look around at other subscription businesses and maybe have more examples of this, but I think there's many subscription apps that hit a ceiling at maybe 10, maybe 20, sometimes $30 million in ARR, and they end up with a model where they can be dependent on paid advertising, mostly converting people in the onboarding. And it's really hard to break the ceiling. We want to avoid the ceiling by working on it now essentially.

Jacob Eiting:

If you scaled the ROAS, eventually, you will have attracted all of the available people you can attract, you can localize and things like this, but eventually you churn and your acquisition will reach an equilibrium probably around in that range.

It is fascinating to see the Duolingo move reversed. It also has to do with capitalization because Duolingo raised a ton of money to basically run unprofitably for a long time before they finally... And then they did turn on a big money machine, which is the bet. But it's interesting to see a more capital efficient probably way of doing this, which is starting from this position and widening out.

One thing I'm curious, is it going to be... To be able to maintain, you come in, it's a premium product, it's a gym for millennial knowledge workers, but then also that same app has to communicate to the 19-year-old freshman in college who's just trying to focus for their midterm.

Kenneth Schlenker:

I think it's hard. The way I look at it is this phase one, we pass level one. Right now, we're passing level two, it's just as hard to pass this level two.

The conviction we have is that if you look at Duolingo, it's like they were able to do democratize learning language, make it fun, and make it a game.

We think the ability to focus, it's not just screen time management, but it's about learning to focus is something that is just as big and maybe even bigger in terms of how many people need to use this.

Jacob Eiting:

Learning a language is a real luxury good. You know what I mean? You only do that if you have a lot of time and you maybe have aspirations to travel and things like this, but I feel like screen time is something that plagues everybody, right?

Kenneth Schlenker:

Yeah, that's the bet.

And so I think even if it's hard, we have this conviction that it's really mass market, a lot of people that could use this. And so the tricky part is... What we did...

So the first phase was simple, day eight ROAS is the main goal, the main metric. Phase two is a little bit more complex. We look at retention of course as a main metric, whether people pay or not, just how many people are still using the app-

Jacob Eiting:

Usage.

Kenneth Schlenker:

Yeah, usage.

And then the second thing is ARPU. We want to make sure that we can increase retention while having an ARPU, which is lower than it was before, but it doesn't go below a certain level.

So basically, we have those numbers and it's if we can increase retention where we want it to be and then reduce our ARPU the first year to above $2, we can change the trajectory of the business massively.

And actually, we've done some of it already.

One of the things we did, for example, is make it easier to dismiss the original onboarding paywall so that more people start using the app and more people start using the free version of the app. And we were able to do this in a way that increased activation and decreased our ARPU in a way that's manageable.

So basically, the net is positive in the long run. That's the way we calculate.

So if you can make those bets, remove a little bit of friction for payments early, and then that increases retention and then make sure that doesn't destroy ARPU too much. And if you can have those two work together, I think that's the key.

Jacob Eiting:

It's a fun example in just evolving your model a little bit, which I've found I've had to do several times at RevenueCat every year.

And sometimes, you forget and then you wake up one day and you're like, "The numbers I'm supposed to be caring about don't really represent what I care about anymore." And you have to go back and you'd be like, "Okay. Actually, these are the numbers we care about." You have to recast the business in a new set of metrics.

And it's good. That's the right way to do it. It's like think about that first, think about what you're trying to do, state your hypothesis, right? Then state how you're going to measure it, then go do things. And that's the real scientific method.

Kenneth Schlenker:

Yeah. And it's so interesting that you also come across that. I think it's fascinating to do this.

And speaking of how you spend your time as a CEO and what you focus on, definitely, the more you can also have time to take a step back and say, "Okay, are we optimizing for the right things? What have we learned in the last few weeks that either confirm or deny the strategy we're on?" is very helpful.

David Barnard:

All right. And then the third phase, and this is coming I don't know how soon, so maybe you can tip us off when you think you'll be able to start moving through this third phase, but the third phase being expansion. So after you start getting this freemium product dialed in, what's the expansion plan from there?

Kenneth Schlenker:

Freemium phase two is mentioned and then phase three is expansion into basically other platforms. So we're only iOS right now, Android, Mac, desktop, any device that you may own or come across, maybe others Vision Pro and others in the future because of course screen time matters everywhere. And also expansion in terms of distribution where we currently do B2C, direct to consumer. We know that we also have a B2B angle that we can sell to colleges, universities, schools, but also employers and many others, that would be really phase three.

Phase two, the goal is to work on this until end of next year and then beyond that, enter the next phase of business.

David Barnard:

You're working backwards from what a lot of apps tend to do. Start with Flutter. "We want to be on every platform from day one. We're going to launch iOS, Android, web, Mac, whatever." Why did you start iOS only? At five million ARR and not having an Android app is surprising compared to what we hear in the industry. "Oh, you should be on Android." And so why did you do that? And how's that playing out?

Kenneth Schlenker:

It's working pretty well. There's just so much to do to figure out the right kind of product and the right kind of way to onboard people that we felt it was better to do it on one platform than multiple platforms.

But also there's a timing thing where Apple essentially introduced the screen time API back in November of last year with iOS 16, and we were the first app to benefit from this API and to build on this API. So the focus was very clear. There's an opportunity right now to build on iOS and to build well.

I do think that there is a lot of value in being a cross-platform, like just even Mac plus iOS, if you work like me with your iPhone and then also your Mac, there's a natural extension there. So we're going to get there. But also quite resource constraints. We just want to do what we can with the resources we have, but this will come very soon.

David Barnard:

Yeah. I think it can be a trap for a lot of apps.

Jacob Eiting:

Yeah. Absolutely, it's a trap. It's very resource intensive. It's not linear in platforms like two platforms, it's not 2X to difficulty. It's probably three or 3X the difficulty.

David Barnard:

And it's not 2X of revenue.

Jacob Eiting:

The ROI is lower than the first one. And if you're still learning in your core platform and you're still iterating, those lessons, not all of them, but a lot of them will translate to your new thing.

And also, you're a moving target right now. This phase of transitioning into freemium may change how the product looks and feels pretty fundamentally potentially. And you don't want to have to do that twice.

Kenneth Schlenker:

Yeah. That's right. That's right.

Jacob Eiting:

Do it once... You can use tools and things like this, but I do think that... Especially as you think about word of mouth too, that's one of the big reasons to have an Android app is to make sure that every time somebody says, "Oh, bull. There's a listener on the other end that can install it and benefit." And from what you've told us from the mission, obviously, that's going to have to happen. But, yeah, for folks, it can really be a trap.

Don't feel the pressure.

And this is why metrics are important too because if you don't have a metric that's telling you you're not good enough yet, you're going to be like, "Hmm, maybe let's build an Android app. That's a good idea." And you're going to waste a bunch of time. It's like no, focus on the ROAS, focus on whatever. When you get past a certain scale, you'll have enough threads.

And this is an interesting transition point where you guys are at seven people or 10 people-ish, 20, 30, 40. Once you get to 30, 40, 50 people, then it's like, okay, we can put four people on an Android app and not derail the company's progress. But at this stage, you have one thing you can think about as an organization basically.

Kenneth Schlenker:

That's totally right.

The other thing I think depends... For us, it's a relatively new use case. People don't have a screen time software on their phones. I think 25% of Americans check their screen time stats, others don't. So it's still pretty nascent.

So there's a lot of discovery to figure out how do you build a product that helps most people focus. And so it's easier in that discovery phase to do on one platform. You're saying when you have two, you're stuck.

Jacob Eiting:

On a very different user basis too. Eventually... And this is, I think, talking into that expansion and distribution phase is like you'll build a brand that then you'll become the ones educating the world on screen time. And that actually has an advantage because when you go into those platforms, you already have this body of brand and experience and makes the conversion much simpler I think.

Kenneth Schlenker:

Absolutely. That's the plan.

To back up what you're saying, Jacob, I've seen a lot of horror stories from other startups and actually end up launching too soon. And when you ask them what's your greatest regret, they'll say, "Well, actually we went on Android too soon," or "We went on desktop or Chrome too soon."

Jacob Eiting:

And it's just a version of the spreading yourself too thin, too fast. We did it for years over and over again until we really started to understand how much can we chew at any given time. But once you do...

And that's the parallels of compounding as your company doubles or doubles and doubles, one day you wake up and you're like, "Okay, we actually have a lot of capacity to tackle more than one thing," and that's what you should wait for. It's like, "Wait."

So it's all about being frenetic and fast, but also patient. Very difficult thing to balance.

Kenneth Schlenker:

We'll do a followup podcast in a year. We'll be-

Jacob Eiting:

Sounds good.

Kenneth Schlenker:

We'll be all over. Every platform, every screen.

David Barnard:

Speaking of frenetic and fast, you've been on my list of people to invite to the podcast for a while now, but you tweeted recently something that reminded me, "Oh, I got to get Kenneth on." And that was you tweeted a screenshot from Notion with this massive list of A/B tests that you'd run. And then somewhere in the tweet you said, "We've run 121 A/B tests." So I wanted to spend the rest of time digging into this testing. That's a high velocity of tests.

The first thing I wanted to ask is what is your setup for testing? How do you actually manage 121 tests as a small seven-person team going from zero to one? It's a lot.

Kenneth Schlenker:

That's a lot. I don't advise everyone do as many. I think we probably could have avoided...

Jacob Eiting:

Only do the ones that work out.

Kenneth Schlenker:

Yeah, exactly. Just pick-

Jacob Eiting:

You save half of them.

Kenneth Schlenker:

Pick better.

No. But we did do that.

I think one of the advantages of having a single metric that you look at in this first phase, which is day eight ROAS, it makes it that much easier to A/B tests on a single metric.

And as I was mentioning earlier, it's also much shorter iteration cycles. So if you run an A/B test, you can get, in our case, because we have a seven-day free trial, you probably get most answers within eight days if you have enough volume, especially in early days where you're looking for big uplifts.

But our setup is... We use Firebase for A/B testing, but we run our tests through Amplitude, the data cohorts, data analysis through Amplitude, basically with a user parameter on Amplitude for baseline versus variance. And then we look at uplifts and figure out statistical significance because we A/B tested a lot on day eight ROAS, it has a lot to do with testing stuff very early in the funnel. The earliest in the funnel, the easiest it is to get statistical significance quite fast.

David Barnard:

Let's talk through, and we're going to have to speed run these, but we created a list of your top three positive results and your top three negative results.

So the first one that you did that was a positive result was showing the paywall early. So tell me about that test, how you executed it and then what the results look like.

Kenneth Schlenker:

Yeah. It's almost like common wisdom, but it is true that we found... We tested many, many iterations where to place the onboarding, where to place the paywall, what kind of paywall, how easy is it to dismiss or not.

David Barnard:

And when you say how easy to dismiss, you mentioned that earlier, you mean fading in the X versus not or full hard paywall or not full hard paywall-

Jacob Eiting:

Making it dance around when you try to click on it.

David Barnard:

That's right.

Jacob Eiting:

Those old ads in 2002.

Kenneth Schlenker:

We didn't try that one. We probably should.

Jacob Eiting:

It worked for Banzai, buddy. I know that.

Kenneth Schlenker:

No, but... Yeah. Definitely hiding or making it as discreet as possible. The dismiss button for example was a winner.

But I would say generally what we learned is you need to get the paywall to be as early as possible in the onboarding, but not too early.

And I think what you're looking for is the peak motivation moments where if you need a few screens to build motivation and build intent and then display the paywall, that's better than putting the paywall right away the first screen. That's what we found at least.

So for example, for us, what's interesting is you download the app, as you might find out if you do this, we are going to ask you what's your screen time. We're going to ask you a few questions and then we're going to build this mini report that will...

One of the things we'll do is it'll give you a lifetime projection of how much you're going to spend on your phone. And the average for a U.S. person is actually 22 years. So-

Jacob Eiting:

Oh, no.

That's terrible.

Kenneth Schlenker:

So the average American gets a smartphone age 12, 77-year-old is the average life expectancy, and then you spend five hours and eight minutes on average, which is if you look at only wake hours because 16 hours a awake and during the day basically adds up to a pretty large portion of your day and then you project that onto a lifetime. So anyway.

We make that projection and then after we make that projection, we tell you, "Hey, but the good news is we can save up to three, four, five years of your life together." That's the moment where there's high motivation and then people willing to sign up to a free trial and a little bit like a gym membership effects for sure, "I'm going to commit to this and commit to this change." And the benefit of this is you get people who of course pay, but also are quite motivated to try out the product and improve it.

So yeah, I think through various iterations we went from about 7% of people who start a trial to 17% of people who start a trial from download. That would be the number one.

David Barnard:

All right. So the second one was onboarding to a scheduled session. This is an interesting one.

Kenneth Schlenker:

Yeah. So what we found is if you onboard people to committing to something where they don't need to remember later to do it, like an automatic action basically.

So for example, for us, when you download the app and you tell us which apps distract you most, so you might say Instagram for example or TikTok, and then automatically we're going to create this Monday to Friday, 9:00 to 5:00 work hours schedule where when your work hours start, these apps will be blocked.

And so what happens is you onboard, you might exit out of the app, do something else, and then the next day, you don't need to remember, it'll just start to block the app and then you'll remember, "Oh, I committed to this, now I'm in." That's clearly something that was a huge win for us.

David Barnard:

Yeah. That's a really cool one.

We talk a lot about retention and onboarding and all these things on the podcast and, yeah, it's just such a powerful thing to onboard to something that's going to be right back in their face soon.

Kenneth Schlenker:

It works for this.

And I think also add one thing is so you basically make it an automatic action, but you allow editing. Editing should be very easy. So you make the commitment, but you can always come back and change it or cancel out of it. It's no problem. But the default is that you onboard and then it just works. You don't need to do anything else.

Jacob Eiting:

Yeah. Don't dump somebody into an empty... You should have an exact idea of how you want every person's experience to be and send everybody, right?

I used to describe onboarding, it's just like you want people to slide down into their first experience. No, you don't want them to get caught. No bumpy elbows, no nothing. They just click app launch and they don't do anything basically. And then boom, they land and they're figured out.

Simultaneous to that, you're pitching them, you're doing some other things, you're describing the value and the thing is you measured it, it actually helps and that's because that first three minutes, three to five screens is so highly leveraged, right? It's like every single dollar you're spending on ads hits that. Every single user has that experience. They're going to form an opinion about your app, probably semi-permanently in that first three to five minutes.

I sometimes see people design tests super poorly. So it's not surprising to me that these are the tests that you saw have really high output because you want to test something big and you want to test something that's frequent that people see a lot.

What would you test these against? Were your variants just like status quo and then new idea? Is that kind of how you evolved at ideas to test?

Kenneth Schlenker:

Yeah. Mostly just a binary test because we didn't have a ton of volume. Sometimes, we did variants where there's a different message or maybe a different placements, paywall position three versus paywall position five, that kind of thing. But mostly baseline versus variant and then adopting the winner and then doing it again.

Jacob Eiting:

The less extreme your test is, the longer it's going to take to get a result or potentially not get a result.

And that's I think something unintuitive about testing is that it's not just good or bad, most of the time it's inconclusive and also the time it takes to get conclusion is related to the difference in the two. So if they're identical, the time it takes to reach a conclusion about that is actually pretty long. So the bigger swings you take, the better.

Kenneth Schlenker:

Yeah. I will say one thing... So we didn't do well... A hundred twenty-one is a lot, but we could have done better at prioritizing the bigger swings, having a clear view of what you think the uplift is likely to be. And that's a little bit like guesswork, but when you can come up with an estimation and then prioritize, especially early, the bigger swings, you get a few percentage points uplift, it's not going to be enough. You need massive uplift at the beginning.

And sometimes, it's a very simple thing, just putting the paywall two screens early, two screens before can be plus 10% and trial started and that's a massive win.

Or sometimes you can spend a lot of time to build this new feature that will end up moving things by a tiny piece.

Jacob Eiting:

I've done that. I've built a long build out feature that has very little impact even though it's beautiful and it's literally just changing the opacity to the X button on the paywall to 80% increases revenue 20%. And you're like, "Okay, why do I even try?"

Kenneth Schlenker:

And it's sounding also to stay motivated when you've done these tests for the team. You work five weeks on something or three weeks on something and then changes nothing or it changes 1%. But yeah, I think definitely testing is like you have to be in the spirit of, but a lot of things will fail.

Jacob Eiting:

I think those failures are good though because I think it teaches you reality. I think too many people have some sort of magical thinking about how. Everybody's Steve Jobs and if they just apply their big product brain and think of the perfect thing and then they make it look beautiful and they ship it, it's going to matter. And sometimes that is true.

But I think most people have an unrealistic understanding of how the world works. And A/B testing is a really good way of putting your nose in it and being like-

Kenneth Schlenker:

Totally.

Jacob Eiting:

... "Nope, nobody cares." Or maybe they do. But I think the sooner you get over that as a product builder...

And that's all levels. You were saying, it's like the engineers being demotivated and even the folks coming up with ideas, teaching them where are the levers of change on our product, that's a thing I think a team...

Because once you've got it baked into the culture of your team two years from now or a year from now when some bright-eyed new product managers coming in, they're like, "What if we..." They're like, "No, we did this. Test number 63, total failure," which may or may not be useful, but that understanding of the physics of your business and what moves the needle and what doesn't I think...

Facebook was really good at this. That's what drove them. They just learned connections. That's everything. And it goes back to this one, simplified metrics and then also the iteration cycles, you guys. There's no way you did 121 tests if you weren't shipping every week. That would be impossible.

Kenneth Schlenker:

Definitely. And so you have to also pick a metric that allows you to do that, but that's very helpful.

Another interesting example is tiered referral reward. So basically, we have in the app, you can invite people and if you invite five people, you can get the subscription for free for a year. That's like an organic mechanism.

But one very small change that we did is we mostly make it look better like referral program and then have a tiered reward system where you invite five people, get the free year, you invite 20, you get a lifetime sub, you get a special prize at 50, and putting that into a beautiful UI that just gives people a goal of how to invite people. And that led to 80% improvements in how many people send the referral to at least one person.

Jacob Eiting:

It's really interesting to think about how... Because that's a very different user that you're trying to optimize around than the person who's buying the $100 annual subscription during onboarding. That's the kind of thing you're going to have to figure out as...

It sounds like you already have started to figure out, okay, make sure that you're coming in. "Are you 25 to 35 professional? Okay, you're going to go down this way." "Are you 17 to 22? You're going to go down to the referral program."

Kenneth Schlenker:

Actually, we do that and I would actually advise everyone to do as well. You just ask people their age and then differentiate the onboarding based on-

Jacob Eiting:

Change the onboarding, yeah.

Kenneth Schlenker:

... Or even the paywall. Yeah.

Jacob Eiting:

Yeah. Yeah, yeah. That makes sense.

David Barnard:

All right. So tiered referrals was the third positive result, but we got to speed run the top three negative result. And to Jacob's point, this platonic ideal of like, "We're going to run this test and it's always going to be meaningful," but it's meaningful whether it's a positive or a negative result and you should want more negative results. So tell me about the top three negative or we'll go one by one, but what's the first negative result that was meaningful because you'll learn so much?

Kenneth Schlenker:

Sure. We failed a lot, for sure.

David Barnard:

Yeah.

Kenneth Schlenker:

So these are the top three most meaningful negative results. But the first one, it's very hard to beat the Blinkist paywall. So I don't know if you know the Blinkist kind of paywall. It was published and I think in the mobile app sub community, it's probably a very well-known fact.

But basically this transparent paywall that shows you, "Hey, you're going to start the free trial now and then in six days you get a reminder and in seven days you'll pay your full subscription." It decreases the anxiety of people to sign up to a free trial. So we have that and that's the main paywall right now in the onboarding. And we tried dozens of others and actually keep trying, but we've never been that paywall. So a lot of failure around that.

Jacob Eiting:

Placement though, you seem to have tested well, but the actual content, it's not a huge lever. Yeah, that makes sense.

Kenneth Schlenker:

The content hasn't been that huge of a lever. We did experiment to get to that Blinkist paywall, but once we've had it, so far, we haven't made gains.

Too much choice in the onboarding is another one. So we tried in a few different ways offering people a choice on how to disconnect.

So I mentioned earlier that the default way right now is work hours, 9:00 to 5:00, Monday to Friday we're going to block all distracting apps and then you can go back and edit and choose, "No, I don't want work hours or something else."

Whenever we try to give people more choice in the onboarding, it failed. And that's a little bit to your point, Jacob, of like you want the smoothest-

Jacob Eiting:

It's got to be slick. Yeah.

Kenneth Schlenker:

Yeah.

Jacob Eiting:

There's limited fatigue budget. Users are only going to make so many choices and you need to make sure you're spending that mental energy on exactly the things you have to.

Kenneth Schlenker:

People live very busy lives and they're onboarding, they might be doing something else. They need something very simple to commit to.

That was meaningful in the sense that we know now that the automatic action is really the best way to do this, and we're still experimenting around this, but this is a clear one.

And then, yeah, the other thing is we've done a lot of tests that didn't work on re-engaging people after the initial onboarding in the first day. And a lot of it failed.

With one notable exception, I'll say. When people cancel their trial, we re-engage them with an offer, and that works quite well.

Jacob Eiting:

It's really interesting that they're non-committed enough to be like, "No, I'm not paying for this." But then you present them with a deal and everybody's like, "Well, I did get a deal."

I always wonder how much of it is that. It's like, "Well, I better take advantage of it." Well, who cares? You don't need to know why.

So how would you do that? But you would detect the trial cancellation via revenuecat.com and then you would just fire an email because you have their email or a push notification or what was the...

Kenneth Schlenker:

We actually do... So we do use RevenueCats, by the way, this is not an ad.

Jacob Eiting:

It is an ad. Don't let them confuse you.

Kenneth Schlenker:

So we do use RevenueCats for subscription status. We actually use a software called OneSignal. Great software. Currently, we send notifications with a 50% off offer actually right now. But it's being A/B tested. You might not get this offer, you might get a different one, but-

Jacob Eiting:

This is the... Don't send it to this.

Kenneth Schlenker:

Yes, please. Don't-

Jacob Eiting:

Somebody's going to send you a nasty email CEO at Opal. "I heard on the podcast that I could get 50% off."

Kenneth Schlenker:

This is not legally binding.

And then also in-app modal when you come back, say, "Hey, you-"

Jacob Eiting:

Ah, okay.

Kenneth Schlenker:

Yeah.

Jacob Eiting:

So making sure you get the visibility on it.

But it's interesting that you're doing it very much targeted in the app experience, which probably makes the most sense. People have the context and all of that.

Kenneth Schlenker:

Well, I think emails can work too. We used to collect them. We don't anymore as part of the onboarding process. We have a phone number sign in.

David Barnard:

I imagine that was an A/B test that collecting the email cause more drop-off?

Kenneth Schlenker:

Still to be determined. But basically, we have a social feature inside the app where you can see your friend's screen time, which I find is very cool.

Speaking about the ideas that turn out into-

Jacob Eiting:

I don't care about other people's screen time, but I think if I showed people my screen time, I think it would make me better about it. Right? That's got to be the idea.

Kenneth Schlenker:

Yeah. There's all this thing about positive peer pressure in the sense [inaudible 00:45:58]-

Jacob Eiting:

Sure, it's positive.

Kenneth Schlenker:

Well, in addiction science, AA, and others support group that you get of people that are supportive, but also checking in on what are you doing can help. And so that's the idea.

But so far it has been A/B tested. It was a slight uplift. We do get people that invite their friends, but it's not in the three most meaningful wins so far. But we're still working on various ideas to essentially focus better with your friends in different ways and have a support group. And so the phone authentication allows for that.

David Barnard:

I think that's a great place to wrap up, 121 A/B tests, a lot of failure.

Jacob Eiting:

How many you're doing in 2024? 200?

Kenneth Schlenker:

It's exponential. It's like the ARR.

Jacob Eiting:

Good.

Kenneth Schlenker:

Thousands.

Jacob Eiting:

You're doing something right.

David Barnard:

It's so much fun talking to you, Kenneth. Thanks for being on the podcast.

Kenneth Schlenker:

Yeah. Thanks so much for the invite, David and Jacob, and congrats on RevenueCat. Amazing software. I'm so excited to be using it every day.

Jacob Eiting:

Oh, geez. Thank you.

Kenneth Schlenker:

But I do want my t-shirts, please.

Jacob Eiting:

Okay. I'll get you T-shirt. Don't worry.

Kenneth Schlenker:

Thanks.

David Barnard:

Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community.