Aha Moment Testing Dashboard at SaveDay

1. Why did we start?

When we started building this new product about 3 months ago, we needed to establish the first version of our key product metrics. In a B2C context, we cared most about growth and retention. At that time, I was working on retention and I didn’t know where to start with defining our retention metric, let alone improving it.

I came across a lesson in Reforge’s Retention + Engagement program about the three user states of activation: the setup moment, the aha moment, and the habit moment. I knew that defining our product’s aha moment would really help us target our efforts to retain users better.

2. How did we implement it?

There were two steps to define our aha moment. First, we had to identify all the actions a user could take that create the potential for them to be retained. And second, we had to determine the threshold for those actions.

In this dashboard, you can see the first step visualized on the retention curve chart. We segmented users based on key actions within the product and compared their retention curves, as well as the retention of all users within the product. The best-performing curves indicate which actions are likeliest to influence retention. Then we move to the second step: finding the threshold. Our goal here is to find the biggest overlap between people who took the action and people who retained. You can see these calculations in the tables below the chart.

3. What will we do next?

This dashboard isn’t the source of decisions; it’s only the source of probability. The next step is to build hypotheses based on this data and test them in the product. For example, if we see that the people who saved photos 3 times have the best retention, we will put effort into ensuring new users save photos 3 times. We might run an A/B test and compare the retention curves to see if the new one lifts. It’s also important to combine the quantitative data with qualitative research to understand which actions are bringing value to users.

Setting up this dashboard wasn’t a one-time project. It evolves over time as we learn more and iterate on the product. We have to continue doing qualitative research with users as well to truly understand all the actions they take and how they get value out of the product. Even our definition of retained users has changed over time; currently, it’s users who are active in the third week after onboarding, but at first it was just one or two weeks.