How PagerDuty Used Analytics to Improve Their Trial Experience

Editor’s Note: This blog post was originally featured on Segment’s blog.

The Business Challenge

Even though we heard from customers that our service was easy to get started with, we knew there was room for improvement.

Occasionally users would finish their trials without seeing all of our features or miss an alert because they hadn’t set up the right notification rules. We wanted to help all of our customers reach success, and we also knew that better surfacing PagerDuty’s value would boost our trial-to-paid conversion rate. But we would first need to understand exactly what users were doing in our product, and where they were getting blocked.

We needed to establish a baseline understanding before making changes to onboarding:

1. How were our users moving throughout the app?
2. Did their paths match our expectations?
3. Did they match the onboarding flow we were trying to push them through?

When we weren’t sure how to answer these questions, we looked into analytics tools and narrowed it down to Kissmetrics and Mixpanel. We wanted to see both of these platforms in action, so we ran them against our live data and tried to use them to answer the questions above.

We were excited that to find a tool that would let us make a single tracking call and view the data, simultaneously, in multiple downstream tools. Segment made it easy to test both platforms at the same time, and with more departments looking at additional analytics tools, the prospect of easily adding future integrations also excited us.

How Kissmetrics Helped

Using Segment, we tracked every page in our web app and as many events as we could think of (excluding a few super-high-volume events such as incident triggering). Our UX, Product, and Product Marketing teams dove into both Kissmetrics and Mixpanel to evaluate how well each could answer our usage questions.

We found that we really liked the funnel handling in Kissmetrics, as well as the multiple methods for side-loading and backloading out-of-band data like sales touches.

After combing through the data, our UX team had a lot of ideas to develop a new onboarding flow. We mocked up several approaches, vetted them against both customers and people inside the company, and tried out the one that we thought would best communicate the value of PagerDuty and help customers get set up quickly.

Here’s a look of our onboarding flow before and after.

Old:

New:

Results

Using Kissmetrics, it was every easy to measure how the new design was working. Our core metrics was a cohorted analysis of “Trial Engagement,” an internal calculation for when an account hits a particular state that shows they’re doing well in their trial. We also used the funnel report to look at fall-off in each stage of the wizard.

We saw a 25% boost in the “engagement” metrics, measured using Kissmetrics’ weekly cohort reports— a huge success. Kissmetrics showed us that with the new wizard, new users were actively using the product earlier in their trial, and using more of the features in the product.

Plus, far fewer new users were ending up in “fail states,” such as being on-call without notification rules set up. What’s great about Kissmetrics is it tells us who is doing something, not just how many. So if we needed to reach out to active users who were ending up in these fail states, Kissmetrics would make this possible.

We learned a lot with our first foray into user analytics. If we were to do this all over again, we would have tracked far fewer events, only collect events that signaled a very important user behavior that contributed to our metrics for this project. When you track too many events, it gets noisy, and blocks you from accessing the full power of the different reports in Kissmetrics.

Going forward, we’re going to try running A/B tests trial experiences to get cleaner data about how our onboarding work is improving our trial users’ experience. With Kissmetrics in our arsenal, we’ll make better decisions on creating the ideal trial experience.

About the Author: David Shackelford is a product manager at PagerDuty, an operations performance platform that helps companies get powerful visibility, super reliable alerts, and faster incident resolution times.