Motivation: every product promises the new buyer something about how it is going to help them make progress in their lives. What is it going to fix or help or change that is it going to be so valuable that you should use it and buy it? As a data scientist I want to be able to quantify that promise, in terms of measurable actions and outcomes, and then assess the progress users of a product are making towards that promise. “Onboarding” a new user to the product and getting them to the value and the progress they are trying to make, and then understanding the steps and the time it should take.
The Promise
What does a new user think that your product, or any product, promises to do for them? On a recent multi-state drive I had the opportunity to listen to hours of radio advertisements (thanks to my 6th grader’s insistence on top-40 stations only) and got to hear many “promises.” “We’ll give you cash back on gas purchases,” “We will protect your privacy when you go online,” and “Find the lowest prices on a new car.” However, what the marketing team thinks is the promise is often not what I, the consumer, understand it to be. In buying our mini-van we didn’t need features or lowest prices, we needed reliability, confidence in the safety of the vehicle (precious cargo onboard!), and enough inner capacity to easily pack everything we could need for a weekend trip to the relatives or camping. We ended up paying more for the vehicle that satisfied, and delivered on, the promises that we were looking for.
Examples
Slack: I was brought on to Slack via collaborators in a research group, and was ‘onboarded’ through needing to communicate with many coworkers in remote settings. There’s an anecdote about that Slack knows when you’ve adopted their product when you’ve sent your X-th message (X=30? 100?). However, this is what their front page is promising today (June 2022), and my critique would be that they could work on their product positioning:
Primary Call-to-action (CTA) on Slack’s homepage, June 2022
I don’t think I’m trying to level up my teamwork (at least that’s not language that resonates with me), I’m just trying to stay in close, less-formal than an email, communication with my coworkers!
Carvana: recently I had a car to sell and I was getting low-ball offers from the dealers. I’d seen Carvana trucks driving on the highway with messages telling me it was easy and they were straight shooters (aka trustworthy). I went to their site, found the 3rd CTA on the page was for me. 2 minutes and they’ll come pick it up, I don’t have to spend time on the phone and worry about taking it somewhere? The beginning (get the offer) and the end (handing off the car) were the most stress inducing parts for me, and they were promising a low-stress option (the offer was great, by the way, and they delivered on the stress-free sale all the way. Would recommend 5 out of 5!)
The “Sell” CTA on Carvana’s homepage, June 2022
Boardable: I am always happy to lay a critical eye on my own company’s product and chide my coworkers, so let’s add it to the mix here. This is the product that I need to measure the “promise” and onboarding success of users and customers, so knowing the ins and outs of how the product is positioned and how the team thinks about its value propositions is foundational to this endeavor.
Positioning message featured on Boardable.com, June 2022
Boardable is a Board Management portal, originally positioned as a “Board Engagement” solution. “From Friction to Focus” appears to be the primary promise, and internally we often discuss it as reducing the chaos of managing many people that are often geographically dispersed, where the primary competition is “digital duct-tape” (aka a mix of Dropbox, Google Drive, Zoom, email, Survey Monkey, and personal calendars). In this case we can’t measure it as X messages sent, or 1 car sold, because “reducing friction” or “obtaining focus” is nebulous and probably means many things to many different people.
How do we measure it?
Retained: to complete this analysis we need a working definition of what qualifies an active, successfully onboarded user, and the definition can be refined later. We need enough data points (enough users) to assert confidence in the results, so one approach could be something like “the most active Y% of users that gives us at least N users.” Measuring “active” as logins per month, or some other generic metric, is sufficient. Another, which I use, is “all active users with a 6+ month tenure”, where ‘active’ is more than 1 log in every month.
If you can identify a set of retained customers we can theoretically work backwards to finding the “onboarded to the promise” metrics. What did these retained (presumably happy with the product) users do in their first 90 days?
But where does onboarding start, one might ask? Don’t over think it! I’d start from the first time they created an account or installed and opened the app. If you have enough samples (accounts) then idiosyncratic behaviors and gaps between getting started and onboarding all the way will wash out over the whole data set. Remember, we’re comparing across a breadth of long-term, retained users who we can hypothesize have successfully found the promise in the product (as they understand it) or else why are they still actively using it?
First, it’s a convenient approximation of 3 months, and thus easy to communicate, and is used commonly across the industry as a benchmark for successful onboarding, making it easy to compare across different authors. Theoretically, if a user hasn’t adopted regular usage of your product within 90 days of picking it up they probably won’t. Unless they come back to try again at some point down the road, but then they’re really starting over and experiencing your product again for the first time.
Conversely, if they become a weekly or daily active user 90 days is not too long and not too short to surmise their longterm retention. Rule of thumb: the bulk of software users try an app or a product for less than 5 minutes and then leave, so if they’re coming back over a 3 month period they can be usefully analyzed as a case of successful adoption.
When looking at a particular action (sending a message, scheduling a meeting, uploading a document, etc.) active users did during their first 90 days we typically find a distribution with a long one-sided tail. Most of these happy, retained users found the promise with a smaller amount of activity, a few really got into it and did our thing lots and lots of times, and some probably didn’t do this particular action
How many accounts (vertical) uploaded how many documents (horizontal) in their first 90 days using Boardable?
One of the primary features of the Boardable platform is cloud-based document storage, helping users consolidate their board related materials (“from friction to focus”). In the first 90 days (see histogram graph) lots of active users uploaded fewer than 100 documents but a smaller number of users went on to multiple hundreds. Because of this long tail (the highly active users) an average will skew to the high side (average=71).
As a number to assess the onboarding achievement of new customers a goal post of 70 is unreasonable, requiring the new user to upload 3 documents every 4 days. We know from customer and prospect interviews that users upload documents at two times: a tranche of organizational policy type documents when they first start, and then a set of reports and materials around each monthly/quarterly/yearly board meeting. Clearly some customers were very active in their first three months, but many others had far fewer documents to store and share. To obtain a more realistic goal post for onboarding assessment we could trim the long tail first, based on some percentage, or take the median of the set. In this case median=32.
| As a sanity/gut check let us look at this same user group and how quickly they uploaded documents, taking the median at 30, 45, 60, 90, and 120 days. It charts as an impressively linear trend. These all paint a picture that is understandable and would smooth out to around 1 document per 3 days for a successfully onboarded customer. I would surmise a successful user has figured out the nuances of document storage after around 5-10 documents (including creating folder structures), and is “activated” by 12-14 (the first 30 days). By 90 days and 30 documents we can hypothesize that a user is comfortable navigating the storage system and uploading and deleting documents. | 
|
How Fast?
What is an ideal speed to get a customer to this onboarding success mark of 32 documents? How quickly are they getting to this “promised land” of product value? We’ve looked across all long-term active, healthy customers and found that the median account uploads 32 documents in their first 90 days. However, some will hit 32 in their first week, and some will take longer than 90 days.
We con do the same thing, counting the days between all of these user’s first document and their 32nd document. It has a very similar distribution with some users taking 90+ days (forming another long tail) and some getting there in a couple days. Taking the median gets us: 60 days!
Conclusion
From this analysis I can report that a good sign of long-term health for a Boardable customer account is uploading around 30 documents within their first 3 months. We can understand this ‘story,’ if you will, as a proof point that the new customer is “reducing friction” (a promise) and finding value in the platform by storing and sharing their materials and documents. Certainly if a user has not uploaded any documents in their first 90 days they have not found value there, and if they exceed the 30 document mark then we can deprioritize outreach to them and focus on the ones that may need more resources, training, or support.
Secondly, while 30 in 3 months is the rule of thumb, the median long-term successful customer actually gets there in 2 months. That would be a helpful expectation to set both with the customer and internal teams to proactively work towards more accounts reaching the land of product promise!
Finally, this analysis should be repeated for every potential primary activity that ties to the promise and product value propositions. At Boardable this includes: meetings scheduled and held, agendas assembled and published, discussions started and replied to, and more. Each of these primary actions is charted out with onboarding milestones at 30, 60, and 90 days, which inform a very robust process to ensure as many customers as possible realize the promise and make the progress they are hiring Boardable for.
Dr. Ben Smith is a Data Scientist and thinker, fascinated by the appearance of computers in our daily lives, creativity, and human struggles. He has had the privilege to think, learn, and write at the University of Illinois, the National Center for Supercomputing Applications, the Cleveland Institute of Art, Case Western Reserve U., IUPUI, and at Boardable: Board Management Software, Inc.
If you have feedback or questions please use Contact me to get in touch. I welcome thoughtful responses and constructive critique.