How to measure product usage?

Becoming a product manager FAQs
, ,
#1

Hello all,

The company I work for has a SaaS B2B product, and I would like to monitor the web dashboard usage. For example how many users download a specific type of report or how many users view specific types of data in our web app.

Can someone recommend me a tool that can be relevant?

Thanks!

9 Likes
#2

Mixpanel can be used here. You can check how many times any CTA was used between a time period. This way you can see how many times a report was downloaded or a filter was used on the analytical dashboard.

9 Likes
#3

How difficult would it be to implement Mixpanel?

9 Likes
#4

@AngelaBlue, I’m going to disagree with the person you are replying to. They can’t accurately guess effort here.

If your product is a data dashboard, I’m guessing it’s a rich application making heavy use of JavaScript. If that is true, you likely have to do a moderate amount of work to meaningfully track user actions (events like viewing a dashboard, downloading a report) and relevant metadata about them (e.g. “specific types of data.”) Google Analytics and Mixpanel track only universally available information out-of-the-box like URLs, page titles, browser info, but anything more complex than that is not going to be automatically tracked by their default installation. You should loop in an architect or the lead front-end developer early in your consideration since only they can advise on how difficult it will be to meet your requirements - this should be treated like an integration.

I would also advise not to skip this advice and simply ask your dev team to copy/paste the installation script as these tools frequently suggest. Worst-case scenario, you will be getting a bunch of invalid and hard-to-interpret data, like a bunch of sessions that appear to only have one page view to your homepage and 100% bounce rate because Google couldn’t pick up on new history states of your application.

Lastly, Google Analytics does not give you a single view of any one customer, so drawing meaningful conclusions about user behavior takes a lot of guess-work and requires you to start first with a hypothesis and then try to figure out how many sessions met that criteria. Mixpanel is better in this regard since they do let you narrow in to the individual and find unpredictable behavior among each (like navigating back-and-forth between reports,etc.)

Tools like Heap, Fullstory, and Logrocket do autotracking, which means they automatically track everything they can about the state of the page the user was on every time they clicked or meaningfully interacted with something on your site. The downside (per the above article) is that everything is getting sent to a third party, so you need to make sure data that shouldn’t get tracked to a third party is explicitly excluded. It’s also a bit obscure to try to model all of the clickstream data they capture from their tool into meaningful metrics for long-term reporting, and they may not actually retain data very long because they capture so much. Your rules can also break as soon as your developers change something about the front-end that you were relying on staying the same. For instance, you can set up a metric that counts how many people clicked a button with the inner text “Log in” or how many people clicked a link with the HTML class “download-report”, but if those attributes change on your site later, those rules will break.

The upside is you can learn a TON about your customers if you are willing to spend time with the tool and you don’t need a significant developer resource to install them since they do a lot out-of-the-box.

If the question is “how do I know what users are using on my application” I frequently advise something that looks like this:

Install an auto-tracker with session recording built-in, like fullstory or Logrocket

Check in on day 2 to make sure it installed OK, but then let it collect some data for a few days

Come back and spend a day observing user sessions. Take lots of notes - what is remarkable and noteworthy about each session? What surprised you? Which behaviors would you like to measure over time, and do you want to know that per-page, per-session, or per-user?

Build your tracking requirements off of this - you’ll probably find a lot of events and application states you didn’t think of

Install something like Mixpanel or GA off of those requirements so you can reliably track those data points and events over time, and use those tools for your trend analysis, cohort analysis, acquisition and attribution analysis, etc.

Check in on your session recording tools a few weeks before you plan your next release so you know there aren’t any new surprises, and monitor them for hypercare after any higher-risk release.

9 Likes
#5

What a great answer! @Michael, This is great. Can you also use HotJar for step 1?

9 Likes
#6

@DonovanOkang, HotJar is the example I give of why the MVP mindset doesn’t always work in a high-competition space. I got an invite to their product when it first came out and it had so many limitations that it wasn’t a good substitute for any other UX tools in our stack. Like you could only collect X session recording per sample and had to install new code if you wanted more. It is probably better now but I already wrote them off.

It looks like they have some of the functionality of the aforementioned tools. Session recordings, yes. Event filters? From what I can see of their docs there’s no autotracking, so you would still need to be explicitly passing in event data to meaningfully filter sessions. And that’s only with their most premium tiers and without any metrics functionality. So you could watch your hundred sample sessions and make your generalizations, but then trying to size up that behavior in Hotjar to verify it’s “worth formal tracking” is going to be a bit harder

8 Likes
#7

It’s not much difficult. Mixpanel has support documents for the team to add the code and test it. A day’s work.

8 Likes
#8

Mixpanel is a little difficult to implement as you have to send all the important events using code to it. You can try Browsee as they have a good free plan as well It is a session recording tool that automatically tracks all the user events and gives an exhaustive search over them. Also, it has a lot of options for privacy settings where you can choose what to record and what not to record.

7 Likes
#9

Thank you! I will check it out.

6 Likes
#10

Amplitude

Google Tag Manager + Google Analytics

6 Likes
#11

Do you have any good resources on implementing this approach. I’ve joined a company with Google analytics which is fine for getting hits on pages but I’d like to see more details like where they fall out of workflows, etc. Not sure Google analytics is suited to that.

5 Likes
#12

In our platform we have inbuilt events, which are added to track the things we’re interested in, such as user creates an account, user creates or modifies a widget, counts of usage of a new feature. This is all pushed into a central postgres database and I have built a daily report in PowerBI which we use to monitor how things are going.

Part of each new development process is some thinking about how we track its usage.

Anything more specific we can ask the development team for a SQL query to be added to the schedule, so we can track non event data via the same DB into PowerBI.

4 Likes
#13

Customer Satisfaction is the end goal we all want. Software tools that track clicks are the most obvious options but end users have to pay a heavy price usually in terms of the performance and then there is also the cost factor if you have to deploy it to a lot of customers.

We have instead resorted to:

  • Daily Email Report - Used for mostly new products/modules
  • Quarterly Reports - where we capture the usage of individual features
3 Likes
#14

@MarcoSilva, The performance point is moderately ridiculous. Most trackers including the default Google Analytics and Adobe Analytics pixels get loaded asynchronously and send data asynchronously. The libraries are under 100kb and get cached the first time a user loads your site, and the data requests are typically under 2kb unless you’re sending huge payloads. But again, all of this is happening async, so your only consideration is if you are serving to users on a metered connection.

3 Likes
#15

@Michael

There are two primary goals to measure product usage - Room for improvement in your tech solution & room for improvement in user behavior(clicks to perform a particular operation). Once products start to mature, Product strategist and PMs are forced to spend bulk of their time to make existing features one bit better with each release. New innovation is hard to come by. Take for example Apple - They are differentiate iphones by the number of camera and for that matter, smart phone industry hasn’t seen any revolutionary innovation in the past 5 years or so.

Google analytics doesn’t even scratch the surface when it comes to the detailed sequential field level tracking across multiple screens needed to understand user behavior when trying to bring change in behavior. Lots of companies, including mine are focused on saving every single click. And If the nature of workflow requires a doctor to click 20 times to load screens, not counting the clicks for documentation, every additional micro sec will count. Lots of software companies boast about their product performance and I am fairly positive this is important in many other industries. I don’t think a stock trading software showing live values would want to put additional load on the processors by deploying a click tracking tool. But then again, I am learning new things everyday and so I could be wrong.

2 Likes
#16

@MarcoSilva, Did you develop this in house? Was this a lot of effort?

2 Likes
#17

First we developed our own internal CRM software and then built all reporting on top of it. Our company believes in doing it ourselves. They even tried to develop a word editing tool instead of using word… lol That didn’t go that far!

1 Like
#18

So this seems to have been somewhat alluded to, but…

  • You say you want to “monitor the web dashboard usage.” That seems a bit ambiguous. Do you mean there’s a web dashboard for users and you want to monitor its usage? Or you HAVE a web dashboard to which you want to add some product KPIs?
  • You say you work for a a SaaS B2B product. Someone there must already have some usage monitoring tools, no? If there’s no PM monitoring things, you either have a major problem or major opportunity. Chances are you’re already using something, which could be anything from the usual tools to internal BI tools running off your own databases and logs.
  • Personally, I like the web tools and have used several; from the ubiquitous Google to Mixpanel, Adobe, etc. I’ll tell you this though, at a certain point, I like to limit Google’s access to all things. At a certain volume, you really should ‘own’ your own stuff. And maybe you graduate out of the basic tools. I also never wholly trust http ping type tools. Why? Because I’ll typically pick a KPI or two and have logs monitored for just couple of them and build a report to compare to the web instrumented metric. Guess what. They often don’t match. There’s lot’s of reasons why, but basically, this provides me with a health report on instrumentation. If there’s a change in divergence, something’s up. (Usually, a poorly tagged web item.)
  • So you need to work with your dev team to determine best and most reliable approach. And ideally, you can just add your metrics / KPIs to an existing dashboard; though someone may have to add tags to the items in question. Remember, pretty much all the time you need to time scope your numbers anyway. So ideally you’ll have that accounted for as well.
  • What about funnel? WHY do you want to monitor these numbers? What will they mean? What if they go up or down? How will you know why? Will you need to do something about it? And are customers who use these products more or less loyal or profitable? You probably need to tie in your numbers to a longer user journey / conversion funnel. Otherwise, they’re really just metrics, not KPIs. And if you can’t sort out just why they might be changing, you’re going to be running more guess testing rather than more sensible hypothesis testing.

Find out what’s being used now. And sort out the usefulness of the numbers you want so when you make your ask of the dev team, you’ve got all your actual product needs covered; rather than just a simple count.

1 Like
#19

@NaomiNwosu, Thanks for the response.

  • Nobody does usage monitoring here.

  • When a customer purchase license, he receives credentials and can access our Web Application which is unique per customer, different dashboards and data. I would like to monitor features inside the Web Application and if possible even per customer. I am not sure this is possible with tools such as Google Analytics / Adobe.

  • I plan to work with R&D but first before I come up with tickets to develop something in house, I want to check if there is already available solutions that can save us the time to develop it in house and spend time on development that sometimes can be very expensive.

  • I am not sure exactly what I want to monitor, once I will have the tool I will add the tracking wherever I want, for example, to count how many users download a specific type of report, visit a specific page, use a specific button.

1 Like
#20

With most tools, there are ways to get around internal permissions, including Google Analytics. There’s just a little more work to do so.

When you say you want to count report downloads, use a button, etc. I’d also suggest you develop an “Even Taxonomy” for such things. That is, there are some singular events of importance, (like a report download), that are maybe somewhat different than specific UI / UX button click events, which you might want to roll up into other category level reports.

The really fastest, cheapest way to do this sort of thing is Google Analytics. Again, I kind of prefer to own my own data, even if that means more expensive Enterprise software buys. But… if that means you’d have to run the procurement gauntlet and justify the price… just slap Google in there and be done with it. You’ll also get the funnel reports, can set up goals, etc. And it should not be hard for dev to implement all this.