How much responsibility should PMs have for user analytics analysis?

Resources for product managers Tools and pro-tips
, ,
1

Hello party planners! Regarding user statistics and product management, I have two inquiries.

  • Do you have an interface that handles user data analysis for you, or how much of it is your responsibility? It would be good to know your title and the size of your company.

  • Do you know of any analytical courses or resources for product management?

I’ve been a PM at a start-up for almost three years, thus I have these questions. We are only now in a situation where we both need and are able to invest in our user data analytics to inform feature development and decisions due to very limited resources. I have little experience with SQL and analysis, therefore I’d like to know
(i) how much it is required of a PM to have and use these skills, and
(ii) what tools are available to me to assist me build them.

To be clear, I believe these are useful skills to have, and I want to study them in any case. Additionally, I just want to be sure I’m making good use of my time as there may be other options (like hiring someone) for filling the gap in our user analytics study.

As a BI/Analytics tool, we now use Looker. Another team is in charge of pushing our user events (as well as other kinds of business data) into Looker, with the goal of providing “non-technical” stakeholders with a single source of information for all of their queries. I’d like to answer some rather complex questions, and this is the skillset gap I’m attempting to close.

A word on how we set up our user event database: We decided long ago to track all user events ourselves while we develop features instead of using a product event tracking vendor because of our healthcare situation. We choose which events are worthwhile recording as part of each feature release, and then the engineers collaborate with our analytics team to agree on name and definition. After that, the analytics team sends the data to Looker.

Your valuable inputs and insights are welcome.

8 Likes
2

The short answer is yes. You need to understand whether consumers are using your features, whether evidence suggests they like them, and whether there is opportunity for development. If you can’t provide facts to support any claims lower than mere revenue, all you have are people’s opinions or the loudest voices.

The long answer is that you acquire the information you have time to gather. I was shocked to discover that we hadn’t previously collected page views on certain significant features, let alone more useful data, as I am currently working on a project that is defining needs for metrics. So, I’ll have to harass the developers to do some time-consuming queries for me to retrieve the data. I don’t do SQL and doing it right now wouldn’t be a wise use of my time. Therefore, the procedure will be more difficult because someone in the past didn’t ensure that a feature had at least Google Analytics level of analytics. You can sure that one of the tasks we undertake in the future will be to obtain at least basic analytics for each product we implement in order to spare our PMs this kind of suffering.

8 Likes
3

I really appreciate your reply. Fortunately, we’ve done a great job of collecting user events; we just haven’t put in the time to percolate that data up in a way that makes it easy to analyse and guide decisions. A team is feeding data into Looker as we only recently started utilising it. This is where I lack the necessary expertise—using Looker to generate insights. I can answer a number of straightforward questions, such as MAUs and aggregate in-product conversion rates, but I feel unqualified to do so given my present level of expertise.

I’ve already determined that I need to acquire this talent in order to succeed in my firm and my current position. I was interested in learning how other businesses handled this. Thank you very much for sharing your wisdom!

7 Likes
4

I’m in charge of it. Despite working in an independent, corporate-sponsored “startup” and holding the position of VP, I currently manage a smaller team. I’d want to have a full ProductOps function, similar to DevOps, with staff members who are more skilled in data science than I am, but that will have to wait for the time being. If necessary, I can rely on the data science and analytics team from a large corporation. My tech lead is also skilled in Tableau and SQL.

However, I only utilise my Pendo dashboard and data explorer tools to look at instrumentation data of what my consumers clicked on. Similar tasks can be accomplished via MixPanel, Amplitude, and Google Analytics. To find out more about what is often done, I would consult the resource that these suppliers publish.

A tip specifically for the healthcare industry: be careful while using these technologies to ensure that PHI isn’t unintentionally captured. We adhered to some of Pendo’s recommended practises in this area.

I wanted to add that Pendo, at least, collects whatever it can see after it is installed in your product. Therefore, you may go back and ask questions regarding clicks, page views, and other things that you didn’t configure in the beginning. This relieves you of a lot of the analytics/data science stress related to “have I thought of everything” Simply deploy the application (Pendo even offers a free entry level tier) and begin answering some of the fundamental questions, such as "which browsers do my users use? understanding that you may look back to the day you implemented the tool in your app to get more advanced later.

7 Likes
5

Thank you for your reply @DianneStinger! We use Looker, and while I can get around it, I feel like I lack certain fundamental analytical abilities. For instance, when I recently reported on the performance of a new feature rollout, some stakeholders questioned the validity of my research. In the end, my interpretation of the data was accurate, but some of the assumptions I was making weren’t backed by the data I was looking at.

Fortunately, we’re quite good at tracking events; it’s just that I have trouble analysing them.

To safeguard both our users’ PHI and ourselves, our business has a remarkably strong security and privacy function. This is another reason why we are unable to use Pendo as we long ago made the decision to keep track of every event ourselves rather than using a third party service.

7 Likes
6

Senses well. Here, the disciplines of data science, analytics, and product management come together with the science of usability. Without knowledge of the three distinct subjects, or at the very least the pairwise combinations, it is difficult to find strategies to learn about the intersection. There isn’t a place that I’m aware of that teaches all three together.

Numerous publications and videos on quantitative usability testing are available at Quantitative User Research: Study Guide from NNG, which also addresses the usability + data/statistics component.

However, the product management team decides which usability metrics to even consider and how to use the results. Making sure you can deliver the tale to your stakeholders in an engaging way is perhaps the most important thing here. The data lend credibility to the story. However, the art of product management—and leadership, in general—requires making judgement calls based on scant information. What commercial premise are you evaluating? What tests are you conducting to evaluate the theory as rapidly and inexpensively as possible? These tests ought to take place long before any software code is produced. Although Strategyzer contains a wealth of information, framing your work may benefit from using Testing Business Ideas: An Experimentation Field Guide for Rapid Testing of Business Ideas Testing Business Ideas - Innovation Process to Reduce Risks

It may not be your talents in quantitative analysis, SQL, and statistics, in my opinion (not that these aren’t valuable skills). However, learning how to structure the data and analysis you do have to explain what it informs you about the hypotheses you are testing may completely avoid having some particular analysis criticised.

You should also have a look at “Product Ops,” which was given that moniker to reflect the DevOps movement. According to this theory, product teams would profit from having a special sub-team with expertise in analytics, scripting, dashboards, and related areas. One of the thought leaders advancing this idea is Todd, the CEO of Pendo, so even if you can’t or don’t want to utilise Pendo, check out his ebook and blog postings, or just Google “product Ops” and follow your nose: Explore Product Operations (Product Ops) Resources | Pendo.io

I sincerely hope this was of some use.

7 Likes
7

In response to your opening question, I believe so. I believe that success requires at least a thorough comprehension of set theory and fundamental statistics. Without those abilities, I personally do not know a single outstanding PM. Although SQL is not required, I believe it is a major benefit. Though I’ve only been in technology for about 5 years, so it’s not like I’ve seen it all, take that with a grain of salt.

Regarding the first bullet item, in my position, I am essentially in charge of conducting my own A/B tests and confirming product concepts. I don’t have a strong group of analysts I can turn to for statistical testing or advice on what metrics I should be keeping an eye on, etc. I create my own Looker dashboards, and when I can’t find the ideal explore, I use the SQL runner rather than submitting a support request to the analysts. I also notify the data team of any flaws in our data and, if they are engineering bugs, follow up. If not, the data team is in charge.

For reference, let’s say that our analysts, who essentially serve as analytics engineers and are in charge of the reporting layer, resemble yours in many ways. They do not serve in that capacity.

Of course, this isn’t ideal, but since I worked as an analyst for a while before changing, I don’t mind. also dabbled in data engineering.

The other PMs at my present employer, however, are all data literate but possess no SQL expertise. They also conduct their own testing, choose the metrics, and observe their A/B tests. Where I am currently, statistical testing is not thought to be necessary. In Looker, they don’t create their own dashboards. I am undoubtedly weak in many aspects compared to the other PMs because they have substantially more PM experience than I do, but my experience as an analyst has been very beneficial, and I also assist others in planning their tests.

Growth PM, size 90, org.

I’m unable to provide you with much assistance that is pertinent to the second bullet item. Coming from an analysis background, I believe my viewpoint would be very different.

6 Likes
8

The answer to “is the product manager responsible for” Yes is the answer. A product manager is conceptually thought of as the product’s little CEO. To create the finest possible product, you turn into the person who must solve problems.

It’s no longer someone else’s job to analyze data. Practically every position in a firm requires data literacy.

Sorry, but I don’t have a boot camp or course I can recommend. Regardless of the business intelligence product your firm uses, there are probably training films available for you to watch.

5 Likes
9

@HerbertWarnick, you are so correct, my friend. I frequently compare my job to an upside-down umbrella that just catches everything needed to make the product (and occasionally the company) function.

I appreciate you responding and letting me know that learning this skill wouldn’t be a waste of time. Every company has a different approach to PMing, and I have already made the decision that in order for me to succeed at my particular company, I must invest in this skill set. I was also interested in learning how other businesses handled this function.

4 Likes
10

Let me begin by stating that, despite my best attempts, there are some methods of data analysis that I do not fully grasp and that call for a lot of assistance.

We presently use a BI product called Metabase, and it has really revolutionized my life.

I would argue that even while conducting the analysis may not be your task, you should be able to obtain the information needed to make wise product judgements. If you can accomplish this, you will find that you are better able to carry out your duties.

4 Likes
11

Many thanks @AnushkaGarg! I have access to the data and will have more shortly; the challenge is in analyzing it to derive conclusions and provide answers. I am able to perform certain basic tasks, but I find it difficult to provide more detailed answers regarding how customers use our product. Based on what you said, I believe I need to develop this talent to acquire the data so that I may perform my job more effectively.

3 Likes
12

I am a member of an ML/AI platform or product team. Without having to ask an analyst to run a query for me, I find it to be quite helpful to be able to rapidly identify how many current users may be impacted by a modification to a qualifying condition, etc. Having said that, some of the PMs on my team don’t know a thing about SQL, but I have a background in data science, so I may be a little biassed.

For simple queries, SQL is incredibly simple. Purchase Ben-whatever’s T-SQL book from Microsoft, enroll in a course on Udemy, or seek assistance from a member of your team or firm.

2 Likes
13

It’s important to have training in maths, statistics, and other related subjects because it plays a significant role in being a PM in mobile games.

As for what to track, start by considering the questions you hope to resolve. make sure you quality-check analytics as well.

Be cautious of population and volume discrepancies as well. For instance, we know to anticipate higher volume, lower average quality consumers if a game is a feature.

1 Like
14

YES.

Even if your company has a dedicated analytics department that creates reports for you, you still need to comprehend how the data is analyzed and how conclusions are drawn. Because SQL is so simple to learn and analysts frequently lack the business logic expertise that a project manager should have, they are able to draw diverse conclusions from the same data.

The business logic of your product is accurately represented by the SQL database schema for your company’s data, which is analogous to knowing how code functions. I frequently use this information while speaking with engineers or other stakeholders. When collaborating with other PMs, they regularly run into obstacles that I easily overcome because I know how our data is organized.

This doesn’t imply that you have to conduct all the analyses yourself, but you should at the very least be able to review the information provided by an Analyst and comprehend how they arrived at their conclusions. Although I’m sure there are many effective PMs who get their information handed to them, I think that limits most people who don’t have a strong support system.

I heartily recommend DataCamp for learning about data analytics and https://academy.vertabelo.com for studying SQL.

15

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.