Anyone here working in a team that delivers results coming from ML or AI models. As a PM is there any expectation on understanding how these models work to maybe challenge the DS devs on any metrics that may seem off from the results you expect.
Trying to understand to what level is it required to be hands deep in data science. And how would the role or responsibilities differ for someone who is a Technical Program Manager.
I’m an ML PM in a large tech company. I probably had to know more about the models etc for the interview process than for my day to day.
I’d say the most important part I play in the model development process is in two areas.
1- defining how the feature will be used and how it can solve a problem. This means having a clear end to end picture of how we intend to use the model once it’s built. Just like any other feature if the teams aren’t aligned you end up building something that doesn’t fit right.
2- understanding the training data you have and what you don’t have. The eng team often doesn’t have access to marketing and user interviews at the level you do. So you need to bring that information to the team. It’s unlikely you’ll have perfect training data in any scenario. So you’ll have to collaborate on the approach.
Being able to code is helpful and having a strong understanding of the importance of a data feature in a model can be handy. But you should also be able to rely on your team to discuss that with you.
@Felipe, I’m curious, as an ML PM do you ever find yourself being pressured to find ML solutions for problems that ML is ill-suited for? Are there ever times that you’re idle because you don’t have ML problems to solve?
@Maria, Yes. Absolutely. I think most often what I’ve seen is that executive leaders want some innovative solution to a problem but they haven’t thought through all the dependencies.
At my company if it’s a very abstract can the tech even do this ask id probably get it out of my scrum team and take it to the research org. Let them work on it for a year or two and then let me know how well the tech can do it. Obviously this means it needs to be compelling from a technology and research perspective but that can be a pretty low hurdle especially if they can give to a PhD intern over the summer.
I will also say that my engineering manager is awesome about saying can we just do this more simply with something else
@Felipe, Would you say your role is more of a technical product manager? Some of the Joh postings for TPM in data science have master degrees in data science degrees. So maybe that whole vertical is another level with the technical PM?
@Carlos, I doubt they really care if you have a masters in ds. If you browse my history you’ll see I’m an mba with a masters in security but I did a ds certificate by adding like 2 classes. The certificate was probably enough paired with a masters degree or experience in general.
But yes I’m more of a general tpm and right now I lead an ML team.
@Felipe, Thanks for sharing your exp. I’d classify myself as an AI /ML enthusiast and hope to switch to a ML product one day, your replies were great to read.
@Ahmad, I think for me it was important to switch into an ML team within a larger org focused on a consumer product. I did interview with a lot of companies focused on ds and ml solution in b2b. They were a lot pickier about education
@Felipe, What certification did you do?
Curios as I have looking into some, and I going 360datascience as a good one so far.
@Natalie, I didn’t do a public cert. I added one into my masters degree since my school also offered a masters of data science.
@MariaWilson, This. I am also Data/AI/ML PM (or rather PO… as I do almost no discovery work).
What one needs to understand is that ML models just take some input and provide some output. Oftentimes one can exchange that black box model with something simpler like a logic based heuristic. What a PM/PO really needs to make sure that the model is embedded in a suitable way in a bigger application. I often find myself aligning logics and data around the model itself with business to ensure the solution solves their problems and pain points.
I have a PhD in applied ML working on automated Driving - that helps a lot but I don’t think it’s a prerequisite for the PM role.
I can’t speak for more data science based products but my team uses AI/ML models when necessary. An example:
A PM on my team works with delivery routing. They looked into the routes that were being taken and noticed that vehicles from one start point were being sent to destinations near other start points, basically route overlap. This was causing some inefficiencies and impacting an important metric so the PM asked the data team to look into the cause. The data team updated the model to improve routing and we’re measuring the impact of it now.
I think the answer depends, of course 
At a high level, yes, you should understand them a bit. Even just a passing familiarity with various models (classical and deep learning) will give you the vocabulary necessary to talk about them with your teams and stakeholders. And if you know even more, like if a given approach is a good fit for the product and the data being fed into the model, all the better.
But you certainly shouldn’t concern yourself with the data management and munging to get it into the right shape so that it can fit a model, or the details of how it is integrated and deployed to the wild. Does it hurt to know these things? Nope! But it’s well beyond what should be expected of you.
Abstracting out a bit, a PM should understand their product’s domain. If that means dipping their toes into ML a little bit, great!
You should understand whether the market want a black box model or not. If the market want transparency you need to make the ML model results explainable. Hence, you need to understand the trade off between different models in detail. This is something 90 % of products seems to fail with, either because their product is a bunch of marketing nonsense or they are doing something against the interest of the users themselves.
The more you can do the better. I am in startups, so we don’t have rigid roles and teams. I love jamming with the devs on AI products in working sessions. Really helps to get customer feedback across. Our latest NLP model we built together, just going back and forth. Since I know how we built it, I can also shield engineering from the stupid requests/suggestions/timelines.
Could I code the models myself? Maybe but it’d take a really long time. But I have a good enough understanding of the architecture/technologies that I can speed up development and help find shortcuts/scope cut. But I didn’t learn by studying machine learning, I learned by hanging out with the devs.
To be good you don’t need to go deep, but you will if you want to be great.
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.