Are companies spending time with multiple solutions and experiments in the real world?

I see a lot of posts and ideas on social media about how it’s bad to just define a problem, the solution and the timeline.

What we are “supposed” to do is something like the following but I’m wondering are companies really doing this and if so how do you get buy in to do this?

  • This is the problem & why

  • These are the opportunities we have

  • These are the possible solutions

  • These are the experiments we could run to test our ideas and how we will measure the success for the experiments

  • This is the timeline for testing the experiments

11 Likes

Yes. This approach is what “product led” companies are all about.

11 Likes

I think you should:

Run your costs of not experimenting to help people see the benefits.

And don’t forget Product Management is hard.

Start with something small and test with measures and talk about the data.

11 Likes

I think of experimentation as less multiple approach and more about having a hypothesis about what effect you should see from something you are trying. When you start showing it to customers you can gauge if your hypothesis was true or not. Did you get the change you wanted?dl Did you get enough change? If not maybe you choose to roll back the change and start over. Maybe you learn something unexpected.

More science experiments and less build 3 solutions and see if they have different results.

10 Likes

Yes we do this in my company too. In fact, I just completed a “Learnings” deck outlining what we learned from experiments in Q1.

9 Likes

@VladPodpoly, Nice, did you run the experiment yourself or a dedicated experiment team? Thanks

9 Likes

@HeatherKurtz, Our team drives the experiments with the support of a data analyst for pre analysis and post analysis. Pre analysis helps us focus on the target segment and length of the experiment as per the base #, then a post analysis is run after the experiment to see if it moved a certain metric as we intended. It’s a lot of fun because you move fast.

2 learnings this Q:

  1. In the checkout, if you want customers to click a CTA like “upgrade” but they need to opt in by clicking a checkbox, place the CTA and checkbox next to eachother if you want people to complete the action. The opposite is also true
  2. In SaaS, when customers request to cancel and schedule to churn on their next renewal, it’s really really hard to win them back in this period. It’s better to prevent cancellation long before the customer even thinks of it
8 Likes

@AnaRodriguez, The last 2 are really important especially as investment is higher in the development of the full solution. The problem I often encounter is that our stakeholders (esp higher levels of management) don’t understand the need to de-scope and de-risk the initial versions of a solution, they just want the problem solved (ideally with every feature, etc.)

Experiments, rapid prototyping and MVPs are really valuable for validating the problem and validation that a certain solution path will solve the problem. And they also allow teams to pivot before sunk costs are too high.

An example right now is we have a tool that stakeholders want to integrate with another third party tool via API to facilitate a bunch of data driven decision making. But they have not proven that the the third party tool/decisions will be useful or meaningful to business impact, so instead of paying to integrate with their API we are experimenting by giving them the data in a spreadsheet so they can manually input into the tool at a far lower cost, they can validate that the tool gives them meaningful business value, and then we’ll integrate with the API after that has been proven.

7 Likes

@Nathan, So for example. If in this quarter we want to add a new section to our website because we think it is something our visitors are looking for, it’s good for SEO and it will differentiate us from our competition.

7 Likes

@Ana, Sounds like a good candidate for A/B testing but one thing I’ve seen to even further de-risk is to add a CTA to the thing you’re trying to test and just point it to a dead (404) page, like if you were a travel deals site and you wanted to see if people are interested in visiting a new page (i.e. “see all flights under $200”) and when they click it it just goes to a 404 page. Then you can use the analytics to measure clicks to infer interest in the new idea. “75% of visitors clicked this” or “5% of visitors clicked this” tell two very different stories about how interested people are.

Obviously you don’t want to leave this up for long, both for impact to SEO purposes and also to avoid losing people off your site from a broken link.

A step in between this might be a CTA that tells people you might offer this and to give their info to be notified if the product launches. For example, look at how Spotify announced “Car Thing”. They basically said “click here to be added to the list at a discount price if we decide to make this thing”.

6 Likes

Can anyone give a few examples of this?

5 Likes

@AhmadBashir, I hope you find this example related. Many candy and chocolate producers like nestle and pladis constantly create new products and some of these products go to pilot markets and their own factory stores. They follow success of the product in the pilot market. They also hire agencies to run tasting tests on random people to see if it succeeds taste wise. If the product meets the success criteria, it is launched to the whole market with new production lines; if it does not its production is ceased, team starts to work on new big thing.

4 Likes

Yes, this is a common practice for most product-led companies.

There are three big buckets of activities for PMs (I work at FAANG, so this might be different for others)

  1. Identify new opportunities from past learnings, competitive analysis, org strategy changes etc…
  2. Validate opportunities via concept testing, offline and online tests etc.
  3. Launch new features + and plan for the next sprint, which follows your typical agile product development.
3 Likes

Very cool answers. I’ve been reading up on PM techniques and strategies i have wondered if people are actually employing them. like running through frameworks, doing all the little things.

2 Likes

@MarioRomero, Yes and no. There are so many frameworks you can’t to them all. Also they are guides and ways of working, which you should ABSOLUTELY modify to meet you individual team and organizational context and culture.

Even orgs that have written the well known blog posts about their process are not applying universally everywhere in the company.

1 Like

In short: yes.

The long answer is it depends to what degree, based on a wide variety of factors (maturity of the product/space, funding amount/profitability, whether its B2C or B2B Enterprise SaaS, etc.).

For my team in particular, it’s about forming a hypothesis during discovery, identifying the one that makes the most sense for us at that time, and seeing what happens. We don’t have the capability to run a few experiments concurrently