Hello Everyone,
Does anyone in here have some tips for measuring search relevancy? When making algorithmic tweaks, how do you know you’re moving the needle? Use case is knowledge management, so typically users are looking for many relevant results, not just one specific result.
Secondly, does anyone have good UX patterns for capturing user feedback from search results? Thinking more on the individual result level so that we could use it to improve search as well.
One way I’ve seen search relevancy measured without a lot of manual involvement is through a metric called mean reciprocal rank (MRR). It gives more weight to interactions with results at a higher rank.
If you have search results A, B, C ranked from highest to lowest then A has reciprocal rank 1, B is 1/2, and C is 1/3. The MRR in this case is (1/2 + 1/2 + 1)/3 or 0.61 assuming all 3 results were clicked on.
Hope this helps!
@Nathan, very helpful, thank you! I had been using a metric “precision at lowest click” that is similar to what you described, but I like MRR better as it places a different weight based on the rank of the click.
This is awesome - search is one of those specialty domains that is hard to get right - both for product and engineering. Also so many services have it but it’s rare to measure it’s efficacy.
This is fascinating, since it’s not an area I know much about.
Why looking at direct measures of relevancy instead of impacts? Like, pages visited, purchases made, etc.?
@AhmadBashir - it is basically the thinking that more relevancy leads to better page views/purchases. Page views can be influenced by factors that are not directly related to search relevancy too. For a commerce site, perhaps, filters, banners, information in product tiles… etc. So core search folks focus on measuring relevancy, coz that is the biggest driver of search conversion.
@MariaWilson - in addition to MRR, would also suggest measuring precision and recall directly.
I love the Precision at lowest at lowest click too! Very interesting metric.
@Ahmad we absolutely look at those metrics too, and it pairs well with the relevancy ones. Relevancy being how well did we do presenting the results to them. What rank did they click thru to view more about the asset? Did they download the materials? Did they save it to favorites?
If they’re doing that for results 1-5 we’re doing pretty good! If they’re doing it for results 10-20, not so much
Thanks @Nathan.
Thanks @Maria
Echoing @Ahmad, this is fascinating, and it has my gears spinning now on how I might start to think about product recommendations differently.
@Nathan (or anyone else) can you recommend and reading/articles on the topic?
@JesusRojas Here’s one that I like
@Jesus, specific to recommendations, I stumbled on this yesterday. It was an awesome read
Thank you, both of you
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.