Does anyone in here have some tips for measuring search relevancy? When making algorithmic tweaks, how do you know you’re moving the needle? Use case is knowledge management, so typically users are looking for many relevant results, not just one specific result.
Secondly, does anyone have good UX patterns for capturing user feedback from search results? Thinking more on the individual result level so that we could use it to improve search as well.
One way I’ve seen search relevancy measured without a lot of manual involvement is through a metric called mean reciprocal rank (MRR). It gives more weight to interactions with results at a higher rank.
If you have search results A, B, C ranked from highest to lowest then A has reciprocal rank 1, B is 1/2, and C is 1/3. The MRR in this case is (1/2 + 1/2 + 1)/3 or 0.61 assuming all 3 results were clicked on.
Hope this helps!
@Nathan, very helpful, thank you! I had been using a metric “precision at lowest click” that is similar to what you described, but I like MRR better as it places a different weight based on the rank of the click.
@AhmadBashir - it is basically the thinking that more relevancy leads to better page views/purchases. Page views can be influenced by factors that are not directly related to search relevancy too. For a commerce site, perhaps, filters, banners, information in product tiles… etc. So core search folks focus on measuring relevancy, coz that is the biggest driver of search conversion.
@MariaWilson - in addition to MRR, would also suggest measuring precision and recall directly.
I love the Precision at lowest at lowest click too! Very interesting metric.
@Ahmad we absolutely look at those metrics too, and it pairs well with the relevancy ones. Relevancy being how well did we do presenting the results to them. What rank did they click thru to view more about the asset? Did they download the materials? Did they save it to favorites?
If they’re doing that for results 1-5 we’re doing pretty good! If they’re doing it for results 10-20, not so much