Deep Learning for Search and Recommender Systems in Practice

System Overview Document Retrieval Scoring and Ranking . Personalization and Re-ranking
Document Retrieval • Simple regex based retrieval . Traditional inverted index based retrieval Embedding based retrieval
Metrics for Evaluation • Multiple level of relevance NDCG (Normalized Discounted Cumulative Gain) . Binary relevance DMAP (Mean Average Precision) MRR Meon Reciprocal Ronk
Normalized Discounted Cumulative Gain Discounted Cumulative Goin
Mean Average Precision Precision: Relevant documents up to rank K/K
Mean Reciprocal Rank Reciprocal Rank
Learning to Rank
Pointwise Ranking Loss function is based on a single (query, document) pair
Regression based pointwise ranking Input (4.x) feature vector responding to the query and a document, Label: y relevance of the document
Classification based pointwise ranking
Ordinal regression based pointwise ranking
Summary of pointwise ranking Pros • Simple, considering one document at a time. • Available algorithms are rich. Most regression/classification algorithms can be used.
Pairwise Ranking Loss function is based on query and a pair of documents.
Listwise Ranking Loss function is based on the query and a list of documents
AdaRank Motivation: commonly used evaluation metrics are not differentiable. So it is not easy to optimize directly. AdaRank minimizes the exponential loss. El below can be NDCG.
List Net / ListMLE Map list of scores to a probability distribution by Plockett-Luce model. • Permutation probability, where 5() is the scoring function.
Summary of listwise ranking Pros
DeText: a Deep Learning Ranking Framework