Abstract:

Pinterest’s mission is to give everyone the inspiration to create a life that they love.  We achieve this by generating highly personalization recommendations along users’ journey on Pinterest.  Over the years, Pinterest users also curated one of the largest graph-structured data on the Internet.  In this talk, we will walk through the unique challenges in the recommendation system at Pinterest, and how we leverage this graph to make some latest development in these real-world applications. We will also showcase how we built the system that has been deployed in production and has delivered significantly better user experience across organic and Ads feeds. 

Abstract:
Operationalizing AI fairness at LinkedIn’s scale is challenging not only because there are multiple mutually incompatible definitions of fairness but also because determining what is fair depends on the specifics and context of the product where AI is deployed. Moreover, AI practitioners need clarity on what fairness expectations need to be addressed at the AI level. In this work, I'll present the evolving AI fairness framework used at LinkedIn to address these three challenges. The framework disentangles AI fairness by separating out equal treatment and equitable product expectations. Rather than imposing a trade-off between these two commonly opposing interpretations of fairness, the framework provides clear guidelines for operationalizing equal AI treatment complemented with a product equity strategy. I'll present the principles we set forth and cover a real-world case study walking us through how we measure and mitigate for algorithmic bias. 

Abstract:

Content creators compete for exposure on recommendation platforms, and such strategic behavior leads to a dynamic shift over the content distribution. However, how the creators' competition impacts user welfare and how the relevance-driven recommendation influences the dynamics in the long run are still largely unknown.

Our recent work provides theoretical insights into these research questions. We model the creators' competition under the assumptions that: 1) the platform employs an innocuous top-K recommendation policy; 2) user decisions follow the Random Utility model; 3) content creators compete for user engagement and, without knowing their utility function in hindsight, apply arbitrary no-regret learning algorithms to update their strategies. We study the user welfare guarantee through the lens of Price of Anarchy and show that the fraction of user welfare loss due to creator competition is always upper bounded by a small constant depending on K and randomness in user decisions; we also prove the tightness of this bound. Our result discloses an intrinsic merit of the myopic approach to the recommendation, i.e., relevance-driven matching performs reasonably well in the long run, as long as users' decisions involve randomness and the platform provides reasonably many alternatives to its users.

Abstract:

Large language models (LLMs) have shown unmatched capacities in language understanding and generation, prompting surges of interest in its adoption for recommendations. In this talk, I will describe our long quest to pinpoint its ability in natural language understanding and generation to facilitate communication between recommendation agents and users, building trust and enabling user agency on recommendation platforms.  I will discuss two lines of work 1) user study on how users rely on recommendation systems to pursue their interest journeys and different techniques to prompt LLMs to provide nuanced and interesting user journey descriptions to assist users in their real life journeys; 2) prompting LLMs to provide different formats of explanations on why a recommendation is made, and improve recommendation with the extracted explanations. The two combined can offer users a glimpse into the recommender agents' competence to build trust, as well as enable users to navigate and control their recommendations. I will discuss research findings discovered along these endeavors and brainstorm challenges and opportunities in integrating LLMs into user-facing products. 

Abstract:

At Shopify, we optimize product discovery for millions of merchants’ online stores, through product recommendations and search. It’s a challenge to provide good recommendations to all of these independent and diverse businesses; a given model might lead to an overall improvement in metrics on aggregate, but degrade the performance for some stores. Moreover, since each store is unique and has a different customer base, design, product inventory, and data quality, we must take care when running experiments to evaluate the impact of our models. In this talk, we describe the unique challenges involved in improving product discovery for millions of merchants and their customers, and the approaches we employ for model development, offline evaluation, and experimentation.

 Abstract:

Large scale deep learning recommender system models, which are used for predicting top-k items given a query (typically a user identifier), learn the user to item propensities from large amounts of training data containing many (user, item, time) tuples. Unlike the simpler bi-linear models such as Matrix Factorization, deep learning recommender system models almost always do not model users explicitly bur rather model them via the items they have consumed in the past, with or without temporal ordering information. This allows the model to extrapolate the preference matching on many unseen users as long as the items these unseen users have interacted with are seen during training. However, pure item identifier based models are often incapable in assigning faithful propensities to novel items not previously seen during training. This is the classical item cold start problem in recommender system. Item content features (e.g. item description, movie synopsis, product image, etc) become useful in such cold start scenarios as they become proxy for the item identifier. In this talk, we investigate a rather extreme cold start situation wherein the entire application is treated as cold start. Imagine two related businesses whose product inventory is similar in content type but not in identity. Further assume that one of them lacks sufficient user-item interactions limiting us from building a personalization model. Question we want to ask is whether we can obtain domain agnostic information from the data rich domain — source domain, and transfer it to the data poor domain — target domain ? To this end, in this talk, we present Zero Shot Transfer Learning model that transfers such learned task-agnostic-knowledge from the source to the target domain. We conduct offline experiments on large scale real world datasets and present the results and findings. Time permitting, we will also talk about a novel concept we first introduced back in 2021 — Language Models as Recommender Systems, where we show how we can leverage large pre-trained language models and prompt engineering to generate recommendations in a zero shot setting. 

Abstract:

Recommender systems seek to answer a policy optimization problem: given data collected from the previously-deployed recommendation policy, our goal is to come up with a new policy that can achieve better outcome when deployed online. However, the optimization for a better policy is counterfactual in nature. The conventional way to address this challenge is through importance sampling correction, which comes with its own practical limitations. In this presentation, we suggest an alternative approach of local policy improvement and present an objective function that is simple to estimate from data without off-policy correction. We discuss the tradeoff between this approach and importance sampling correction and offer some practical arguments for favoring one over the other.

Abstract:

At Netflix, personalization is a top priority in the constantly evolving entertainment industry, and we strive to use our insights from a member’s streaming data to enhance their viewing experience. To achieve this, we employ a two-stage process, beginning with a Foundation Model to understand users' long-term preferences. We then use precision-based adaptation to fine-tune it, ensuring that the recommendations align with our subscribers' preferences and needs. 

The Foundation Model is specifically designed to capture users' long-term preferences and provide accurate representations of Netflix's content. We use large-scale self-supervised learning to analyze users' interaction histories and gain insight into their interests and the evolution of their tastes over time. In this talk, we will explore the similarities and differences between our Foundation Model and large language models, and discuss the challenges we have overcome, such as addressing the cold-start problem and mitigating popularity bias.

We will also discuss how the Foundation Model is utilized by the downstream canvas-serving page construction model. This is achieved by providing embeddings or performing fine-tuning based on business objectives. To optimize the personalized page, we take into account user-specific feedback and contextual features such as time of day, device type, and in-session user signals. Our ultimate goal is to create a personalized page that strikes a perfect balance between factors including user preferences, diversity, and novelty. We achieve this through our page optimization model, which employs user-specific data and contextual information to generate highly personalized recommendations. We continually refine our model by training it with context-specific data to ensure that our recommendations remain current and relevant.