Selective Preference Aggregation

Many tasks in Machine Learning are shaped by procedures where items are ordered based on the preferences of a group—such as funding proposals, recommending products, or improving the helpfulness of responses from a large language model. In such settings, individuals express their preferences over items as votes, ratings, and rankings. Given a dataset of individual preferences, preference aggregation methods rank the items in a way that summarizes their collective preferences as a group. Standard methods for preference aggregation are designed to arbitrate dissent. When individuals express conflicting preferences between items, methods will rank one item over another—resolving disagreement based on axioms of social choice. In this work, we introduce a paradigm for Selective Aggregation in which we abstain rather than arbitrate dissent. Given a dataset of ordinal preferences from a group of judges, we aggregate their preferences into a Selective Ranking—i.e., a partial order over items where every comparison is aligned with $1-\tau$\% of judges. We handle missing data effectively, ensuring robust performance across diverse settings. We develop an algorithm to construct selective rankings that achieve all possible trade-offs between comparability and disagreement.