We will overview the competition, the 4 tracks, results and introduce the winners of each track.

The rise of metric embeddings as a crucial tool in search, recommendation, and large language model applications has created significant interest in complex search queries over vectors, such as restricted vector search based on per-vector metadata ("filtered ANNS"). The NeurIPS'23 BigANN competition track for filtered ANNS evaluated submissions based on query throughput above a target level of recall on a 10M vector dataset, with binary per-vector metadata (labels), and with query predicates requiring either one or two specified labels to be present for all vectors returned. Existing state of the art approaches for filtered ANNS struggle to perform such 'AND' queries, which require the returned vectors to have all of a set of specified binary labels. Perhaps surprisingly, we find that a more combinatorial view of the problem leads to highly efficient solutions, matching or exceeding the throughput of unfiltered search on the full dataset. We present a novel approach to indexing these queries which leverages classical and inverted file indices in tandem to dramatically reduce the number of vectors needing to be considered before comparing any of them to the query vector. We demonstrate empirically strong results on the competition dataset (11.5x speedup over the baseline), which we believe can serve as a practical solution to the filtered ANNS problem, and as a strong baseline for future work.

Ben Landrum University of Maryland Magdalen Dobson Manohar CMU Mazin Karjikar University of Maryland Laxman Dhulipala University of Maryland

Approximate Nearest Neighbor Search (ANNS) is a fundamental and critical component in many applications, including recommendation systems and large language model-based applications. Cross-modal ANNS, which embeds data from different modalities into one high-dimensional space as feature vectors, uses the data from one modality to query another. However, there is an inherent distribution gap between embeddings from different modalities. Queries in cross-modal ANNS become Out-of-Distribution (OOD) to the base data, and state-of-the-art ANNS approaches suffer low performance for the workloads.

We quantitatively analyze the properties of the cross-modal datasets to gain an understanding of the ANNS efficiency. Unlike single-modal workloads, we find OOD queries are distant from the base data, and the k-nearest neighbors of an OOD query are distant in the embedding space. The property breaks the assumptions of existing ANNS approaches and mismatches their design for efficient search. With the insights from the OOD workloads, we propose pROjected bipARtite graph for ANNS (RoarANN), an ANNS approach that builds a graph index under the guidance of query distribution. Extensive experiments show that RoarANN significantly outperforms state-of-the-art approaches on modern cross-modal datasets, achieving up to 3.3X faster search speed at a 90% recall rate for OOD queries. Besides, RoarANN also demonstrates robust performance for in-distribution queries.

Authors: Meng Chen, Fudan University, Yue Chen, Fudan University, Rui Ma, Fudan University, Kai Zhang, Fudan University, Yuzheng Cai, Fudan University, Jiayang Shi, fudan university, Yizhuo Chen, Fudan University, Weiguo Zheng, Fudan University

In this talk, we delve into the groundbreaking vector search framework, PyANNs, highlighting its four key features and main contributions to the field. Firstly, PyANNs unifies the search process for diverse vector types, seamlessly handling both dense and sparse vectors, float and integer vectors. Secondly, its adaptability extends to different graph types, unifying the search process for multi-layered graphs like HNSW and single-layered graphs like Vamana. Thirdly, PyANNs achieves unprecedented computation speeds through the utilization of various vector instruction sets, establishing itself as the pinnacle of performance. Lastly, the framework incorporates an adaptive computation and memory optimization process, ensuring maximum vector computation bandwidth across diverse machines.

Puck proposes a multi-level index structure towards the ever growing datasets in practice. Two key methods, dynamic trimming and unifying PQ-table, are designed to accelerate training and search of multi-level quantization-based index. While maintaining a recall ratio >95% towards the SIFT10M dataset, a four-level quantization-based index achieves significantly better search throughput than Faiss-ivf and HNSW. The repository in GitHub is https://github.com/baidu/puck .

Authors： Jie Yin, Baidu Ben Huang, Baidu

Artificial intelligence, which often depends on general-purpose GPU (GPGPU) computing, continues to disrupt nearly every industry–from automobiles and healthcare to finance and the arts. Nvidia offers accelerated algorithms for vector search via RAPIDS RAFT, an open source library for applications and library developers running on GPUs. In this talk, I present the benefits of using GPU-accelerated vector search algorithms, along with some of the challenges we overcame while building them. I will also show how adopting RAFT helps developers accelerate other key components in their AI workflows.

In this talk, I am going to discuss problems and research regarding Approximate Nearest Neighbor Search in the Recommender Systems. In particular: • The role of fast Approximate Nearest Neighbor search in the multi-stage funnel design or a typical Recommender System. • Research on Approximate Nearest Neighbor search with neural ranking distances and its transformative effects on the end-to-end funnel design. • Existing work on compression-friendly embeddings and parameter tuning.

We will summarize other entries not presented in the session and will open the floor for discussion on the future of this benchmark.