Hidden Positives: Why Code Retrieval Benchmarks Underestimate Model Quality

Text-code embeddings underpin modern code intelligence systems, but progress in this area is held back by a systemic problem: code retrieval benchmarks routinely treat valid alternative implementations as negatives, making cross-model comparison unstable and depressing measured performance. We document this problem with a detailed analysis of six widely used benchmarks and show that, depending on the dataset, between 44\% and 87\% of items a strong LLM judge marks as relevant are missing from the original labels. To address this, we adopt a unified evaluation protocol in which a single LLM judge re-annotates the top-10 retrieved candidates for every model under a fixed prompt, replacing fragmented benchmark heuristics with a consistent semantic standard. Applying the same idea at training time, we use a smaller LLM-supervised reranker to construct MegaCode, the largest semantically curated dataset for code search to date of 193M positive (text, code) pairs, roughly an order of magnitude larger than prior curated corpora, and train a family of embedding models on it at four scales (0.5B-7B) with contrastive learning. Under a unified LLM-judged protocol, our 0.5B model matches or exceeds all prior open code-retrieval embeddings up to 8B parameters, and our 7B model improves average MRR@10 by 3.5 points over the strongest baseline.