Machine learning has great potential to revolutionize experimental materials research; however, the degree to which these approaches accelerate novel discovery is rarely quantified. To this end, we propose a framework for characterizing the rate of “first discovery” of scientific hypotheses in the form of materials families. We use a combination of the SuperCon and Materials Project databases to simulate a scientific needle-in-a-haystack discovery problem as a motivating example. We use this approach to compare the ability of different adaptive sampling strategies to rediscover promising superconductor families, such as the Cuprates and iron-based superconductors. This methodology can be applied using various notions of novelty, making it applicable to discovery problems more broadly.