arXiVeri: Automatic table verification with GPT

Without accurate transcription of numerical data in scientific documents, a scientistcannot draw accurate conclusions. Unfortunately, the process of copying numericaldata from one paper to another is prone to human error. In this paper, we propose tomeet this challenge through the novel task of automatic table verification (AutoTV),in which the objective is to verify the accuracy of numerical data in tables bycross-referencing cited sources. To support this task, we propose a new benchmark,arXiVeri, which comprises tabular data drawn from open-access academic paperson arXiv. We introduce metrics to evaluate the performance of a table verifier intwo key areas: (i) table matching, which aims to identify the source table in a citeddocument that corresponds to a target table, and (ii) cell matching, which aims tolocate shared cells between a target and source table and identify their row andcolumn indices accurately. By leveraging the flexible capabilities of modern largelanguage models (LLMs), we propose simple baselines for table verification. Ourfindings highlight the complexity of this task, even for state-of-the-art LLMs likeOpenAI’s GPT-4.