Automating Enterprise Data Engineering with LLMs

Automating data engineering tasks is of great value for enterprises to increase efficiency and reduce the manual effort associated with handling large amounts of data. Recently, LLMs have shown promising results in enabling this automation.However, data engineering tasks in real-world enterprise scenarios are more complex than the way they are typically formulated in the research community. In this paper, we study the nature of challenges that arise when automating enterprise data engineering tasks with LLMs.As part of the paper, we also perform a case study on the problem of entity matching in enterprise scenarios. Furthermore, we release a hand-crafted dataset and the task based on a real enterprise scenario to enable the research community to work on such complexity of enterprise tasks.