Measurement in the Age of LLMs: An Application to Political Ideology Scaling

Much of social science is centered around terms like “ideology” or “power”, which generally elude precise definition, and whose contextual meanings are trapped in surrounding language. This talk explores the use of large language models (LLMs) to flexibly navigate the conceptual clutter inherent to social scientific measurement tasks. We rely on LLMs’ remarkable linguistic fluency to elicit ideological scales of both legislators and text, which accord closely to established methods and our own judgement. A key aspect of our approach is that we elicit such scores directly, instructing the LLM to furnish numeric scores itself. This approach is methodologically "dumb" and shouldn't "work" according to classical principles of measurement. We nevertheless find surprisingly compelling results, which we showcase through a variety of different case studies.