Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots

LLMs increasingly serve as simulation engines for honeypots. They enable defenders to construct high-interaction honeypots with virtually no system security risks. However, LLM-powered honeypot development lacks a unified evaluation framework and protocol. Most evaluations consist of response similarity measurements against a fixed set of commands, manual testing, or online deployment. These evaluation methods are often not scalable for development, reproducible across evaluations, representative of practical attacks, and adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots in 16 backend applications, using AI hacking agents as attackers, employing two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. Using Honeyval, we conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while preserving a running cost advantage against agentic attackers. Further, we experiment with different counter-offensive configurations for the honeypots, and observe unique trade-offs, such as longer interactions at the cost of increased detection.