Effect-Level Validation for Causal Discovery in Interactive Telemetry

Causal discovery is increasingly used to analyze telemetry from interactive systems, but a plausible graph does not guarantee an identifiable or reliable product decision. We study this problem for one query: whether early competitive gameplay increases Day-1 retention in a deployed Role-playing game (RPG). We propose an admissibility-first framework that treats discovered graphs as structural hypotheses, keeps only graphs that identify the target effect and have adequate treatment-control overlap, and validates admissible effects through cross-algorithm stability, placebo, subsampling, and E-value diagnostics. On real telemetry, only about one third of discovery runs support the target effect after temporal and semantic constraints. Some admissible Directed Acyclic Graphs (DAGs) converge to a positive risk-difference average treatment effect (ATE), down from a raw retention gap, and the estimate survives all refutation checks. Graph-level recovery is therefore an inadequate proxy for causal reliability; discovery pipelines for decision support should be evaluated at the query and effect level.