Detecting dependencies among variables is a fundamental task across all scientific disciplines. In this work, we propose a novel neural network model for supervised graph structure inference, which aims to learn a mapping between observational data and their underlying dependence structure. The model is trained with variably shaped and coupled simulated input data and requires only a single forward pass through the trained network for inference. We introduce a novel bilinear attention mechanism (BAM) for explicit processing of dependency information, which operates on the level of covariance matrices of transformed data and respects the geometry of the manifold of symmetric positive definite matrices. Empirical evaluation demonstrates the robustness of our method in detecting a wide range of dependencies, excelling in undirected graph estimation and proving competitive in completed partially directed acyclic graph estimation through a novel two-step approach. The trained model demonstrates the ability to detect the presence of a causal relationships, regardless of their specific parameterizations.