Abstract: We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show a new result that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $\Omega(\epsilon)$ rather than the classical error rate of $\mathcal{O}(\epsilon^3)$.This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations, highlighting the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks.