In this work we propose a new class of inter-domain variational Gaussian process, constructed by projecting onto a set of compactly supported B-Spline basis functions. Our model is akin to variational Fourier features. However, due to the compact support of the B-Spline basis, we produce sparse covariance matrices. This enables us to make use of sparse linear algebra to efficiently compute matrix operations. After a one-off pre-computation, we show that our method reduces both the memory requirement and the per-iteration computational complexity to linear in the number of inducing points.