We propose a class of hierarchical Gaussian process priors in which each layer of the hierarchy controls the lengthscales of the next. While this has been explored, our proposal extends previous work on the Mahalanobis distance kernel bringing an alternative construction to non-stationary RBF-style kernels. This alternative take has more desirable theoretical properties restoring one of the interpretations for input-dependent lengthscales. More specifically, we interpret our model as a GP that performs locally linear non-linear dimensionality reduction. We directly compare it with compositional deep Gaussian process, a popular model that uses successive mappings to latent spaces to alleviate the burden of choosing a kernel function. Our experiments show promising results in synthetic and empirical datasets.