Differentiable digital signal processing (DDSP) allows us to constrain the outputs of a neural network to those of a known class of signal processor. This can help us train with limited data, reduce audio artefacts, infer parameters of signal models, and expose human interpretable controls. However, numerous failure modes still exist for certain important families of signal processor. This talk illustrates two such challenges, frequency parameter non-convexity and permutation symmetry, and introduces promising approaches to solving them.