We consider the problem of combining a symbolic planner and a Deep RL agent to achieve the best of both worlds -- the generalization ability of the planner with the effective learning ability of Deep RL. To this effect, we extend a previous work of Kokel et al. ICAPS 2021, RePReL, to Deep RL. As we demonstrate in experiments in two relational worlds, this combination enables effective learning, transfer and generalization when compared to the use of only Deep RL.