Recent advancements in text-to-image diffusion models have enabled the personalization of such models to generate custom images from textual prompts. This paper presents an efficient LoRA personalization approach for on-device subject-driven generation, where pre-trained diffusion models are fine-tuned with user-specific data on resource-constrained devices. Our method, termed the Hollowed Net, significantly enhances the memory efficiency in fine-tuning stage by modifying the architecture of a diffusion U-Net such that non-essential deep layers are removed-creating a hollowed structure. We directly address the on-device memory constraints, and our Hollowed Net is beneficial for considerably reducing the training GPU memory, while previous methods mostly focus on minimizing training steps and the parameters to update. Additionally, our personalized Hollowed Net can be transferred back into the original U-Net, enabling the inference without additional memory requirements. Quantitative and qualitative analyses demonstrate that our approach not only reduces the training memory to the level as low as that required for inference, but also maintains or improves the personalization performance compared to existing methods.