The emergence of Large Language Models (LLMs) has necessitated the formulation of training methodologies that curtail the generation of unethical language and effectively handle toxic user queries. Addressing the prevailing challenges associated with human labor constraints and data paucity, we introduce KoTox, encompassing 39K unethical instructions. This study utilizes a novel approach to automatic data generation on toxic instructions, fostering data efficiency in training LLMs. Our investigation addresses the issue of data scarcity by offering an efficient means of constructing an instruction dataset and further encourages more secure and ethical interactions in Natural Language Processing (NLP) applications.