Recent advancements in large language models (LLMs) have significantly enhanced natural language understanding and generation, enabling complex tasks such as text comprehension and creative text production. The efficacy of these models hinges on instruction-tuning, which relies critically on the quality of the training datasets. However, collecting high-quality and diverse data requires substantial human and time costs. In this paper, we introduce an innovative Star-Agents framework, which automates the enhancement of data quality across datasets through multi-agent collaboration and assessment. This framework adopts a three-pronged approach: it begins with generating diverse instruction data by various LLM agents through a bespoke sampling strategy. Subsequently, the generated data undergoes a rigorous evaluation using a Dual-model metric that assesses both difficulty and quality. Finaly, the above process will evolve in a dynamic refinement phase, where more effective LLMs are prioritized, enhancing the overall data quality. Our empirical studies, including instruction-tuning experiments with models such as Pythia and LLaMA, demonstrate the framework’s effectiveness. Optimized datasets have led to performance improvements, with an average increase of 12\% and notable gains in specific metrics, such as a 40\% improvement in Fermi, as evidenced by benchmarks like MT-bench, Vicuna bench, and WizardLM testset.