CoDyn: Dynamic LLM Routing for Coding Tasks

Large language models (LLMs) have become integral tools in various business applications, including software engineering, due to their ability to process and generate text. However, the diverse landscape of LLMs, encompassing both proprietary and open-source models with varying architectures and training methodologies, presents opportunities for optimizing cost-performance through model routing. Model routing is a meta-learning paradigm that dynamically selects the optimal model based on user prompts and preferences, leveraging the strengths of different LLMs for specific tasks. Although model routing has gained popularity in natural language tasks, its potential has not been extensively explored for software engineering tasks. In this study, we investigate the dynamic routing capability for various code-based tasks. Initially, we select five models and assess their effectiveness across five coding-related tasks. To create the router, we fine-tune low-cost LLMs using three distinct fine-tuning techniques. We then evaluate these techniques based on three research questions. Our experimental results demonstrate that LLM Classifier-based router consistently match or surpass of the effectiveness of the strongest model while offering 43% average cost savings and predictably scaling with varying the cost weight hyperparameter to achieve even greater savings for a moderate degradation in task effectiveness.