๐ฅ๐ผ๐๐๐ฒ ๐๐ผ๐๐ฟ ๐พ๐๐ฒ๐ฟ๐ ๐๐ผ ๐ฎ ๐๐บ๐ฎ๐น๐น๐ฒ๐ฟ ๐๐๐ ๐๐ต๐ฒ๐ป ๐ฝ๐ผ๐๐๐ถ๐ฏ๐น๐ฒ
RouteLLM โ ๐ฐ๐๐ ๐ฑ๐ฌ% ๐ผ๐ณ ๐ฐ๐ผ๐๐ โ๏ธ
Curiosity: What insights can we retrieve from this? How does this connect to innovation in the field?
Curiosity: The LMSys team maintains the ChatbotArena, which is a great evaluation system based on thousands of matches: when a user submits a query, they receive the answers from two hidden models A and B, and vote between the two. This preference data allows them to create an ELO ranking, which is a great indicator of model strength.
The team has found another great usage of this preference data they gathered: train a router to route user queries to the most appropriate model.
The main idea is that ๐ข๐๐ฃ๐ฎ ๐ฆ๐ช๐๐ง๐๐๐จ ๐๐ค ๐ฃ๐ค๐ฉ ๐ง๐๐ฆ๐ช๐๐ง๐ ๐ ๐จ๐ฉ๐ง๐ค๐ฃ๐ ๐ข๐ค๐๐๐ก: for instance โsummarize this paragraph in 1 sentenceโ can be solved very well by a small model like Llama-3-8B, which is orders of magnitude cheaper to run than the usual behemoths. If you manage to selectively route all easy queries to the smaller LLM, you can save a lot on the costs with minimal performance reduction (a queries will be poorly answered due to mis-routing)
So the team set on to train a router that given a query, chooses the most appropriate LLM to answer it, between a strong/expensive one and a weak/cheap one.
๐ ๏ธ Create a router between GPT-4 (strong model) and Mixtral-8x7B (small model)
๐ข Use preference data from 80k labels
- โ Augment this with gold preference data for specific benchmarks
- โ Define custom metrics to measure perf gain from routing
- โ Test on MT-Bench, GSM8k, and MMLU
๐ฅ Achieve 95% of GPT-4 quality on MT-Bench for over 2x cost reduction
โจ Overhead cost are minimal, even the most expensive routing method introduces an overhead under 0.4% of GPT-4 generation
๐ง Grain of salt: MT-Bench is really the benchmark where this method performs best, and introducing โgold dataโ from the benchmark probably biased results upwards. So the โ95% perf for 2x cost reductionโ will not be as impressive in a real setting
- ๐๐๐๐ ๐ฉ๐๐ ๐ฅ๐๐ฅ๐๐ง ๐๐๐ง๐ ๐ https://huggingface.co/papers/2406.18665
- ๐พ๐ค๐๐ ๐ง๐๐ฅ๐ค ๐๐จ ๐๐๐ง๐ (already 1.7k stars) ๐ https://github.com/lm-sys/RouteLLM
