MT-Bench: Comparing different LLM Judges

By default, MT-Bench uses OpenAI as a service provider with a gpt-4 model ID, which is a vanilla GPT-4 model with 8k context introduced back in Spring 2023. However, it is possible to override the model ID via the --judge-model argument.

As of June 2024, GPT-4 series models have the following pricing (per million tokens):

By running the MT-Bench using GPT-4 Turbo or Omni one can potentially save 6 times on API calls for one evaluation. But how will the score change? Let's find out :)

Costs

I have used Phi-3 Medium with 8K context and quantized at 8-bits (running inference server via LM Studio). I have executed answer generation 4 times. Then for each of the sets, I have run one judgment generation with the three models.

OpenAI API consumption cost per one eval*:

*I only collected the total tokens consumed by gpt-4-0613 (621805) during 4 runs. For the calculation, I assumed that each model had similar token consumption with 580k prompt and 60k completion tokens

Reviewing the Scores

The below findings can not be generalized as they take a small sample of results for just one target model (Phi-3). Still...

For each of the LLM judges, I have calculated the mean (out of 4 runs) and standard deviation as a percentage of the mean. As you can see:

• Omni tends to inflate the score by a factor of 12%
• All models are quite consistent with just a 1-3% deviation in scores
• The vanilla GPT-4 shows the most consistency across turns

GPT-4 Turbo is the closest to the baseline of GPT-4 8K, 2nd turn sees the most deviation:

Both Omni and Turbo see the least drop in 2nd turn scores:

Raw Scores

About

MT-Bench is a quick (and dirty?) way to evaluate a chatbot model (fine-tuned instruction following LLM). When a new open-source model is published at Hugging-face it is not uncommon to see the score presented as a testament of quality. It offers ~$5 worth of OpenAI API calls towards getting a good ballpark of how your model does. A good tool to iterate on fine-tuning an assistant model.

MT-Bench is a Python program that asks the target model 80 predefined questions (doing inference via HF Transformers or OpenAI compatible API endpoint). The questions cover Humanities, STEM, Extraction, Roleplay, Writing, Reasoning, and Coding. There are 2 turns - it asks a question and gets the answer (1st turn), then adds a follow-up question and collects the 2nd answer (2nd turn). It then iterates through all questions and asks the GPT-4 model (the legacy 8K model from Spring 2023) to score both answers on a scale from 1 to 10 (hence the lowest a model can get is 1, not 0 :). The results are 3 aggregate scores: 1st turn, 2nd turn, and average score.

########## First turn ##########
                                        score
model                          turn
stablelm-2-brief-1_6b_2        1     3.240506

########## Second turn ##########
                                        score
model                          turn
stablelm-2-brief-1_6b_3        2     2.443038


########## Average ##########
                                   score
model
stablelm-2-brief-1_6b_3         2.822785

As explained in this paper, which introduced the MT-Bench and investigated the utility of LLM as an evaluator, the score shows high agreement with human preferences. I.e. the larger the MT-Bench score the higher the model gets on LMSYS Chatbot Arena.

Another popular option for LLM evaluation is AlpacaEval. This one uses a newer and cheaper GPT-4 Turbo model as a baseline. The authors of AlpacaEval provided correlation coefficients of different evals with LMSYS Arena showing a strong association between LLM judges' scores and human preferences at the Arena: