Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

This is a Plain English Papers summary of a research paper called Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. If you like these kinds of analysis, you should subscribe to the AImodels.fyi newsletter or follow me on Twitter.

Overview

• Human evaluation of large language models (LLMs) is challenging and limited, leading to increased use of LLM-generated evaluators
• However, LLM-generated evaluators inherit the problems of the LLMs they evaluate, requiring further human validation
• The paper presents a "mixed-initiative" approach called EvalGen to help align LLM-generated evaluation functions with human requirements

Plain English Explanation

Evaluating the outputs of large language models (LLMs) like ChatGPT can be difficult and time-consuming for humans. As a result, researchers are increasingly using LLM-generated tools to help with this evaluation process. However, these LLM-generated evaluators simply inherit all the problems of the LLMs they are trying to evaluate, so they still need to be validated by humans.

The researchers in this paper present a new approach called EvalGen to help address this issue. EvalGen provides automated assistance to users in generating evaluation criteria and implementing assertions to assess LLM outputs. As EvalGen generates candidate evaluation functions (like Python code or prompts for LLMs), it also asks humans to grade a sample of the LLM outputs. This human feedback is then used to select the evaluation functions that best align with the user's requirements.

The researchers found that this approach was generally supported by users, but also highlighted the subjective and iterative nature of the alignment process. They observed a phenomenon they call "criteria drift" - users need initial criteria to grade outputs, but grading the outputs helps them refine and define their criteria further. Additionally, some evaluation criteria seem to depend on the specific LLM outputs observed, rather than being independent criteria that can be defined ahead of time.

These findings raise important questions for approaches that assume evaluation can be done independently of observing the model outputs, which is a common assumption in LLM evaluation research. The researchers present their interface, implementation details, and compare their approach to a baseline, providing insights for the design of future LLM evaluation assistants.

Technical Explanation

The paper presents a "mixed-initiative" approach called EvalGen to help align LLM-generated evaluation functions with human requirements. EvalGen provides automated assistance to users in generating evaluation criteria and implementing assertions to assess LLM outputs.

The EvalGen system works as follows:

1. Users provide initial evaluation criteria or prompts.
2. EvalGen generates candidate implementations of these criteria, such as Python functions or LLM grading prompts.
3. EvalGen asks users to grade a subset of LLM outputs using these candidate implementations.
4. EvalGen uses the human feedback to select the implementations that best align with the user's requirements.

The researchers conducted a qualitative study to evaluate EvalGen. They found overall support for the approach, but also identified several key challenges:

1. Criteria Drift: Users need initial criteria to grade outputs, but grading the outputs helps them refine and define their criteria further. This suggests the evaluation process is iterative and subjective.
2. Criteria Dependence: Some evaluation criteria appear to depend on the specific LLM outputs observed, rather than being independent criteria that can be defined a priori. This raises issues for approaches that assume evaluation can be done independently of observing model outputs.

The paper also includes a comparison of EvalGen's algorithm to a baseline approach, as well as implications for the design of future LLM evaluation assistants.

Critical Analysis

The paper highlights important challenges in the design of LLM evaluation assistants. The finding of "criteria drift" - where users refine their evaluation criteria based on observing model outputs - is a significant obstacle for approaches that assume evaluation can be done independently.

Additionally, the observation that some evaluation criteria appear to depend on the specific outputs observed, rather than being independent, is a crucial insight. This suggests that LLM evaluation may require an iterative, interactive process, rather than a one-time, fixed set of criteria.

While the paper presents a novel approach in EvalGen, the qualitative study reveals the inherent subjectivity and complexity of the evaluation process. The researchers acknowledge that further research is needed to fully understand the dynamics of aligning LLM-generated evaluations with human requirements.

One potential limitation of the study is the small sample size of the qualitative evaluation. Conducting a larger-scale user study could provide additional insights and help validate the findings.

Overall, this paper makes an important contribution by highlighting the challenges in developing effective LLM evaluation tools. The insights around criteria drift and dependence on observed outputs should inform the design of future evaluation systems, encouraging a more nuanced and iterative approach to assessing LLM capabilities.

Conclusion

This paper presents a mixed-initiative approach called EvalGen to help align LLM-generated evaluation functions with human requirements. While EvalGen was generally supported by users, the study uncovered significant challenges in the LLM evaluation process.

The key findings include the phenomenon of "criteria drift", where users refine their evaluation criteria based on observing model outputs, and the observation that some criteria appear to depend on the specific outputs observed, rather than being independent. These insights raise serious questions for approaches that assume evaluation can be done independently of model outputs.

The paper's findings have important implications for the design of future LLM evaluation assistants. Acknowledging the subjectivity and iterative nature of the evaluation process, as well as the potential dependence of criteria on observed outputs, will be crucial for developing effective tools to help humans assess the capabilities of large language models.

If you enjoyed this summary, consider subscribing to the AImodels.fyi newsletter or following me on Twitter for more AI and machine learning content.