Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

12-2023

Abstract

The use of modern Large Language Models (LLMs) as chatbots still has some problems such as hallucinations and lack of empathy. Identifying these issues can help improve chatbot performance. The community has been continually iterating on reference-free dialogue evaluation methods based on large language models (LLMs) that can be readily applied. However, many of these LLM-based metrics require selecting specific datasets and developing specialized training tasks for different evaluation dimensions (e.g., coherence, informative). The developing step can be time-consuming and may need to be repeated for new evaluation dimensions. To enable efficient and flexible adaptation to diverse needs of dialogue evaluation, we propose a dimension-agnostic scoring method that leverages the in-context learning (ICL) capability of LLMs to learn from human scoring to the fullest extent. Our method has three key features. To begin with, rather than manual prompt crafting, we propose automatically generating prompts, allowing the LLM to observe human labels and summarize the most suitable prompt. Additionally, since the LLM has a token limit and ICL is sensitive to demonstration variations, we train a selector to finely customize demonstrations and prompts for each dialogue input. Finally, during inference, we propose to request the LLM multiple times with a subgraph of demonstrations and prompts that are diverse and suitable to maximize ICL from various human scoring. We validate the efficacy of our method on five datasets, even with a small amount of annotated data, our method outperforms all strong baselines. Code is available at https://github.com/iamlxb3/EMNLP2023-ADOROR.

Keywords

Chatbots, Context learning, Dialogue evaluation, Evaluation methods, In contexts, Language model

Discipline

Artificial Intelligence and Robotics | Numerical Analysis and Scientific Computing

Publication

2023 Conference on Empirical Methods in Natural Language Processing: Singapore, December 6-10: Proceedings

First Page

9472

Last Page

9496

ISBN

9798891760608

Identifier

10.18653/v1/2023.emnlp-main.590

Publisher

Association for Computational Linguistics (ACL)

City or Country

Stroudsburg, PA

Additional URL

https://doi.org/10.18653/v1/2023.emnlp-main.590

Download

Share

COinS