Automating dataset updates towards reliable and timely evaluation of Large Language Models
Publication Type
Conference Proceeding Article
Publication Date
12-2024
Abstract
Large language models (LLMs) have achieved impressive performance across various natural language benchmarks, prompting a continual need to curate more difficult datasets for larger LLMs, which is costly and time-consuming. In this paper, we propose to automate dataset updating and provide systematic analysis regarding its effectiveness in dealing with benchmark leakage issue, difficulty control, and stability. Thus, once the current benchmark has been mastered or leaked, we can update it for timely and reliable evaluation. There are two updating strategies: 1) mimicking strategy to generate similar samples based on original data, preserving stylistic and contextual essence, and 2) extending strategy that further expands existing samples at varying cognitive levels by adapting Bloom's taxonomy of educational objectives. Extensive experiments on updated MMLU and BIG-Bench demonstrate the stability of the proposed strategies and find that the mimicking strategy can effectively alleviate issues of overestimation from benchmark leakage. In cases where the efficient mimicking strategy fails, our extending strategy still shows promising results. Additionally, by controlling the difficulty, we can better discern the models' performance and enable fine-grained analysis neither too difficult nor too easy an exam can fairly judge students' learning status. To the best of our knowledge, we are the first to automate updating benchmarks for reliable and timely evaluation.
Keywords
Large language models, LLM, Dataset update, Benchmark update, Automation
Discipline
Artificial Intelligence and Robotics
Research Areas
Intelligent Systems and Optimization; Data Science and Engineering
Publication
Conference on Neural Information Processing Systems, NeurIPS 2024 Track Datasets and Benchmarks
Identifier
doi.org/10.48550/arXiv.2402.11894
Publisher
Conference on Neural Information Processing Systems, NeurIPS
City or Country
Vancouver
Citation
YING, Jiahao; CAO, Yixin; BAI, Yushi; SUN, Qianru; WANG, Bo; TANG, Wei; DING, Zhaojun; YANG, Yizhe; HUANG, Xuanjing; and YAN, Shuicheng.
Automating dataset updates towards reliable and timely evaluation of Large Language Models. (2024). Conference on Neural Information Processing Systems, NeurIPS 2024 Track Datasets and Benchmarks.
Available at: https://ink.library.smu.edu.sg/sis_research/9439
Additional URL
https://doi.org/10.48550/arXiv.2402.11894