Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

5-2024

Abstract

This study presents a comprehensive evaluation of Microsoft Research’s Orca 2, a small yet potent language model, in the context of Retrieval Augmented Generation (RAG). The research involved comparing Orca 2 with other significant models such as Llama-2, GPT-3.5-Turbo, and GPT-4, particularly focusing on its application in RAG. Key metrics, included faithfulness, answer relevance, overall score, and inference speed, were assessed. Experiments conducted on high-specification PCs revealed Orca 2’s exceptional performance in generating high quality responses and its efficiency on consumer-grade GPUs, underscoring its potential for scalable RAG applications. This study highlights the pivotal role of smaller, efficient models like Orca 2 in the advancement of conversational AI and their implications for various IT infrastructures. The source codes and datasets of this paper are accessible here (https://github.com/inflaton/Evaluation-of-Orca-2-for-RAG.).

Keywords

Large Language Model (LLM), Generated Pre-trained Transformer (GPT), Retrieval Augmented Generation (RAG), Question Answering, Model Comparison

Discipline

Artificial Intelligence and Robotics | Databases and Information Systems | Numerical Analysis and Scientific Computing

Research Areas

Data Science and Engineering

Areas of Excellence

Digital transformation

Publication

Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2024 Workshops, RAFDA and IWTA, Taipei, May 7-10: Proceedings

Volume

14658

First Page

3

Last Page

19

ISBN

9789819726509

Identifier

10.1007/978-981-97-2650-9_1

Publisher

Springer

City or Country

Cham

Copyright Owner and License

Authors

Additional URL

https://doi.org/10.1007/978-981-97-2650-9_1

Download
Download Research Data

Share

COinS