Publication Type
Conference Proceeding Article
Version
publishedVersion
Publication Date
7-2023
Abstract
Large language models (LLMs) have recently been shown to deliver impressive performance in various NLP tasks. To tackle multi-step reasoning tasks, few-shot chain-of-thought (CoT) prompting includes a few manually crafted step-by-step reasoning demonstrations which enable LLMs to explicitly generate reasoning steps and improve their reasoning task accuracy. To eliminate the manual effort, Zeroshot-CoT concatenates the target problem statement with “Let’s think step by step” as an input prompt to LLMs. Despite the success of Zero-shot-CoT, it still suffers from three pitfalls: calculation errors, missing-step errors, and semantic misunderstanding errors. To address the missing-step errors, we propose Planand-Solve (PS) Prompting. It consists of two components: first, devising a plan to divide the entire task into smaller subtasks, and then carrying out the subtasks according to the plan. To address the calculation errors and improve the quality of generated reasoning steps, we extend PS prompting with more detailed instructions and derive PS+ prompting. We evaluate our proposed prompting strategy on ten datasets across three reasoning problems. The experimental results over GPT-3 show that our proposed zero-shot prompting consistently outperforms Zero-shot-CoT across all datasets by a large margin, is comparable to or exceeds Zero-shot-Program-of-Thought Prompting, and has comparable performance with 8-shot CoT prompting on the math reasoning problem. The code can be found at https://github.com/AGIEdgerunners/Plan-and-Solve-Prompting.
Keywords
Calculation error, Language model, Large margins, Multisteps, Performance, Reasoning problems
Discipline
Artificial Intelligence and Robotics | Databases and Information Systems | Programming Languages and Compilers
Research Areas
Data Science and Engineering
Publication
61st Annual Meeting of the Association for Computational Linguistics, ACL 2023: Toronto, July 9-14: Proceedings
First Page
2609
Last Page
2634
ISBN
9781959429722
Identifier
10.18653/v1/2023.acl-long.147
Publisher
ACL
City or Country
Toronto
Citation
WANG, Lei; XU, Wanyu; LAN, Yihuai; HU, Zhiqiang; LAN, Yunshi; LEE, Roy Ka-Wei; and LIM, Ee-peng.
Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. (2023). 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023: Toronto, July 9-14: Proceedings. 2609-2634.
Available at: https://ink.library.smu.edu.sg/sis_research/8054
Copyright Owner and License
Publisher
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.18653/v1/2023.acl-long.147
Included in
Artificial Intelligence and Robotics Commons, Databases and Information Systems Commons, Programming Languages and Compilers Commons