SEER: A knapsack approach to exemplar selection for in-context hybridQA

Tonglet, Jonathan, Reusens, Manon, Borchert, Philipp and Baesens, Bart (2023) SEER: A knapsack approach to exemplar selection for in-context hybridQA. Bouamor, Houda, Pino, Juan and Bali, Kalika (eds.) In EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL). pp. 13569-13583 . (doi:10.18653/v1/2023.emnlp-main.837).

Abstract

Question answering over hybrid contexts is a complex task, which requires the combina tion of information extracted from unstructured texts and structured tables in various ways. Re cently, In-Context Learning demonstrated sig nificant performance advances for reasoning tasks. In this paradigm, a large language model performs predictions based on a small set of supporting exemplars. The performance of In-Context Learning depends heavily on the selection procedure of the supporting exem plars, particularly in the case of HybridQA where considering the diversity of reasoning chains and the large size of the hybrid con texts becomes crucial. In this work, we present Selection of ExEmplars for hybrid Reasoning (SEER), a novel method for selecting a set of exemplars that is both representative and di verse. The key novelty of SEER is that it for mulates exemplar selection as a Knapsack Inte ger Linear Program. The Knapsack framework provides the flexibility to incorporate diversity constraints that prioritize exemplars with desir able attributes, and capacity constraints that en sure that the prompt size respects the provided capacity budgets. The effectiveness of SEER is demonstrated on FinQA and TAT-QA, two real-world benchmarks for HybridQA, where it outperforms previous exemplar selection meth ods.

This record has no associated files available for download.

Contributors

Author: Bart Baesens

Download statistics