Enhancing Syllogistic Reasoning in Biomedical NLI: Key Insights and Challenges

Written by largemodels | Published 2024/12/10
Tech Story Tags: natural-language-inference | syllobio-nli | syllogistic-reasoning | biomedical-ontologies | syllogistic-schemes | zero-shot-learning-(zs) | llms-in-biomedicine | ai-evaluation-frameworks

TLDR The study introduces SylloBio-NLI to assess LLMs' syllogistic reasoning in biomedical contexts, finding that while few-shot techniques show promise, current models lack the robustness needed for safe biomedical applications.via the TL;DR App

Table of Links

  1. Abstract and Introduction
  2. SylloBio-NLI
  3. Empirical Evaluation
  4. Related Work
  5. Conclusions
  6. Limitations and References

A. Formalization of the SylloBio-NLI Resource Generation Process

B. Formalization of Tasks 1 and 2

C. Dictionary of gene and pathway membership

D. Domain-specific pipeline for creating NL instances and E Accessing LLMs

F. Experimental Details

G. Evaluation Metrics

H. Prompting LLMs - Zero-shot prompts

I. Prompting LLMs - Few-shot prompts

J. Results: Misaligned Instruction-Response

K. Results: Ambiguous Impact of Distractors on Reasoning

L. Results: Models Prioritize Contextual Knowledge Over Background Knowledge

M Supplementary Figures and N Supplementary Tables

5 Conclusions

In this work, we proposed a novel methodological framework, SylloBio-NLI, designed to evaluate the syllogistic reasoning capabilities of state-of-the-art LLMs within the biomedical domain. Through comprehensive analysis across 28 syllogistic schemes, we assessed the performance of eight different models under varying conditions, including zero-shot and few-shot settings. Our results show that both techniques exhibit high sensitivity to superficial lexical variations, highlighting a dependency between reliability, models’ architecture, and pre-training regime. Overall, our evaluation indicates that, while few-shot strategies have the potential to elicit syllogistic reasoning in LLMs, existing models are still far from achieving the robustness and consistency required for safe biomedical NLI applications.

Authors:

(1) Magdalena Wysocka, National Biomarker Centre, CRUK-MI, Univ. of Manchester, United Kingdom;

(2) Danilo S. Carvalho, National Biomarker Centre, CRUK-MI, Univ. of Manchester, United Kingdom and Department of Computer Science, Univ. of Manchester, United Kingdom;

(3) Oskar Wysocki, National Biomarker Centre, CRUK-MI, Univ. of Manchester, United Kingdom and ited Kingdom 3 I;

(4) Marco Valentino, Idiap Research Institute, Switzerland;

(5) André Freitas, National Biomarker Centre, CRUK-MI, Univ. of Manchester, United Kingdom, Department of Computer Science, Univ. of Manchester, United Kingdom and Idiap Research Institute, Switzerland.


This paper is available on arxiv under CC BY-NC-SA 4.0 license.


Written by largemodels | The Large-ness of Large Language Models (LLMs) ushered in a technological revolution. We dissect the research.
Published by HackerNoon on 2024/12/10