NevIR: Negation in Neural Information Retrieval

Explore the impact of negation on neural information retrieval systems in this 11-minute conference talk from EACL 2024. Delve into a study conducted by researchers at the Center for Language & Speech Processing (CLSP) at Johns Hopkins University, which addresses a significant gap in understanding how negation affects modern IR architectures. Learn about the straightforward benchmark developed to assess IR models' ability to rank documents differing only by negation. Discover the varying performance across different IR architectures, with cross-encoders performing best, followed by late-interaction models, and bi-encoder and sparse neural architectures lagging behind. Examine findings that reveal most information retrieval models, including state-of-the-art ones, struggle with negation, often performing no better than random ranking. Gain insights into potential improvement strategies, such as continued fine-tuning on contrastive datasets containing negations and increasing model size, while acknowledging the persistent gap between machine and human performance in this area.