Analyzing and Improving the Scalability of In-Memory Indices for Managed Search Engines

Explore a comprehensive analysis of in-memory indices for managed search engines in this 19-minute video presentation from ISMM 2023. Delve into the challenges faced by popular search engines like Apache Solr and Elasticsearch when hosting large inverted indices in main memory. Examine the trade-offs between compressed and uncompressed indices, considering factors such as storage footprint, query response times, and decompression latency. Discover how emerging non-volatile memory (NVM) technologies offer potential solutions to these challenges. Learn about rigorous performance evaluations comparing DRAM and NVM-backed indices, and understand the impact of spatial locality on search algorithms. Gain insights into new space-time tradeoffs for storing in-memory inverted indices and explore the scalability of uncompressed indices on NVM-backed heaps with large core counts and index sizes. This presentation by Aditya Chilukuri and Shoaib Akram from the Australian National University offers valuable insights for researchers and practitioners working with managed search engines and large-scale data indexing.