Combining Buffered I/O and Direct I/O in Distributed File Systems

Explore a 22-minute conference talk from USENIX FAST '24 that delves into combining buffered I/O and direct I/O in distributed file systems. Learn about the advantages and disadvantages of both I/O modes in high-performance computing applications. Discover a novel approach that dynamically switches between buffered and direct I/O based on factors such as I/O size, file lock contention, and memory constraints. Examine the implementation of this technique in the Lustre client and server, including additional features like delayed allocation. Understand how this approach significantly improves throughput, achieving up to 3 times higher performance than the original Lustre and outperforming other distributed file systems by up to 13 times under various conditions and real-world workloads.