Explore an in-depth technical analysis of I/O performance issues for Windows guests in OpenStack Ceph clouds during this 45-minute conference talk from linux.conf.au. Dive into the root causes of slow I/O performance, including the misalignment between Windows' 512-byte and Linux/Ceph's 4096-byte I/O boundaries. Follow the speaker's journey through problem identification, analysis of Ceph daemon performance, and the development of multiple fixes involving Ceph, Nova, Cinder, and Windows VirtIO drivers. Gain insights into file system operations, disk alignment challenges, and key takeaways for optimizing performance in hybrid cloud environments. Learn valuable lessons applicable to large-scale OpenStack deployments with significant Windows guest footprints.
Overview
Syllabus
Intro
User Complaints
ceph daemon osd. 1 perf dump (latency)
ceph daemon osd. 1 perf dump (throttle)
ceph daemon osd. 1 dump_ops_in_flight
ceph tell osd.* config set debug_filestore 10
Flamegraph
How FileStore works
Possible Fix #1
Reproduction
madvise(FADVISE_DONTNEED)
Possible Fix #2
512b/4k Disk Alignment ("4K Drives")
Additional Fix #3
Additional Fix #4
Windows 4K alignment
Additional Fix #5
Key Takeaways
Taught by
linux.conf.au
