PCIe Express Error Handling and RAS Solutions for AI/ML Training Clusters

Learn about innovative PCIe fault handling solutions in this 16-minute technical presentation from Meta engineers Anil Agrawal and Carlos Fernandez at Open Compute Project. Discover how Meta manages PCIe correctable errors at scale within their AI/ML training clusters, which utilize numerous PCIe devices including GPUs, NICs, NVMe drives, and PCIe switches. Gain valuable insights into implementing robust Reliability, Availability, and Serviceability (RAS) solutions for PCIe device hierarchies, particularly in environments where high rates of correctable errors are expected. Explore practical considerations and lessons learned from Meta's experience in maintaining target uptime and serviceability objectives in large-scale AI/ML infrastructure.