Explore the challenges and solutions in high-throughput logging with low resource usage in this 35-minute conference talk from KubeCon + CloudNativeCon North America 2021. Dive into the journey of performance optimization in the Fluent Bit project, focusing on Network I/O + TLS, filesystem buffers, routing, and multiplexing. Learn how to scale logging services in distributed environments while managing increased data volumes efficiently. Discover techniques to improve processing speed from 5k/sec to over 30k/sec using a single-core CPU through design improvements and optimal use of Linux OS interfaces. Gain insights into balancing performance and cost, understanding logging workflows, data analysis, and various aspects of the logging pipeline including input/output, processing, routing, and network setup. Compare JSON vs Message Pack for data serialization and explore memory serialization and buffering techniques. Conclude with a practical demonstration of the concepts discussed.
High Throughput with Low Resource Usage - A Logging Journey
CNCF [Cloud Native Computing Foundation] via YouTube
Overview
Syllabus
Introduction
Performance vs Cost
Fluency
Logging
Debug Mode
Logging Workflow
Data Analysis
IO
Data Processing
Output
Pipeline
System Code
Processing
Routing
Network Setup
JSON vs Message Pack
Data serialization
Memory serialization
Memory buffering
Demo
Taught by
CNCF [Cloud Native Computing Foundation]
