Tuning Kafka

Introduction

• Tuning Kafka

1. Introducing Kafka Tuning

• Prerequisites
• An overview of Kafka
• Producers and consumers
• Kafka optimization theorem
• End-to-end latency in Kafka
• Install and set up Apache Kafka
• Publish and consume messages using console scripts

2. Tuning Kafka Producers

• Running a Kafka cluster with three brokers
• Running basic producer performance tests
• Effects of partitioning on producers
• Effects of replication on producers
• Tuning producer acks
• Tuning producer compression
• Tuning batch size and linger time
• Effect of message size and record count

3. Tuning Kafka Consumers

• Running a basic consumer performance test
• Tuning fetch bytes and wait time for consumers
• Tuning session timeout and heartbeat interval
• Tuning max poll interval and max poll records

4. Tuning Kafka Brokers

• End-to-end latency with replication
• Tuning throughput
• Tuning log retention

5. Performance Testing with Python

• Producer performance testing using Python
• Consumer perf testing using Python

Conclusion

• Summary and next steps