Explore the optimization techniques for enhancing a distributed SQL database engine in this conference talk from Conf42 DevOps 2024. Dive into YQL architecture, massive data handling, and benchmark-driven approaches using TPC-H. Learn about continuous integration strategies, running distributed engines in single and multi-process configurations, and performance measurement techniques. Discover Linux performance tools for investigating bottlenecks, including slow joins and memory allocation issues. Examine containerization challenges, user code isolation, and execution pipeline optimizations using Unix pipes. Gain insights into Linux IPC performance and future directions for distributed database optimization.
Enhancing a Distributed SQL Database Engine - Performance Optimization Techniques
Overview
Syllabus
intro
preamble
about me
outline
yql: distributed sql database engine
massive data handling
yql architecture overview
example
execution plan
bechmark-driven approach
tpc-h benchmark
data generation
continuous integration ci
run distributed engine in one process
run distributed engine in multi-process configuration
unixbench's style measures
tpc--h 100: target values
hardware
linus performance tools
more linux performance tools
slow join
perf top
stackcount
memory allocator
results
user code isolation
execution plan
executions pipeline with unix pipe
slow results
linux ipc performance
pipe performance
results: dq @pipe+ pipe
what's next?
thanks you
Taught by
Conf42
