xprobes: Hybrid User/Kernel eBPF Probes for Cross-Layer Observability

Explore the concept of xprobes, a hybrid approach combining uprobes and kprobes for enhanced performance diagnostics in production environments. Delve into the limitations of existing tools like flamegraphs, distributed tracing, and BCC probes for identifying root causes of performance issues. Learn how xprobes provide cross-layer observability by maintaining aggregate distributions for functions of interest, updated by both userspace and kernelspace events. Examine the cpu-latency xprobe as an example, which measures on-cpu time of userspace functions while accounting for context switching. Discover how xprobes can offer powerful performance insights by comparing wall clock latency distributions with actual on-cpu and I/O latency distributions. Consider potential strategies for managing xprobe overheads and their implications for future performance analysis techniques.