Obtain, Distribute and Use Temporary Credentials Automatically in Apache Spark and Apache Flink

Learn how to implement secure authentication in distributed data processing systems using Apache Spark and Apache Flink. Explore the challenges of continuous data processing at high load and discover solutions for obtaining, distributing, and using temporary credentials automatically. Understand the new delegation token framework in Apache Spark and the more generic implementation in Apache Flink, which enable workloads to scale securely to thousands of workers. Gain insights into common pitfalls of distributed system authentications and learn how to easily add security features to Apache Spark or Apache Flink projects. This 23-minute conference talk, presented by Gabor Somogyi, a Data Platform Engineer at Apple and Apache Flink + Spark Committer, covers essential aspects of implementing robust security measures in open-source Big Data frameworks.