Serverless Jenkins DevOps Pipeline with K8s and Spark

Explore a comprehensive DevOps pipeline for Spark/BigData applications using serverless Jenkins, Kubernetes, and Spark in this 48-minute conference talk from Devoxx Poland 2019. Learn how to overcome the challenges of centralized services in large enterprises and implement a true DevOps lifecycle with on-demand resource creation. Follow a step-by-step live demonstration of building the pipeline, including creating ephemeral Jenkins instances, allocating build agents on Kubernetes clusters, tailoring a minified Hadoop/HDFS cluster for testing, and executing tests with result collection. Discover how to leverage declarative style Jenkinsfile pipelines for seamless integration across different runtime environments. Gain insights into Continuous Integration, Delivery, and Deployment practices, as well as the benefits of DevOps in Big Data testing scenarios. Understand key concepts such as Jenkins master-slave architecture, Kubernetes basics, and environment setup for testing.