Explore a comprehensive DevOps pipeline for Spark/BigData applications using serverless Jenkins, Kubernetes, and Spark in this 48-minute conference talk from Devoxx Poland 2019. Learn how to overcome the challenges of centralized services in large enterprises and implement a true DevOps lifecycle with on-demand resource creation. Follow a step-by-step live demonstration of building the pipeline, including creating ephemeral Jenkins instances, allocating build agents on Kubernetes clusters, tailoring a minified Hadoop/HDFS cluster for testing, and executing tests with result collection. Discover how to leverage declarative style Jenkinsfile pipelines for seamless integration across different runtime environments. Gain insights into Continuous Integration, Delivery, and Deployment practices, as well as the benefits of DevOps in Big Data testing scenarios. Understand key concepts such as Jenkins master-slave architecture, Kubernetes basics, and environment setup for testing.
Serverless Jenkins DevOps Pipeline with K8s and Spark
Overview
Syllabus
Intro
Story behind
Before we start
Continuous Integration - how?
Continuous Delivery - gains
Continuous Delivery - how?
Continuous Deployment
DevOps - why?
Testing in Big Data
Jenkins master-slave architecture
Kubernetes 101
Kubernetes - basic terms
Environment for tests
Taught by
Devoxx
