Completed
Hadoop-on-Lustre architecture Comparison between HDFS and Lustre
Class Central Classrooms beta
YouTube videos curated by Class Central.
Classroom Contents
Chameleon: Expanding Open-Source Ambari for HPC
Automatically move to the next video in the Classroom when playback concludes
- 1 Chameleon Expanding Open-Source Ambari for HPC
- 2 Motivation (Trend Perspective) HPC and Bigdata is converging
- 3 Motivation Application Perspective Genome analysis by Our scientist Some stages of data pipeline begin to support bigdata platform
- 4 Ambari Overview Apache Ambari is a 100% open source platform for provisioning managing and monitoring Hadoop clusters
- 5 Extension Points for custom service development Ambari view is a plugin that provides a way to connect custom functions to the web UI Ambari stack defines a set of everything needed to define service…
- 6 lustrefs Management Service Lustre Kernel Installation function(LustrekernelUpdater)
- 7 Account Management Service Hadoop does not support strong authentication by default • Hadoop supports Kerberes for that, but, causes performance
- 8 Hadoop-on-Lustre architecture Comparison between HDFS and Lustre
- 9 Related works for Hadoop-on-Lustre Xyrates • MapReduce Job shows theoretical performance gains on an appropriately designed Lustre based HIPC cluster with Infinband network Seagate's lustrels plugin
- 10 2 Hadoop-on-Lustre execution environment Hadoop-on-Lustre execution environment • Works for diskless cluster backed by Lustre • Uses secure container configuration for multi-tenancy
- 11 Motivation Dynamic metrics management is required
- 12 YARN Application Monitoring Service Time-series data monitoring
- 13 3 TimeScaleDB Open-Source time-series database optimized for fast
- 14 3 Data management structure Alter Table
- 15 HPC Resources Monitoring Provides HPC monitoring information through web UI
- 16 Summary HPC and big data convergence makes the distinction between data analytics and computational science's ecosystem disappear. Chameleon is a Bigdata platform operation management system consider…
