Rust for Data Engineering

1. Getting Started with the Modern Rust Development Ecosystem

• Meet the instructor and course overview
• Introduction to the AI coding paradigm shift
• Introduction to cloud-based development environments
• Introduction to GitHub Copilot ecosystem for Rust
• Prompt engineering with GCP BigQuery SQL
• Introduction to AWS CodeWhisperer for Rust
• Using Google Bard to enhance productivity
• Continuous integration with Rust and GitHub actions

2. Rust Sequences and Maps

• Introducing Rust sequences and maps
• Demo: Print Rust data structures
• Demo: Vector fruit salad
• Demo: VecDeque fruit salad
• Demo: LinkedIn list fruit salad
• Demo: Fruit salad CLI
• Demo: HashMap frequency counter
• HashMap language comparison

3. Rust Sets, Graphs, and Miscellaneous Data Structures

• Analyzing UFC fighter network using graph centrality in Rust
• Storing unique fruits using HashSet in Rust
• Maintaining sorted and unique fruits using BTreeSet in Rust
• Creating a fig-priority fruit salad using BinaryHeap in Rust
• PageRank algorithm for sports data
• Showing shortest path with Dijkstra
• Detecting strongly connected components: A deep dive into Kosaraju's algorithm
• Simple charting of data structures in Rust

4. Rust Safety and Security Features

• Multifactor authentication
• Network segmentation
• Least privilege access
• Encryption
• Mutable fruit salad
• Customize fruit salad with a CLI
• Data race example

5. Security Programming with Rust

• High availability
• Understanding the Homophonic cipher: A cryptographic technique
• Decoding the secrets of the Caesar cipher
• Building a Caesar cipher command-line interface
• Creating a decoder ring: A practical guide
• Detecting duplicates with SHA-3: A data integrity tool
• Incident response
• Compliance

6. Concurrency with Rust

• Core concepts in concurrency
• Dining philosophers
• Web crawl Wikipedia with Rayon
• Intelligent chatbot with Tokio
• Multi-threaded deduplication with Rust
• Energy efficiency Python vs. Rust
• Concurrency stress test with a GPU
• Host efficiency serverless optimization problem

7. Using Rust to Manage Data, Files, and Network Storage

• Process CSV files in Rust
• Using Cargo Lambda with Rust
• List files on AWS EFS with Rust
• Use AWS S3 storage
• Use AWS S3 storage from Rust
• Write encrypted data to tables or Parquet files

8. DataFrames with Rust, Python, and Notebooks

• What is Colab?
• Using Bard to enhance notebook development
• Exploring life expectancy in a notebook
• Load a DataFrame with sensitive data
• Using MLFlow with Databricks Notebooks
• End to End ML with MLFlow and Databricks
• Exploring global life expectancy with Polars

9. Using Rust with Cloud SDKs and CLIs for Data Engineering

• Cloud developer workspace advantage
• Onboarding to GCP with Python and Rust
• Using GCP Cloud Shell with Rust
• Learn AWS CloudShell
• Prototyping AI APIs with AWS CloudShell
• Cloud9 with CodeWhisperer
• Demo: GCP App Engine Rust Deploy
• Containerized Rust Actix Microservice on AWS

10. Getting Started with Rust Data Pipelines (Including ETL)

• Jack and the Beanstalk data pipelines
• Open source data engineering: Pros and cons
• Core components of data engineering pipelines
• Rust AWS step functions pipeline
• Rust AWS Lambda Async S3 size calculator
• What is Distroless?
• Demo: Deploying Rust microservices on GCP

11. Using Rust and Python for LLMs, ONNX, Hugging Face, and PyTorch Pipelines

• Introduction to Hugging Face Hub
• Rust PyTorch pre-trained model ecosystem
• Rust GPU Hugging Face translator
• Rust PyTorch high-performance options
• EFS ONNX Rust inference with AWS Lambda
• Theory behind model fine-tuning
• Doing fine-tuning

12. Building SQL Solutions with Rust, Generative AI, and Cloud

• Selecting the correct database on GCP
• Rust SQLite Hugging Face zero-shot classification
• Prompt engineering for BigQuery
• BigQuery to Colab pipeline
• Exploring data with BigQuery
• Using public data sets for data science
• Querying log files with BigQuery
• There is no one-size database
• Course conclusion