Data Management with Apache NiFi

Introduction

• Data management with Apache NiFi

1. Getting Started with Apache NiFi

• Prerequisites
• Apache NiFi core concepts
• Apache NiFi architecture
• macOS: Download Apache NiFi
• Windows: Download Apache NiFi
• macOS and Windows: Starting the NiFi server
• macOS: Logging into NiFi
• Windows: Logging into Apache NiFi
• Exploring the NiFi UI

2. Building and Running Data Flows

• Adding and configuring processors
• Running a dataflow
• Flow summaries and flow operations
• Creating, saving, and importing flow templates

3. Integrating NiFi with PostgreSQL

• Set up a PostgreSQL table
• Add GetFile and PutDatabase processors
• Configuring the CSVReader controller service
• Configuring the DBConnectionPool controller service
• Running and encountering dataflow errors
• Configuring the ReplaceText processor
• Running a dataflow to insert data into PostgreSQL
• Configuring the Run schedule
• Configuring task concurrency
• Configure processors to execute SQL queries

4. Exploring NiFi Processors and Process Groups

• Configure processors to convert data formats
• Updating attributes and writing files
• Running the end-to-end flow
• Modularize dataflows with process groups
• Running a dataflow with a process group

5. Integrating NiFi with Amazon S3

• Creating a PostgreSQL table, Amazon S3 bucket, and an IAM user
• Reading from files and storing in a database
• Configure the PutS3Bucket processor
• Running the flow to read from file and write to database and Amazon S3

6. Using Advanced Features of NiFi Dataflows

• Configuring back pressure
• Observing the effects of back pressure
• Configuring an InvokeHTTP processor
• Using a funnel in the dataflow
• Running the dataflow to read from URL and write to files
• Setting up monitoring for the dataflow
• Configuring alerting for the dataflow
• Email alerts on dataflow success
• Email alerts on change in activity status

Conclusion

• Summary and next steps