Protecting PII and PHI Data in Data Lakes with Column-Level Encryption

Learn how to implement column-level encryption to protect Personally Identifiable Information (PII) and Protected Health Information (PHI) in data lakes. Explore a comprehensive approach to data security, including design principles for column-level encryption, methods to safeguard PII data used in joins, techniques for authorized runtime decryption, and strategies for encryption key rotation. Discover how Northwestern Mutual leverages Fernet and AES encryption libraries, user-defined functions, and Databricks secrets to create a secure process for encrypting PII, limiting access to authorized personnel only. Gain insights into building a secure data lake that mitigates both internal and external data breach threats without duplicating data or metadata. The presentation covers the importance of data protection, recent breach examples, encryption in transit, and practical tips for implementation, concluding with a discussion on Business Intelligence and PII handling.