Use Spark from Anywhere - A Spark Client in Scala Powered by Spark Connect

Explore the evolution of Apache Spark and its new client-server architecture in this Scala Days 2023 Seattle conference talk. Discover how Spark Connect enables remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans. Learn about the benefits of separating client and server, allowing Spark to be leveraged from various environments including modern data applications, IDEs, notebooks, and programming languages. Gain insights into the development of the Scala client for Spark Connect, its integration with the server, and efforts to solve application isolation challenges. Dive deep into Spark Connect's architecture and understand how the community can extend it for new programming languages and frameworks. Presented by Denny Lee, a Databricks Developer Advocate with expertise in distributed systems and data sciences, and Ginger Holt, a Senior Staff Data Scientist at Databricks specializing in forecasting and predictive models.