How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities

Explore a technical presentation that delves into HP Inc.'s innovative approach to scaling catalog ingestion for hundreds of thousands of entities in Backstage. Learn about the challenges faced when ingesting over 100,000 entities from a data source and discover the solutions implemented through careful planning, properly scaled computing resources, and clever customization. Gain insights into extending basic Entity Provider and Catalog Processor functionality for large-scale enterprise use. Understand the scope of the technical challenge, including the various types of entities and data sources utilized by HP Inc. Dive deep into the intricacies of their entity ingestion process, exploring the extended features of the base Entity Provider mechanism that enable handling of extremely large datasets. Examine important considerations such as entity validation and relations in this comprehensive overview of scaling Backstage for enterprise-level catalog ingestion.