Articles

The medallion architecture applied to large-scale ETL pipelines and Azure Data Lake is examined. The Azure-specific aspects of medallion architecture are deciphered for improved design, operation, and governance of ETL pipelines. Bronze-layering covers data ingestion and retention, storage organization, naming conventions, schema-on-read and schema-on-write trade-offs, and accessibility by non-technical users. The design of the silver layer focuses on cleansing, standardization, deduplication, conformance, and data enrichment. Finally, gold-layering deals with the creation of data products—curated datasets that are cataloged, ready to be consumed by visual-analysis tools, and offered through published APIs. The work centers on using medallion architecture to tackle the challenges of ETL pipelines running in Azure Data Lake. Such pipelines face increasing volumes and complexities, including new data sources, customer mandates for improved product quality, stricter regulatory requirements, and the need for customer-oriented APIs. The analysis is further filtered by success criteria expressed in data quality, observability, and monitoring för the whole medallion architecture, and data-organizational strategy for the silver and gold layers with specific focus on the definition and creation of data products.