# Building a Medallion Architecture: A Practical Guide
Modern data platforms quickly become unmanageable without clear structure. Data arrives from multiple sources in different formats, pipelines evolve over time, and before long, no one trusts the numbers in the dashboard. The medallion architecture is a practical response to this problem. By dividing your platform into clear layers—bronze, silver, and gold—you create a system that is easier to understand, debug, and trust.
## What Is the Medallion Architecture?
The medallion pattern divides a data platform into three progressive layers, each with a specific purpose and data quality standard. The terminology was popularized by Databricks, but the underlying concepts have been used in data engineering for years.
– **Bronze**: The raw landing zone where data arrives exactly as received from source systems.
– **Silver**: The cleaned and standardized layer, where data quality rules are applied.
– **Gold**: The curated layer, optimized for specific business questions and analytics.
At the boundary of each layer, there should be a clear, documented definition of what data lives there and how it got there. This is critical for long-term maintainability.
## Layer Responsibilities in Detail
### Bronze Layer
The bronze layer stores immutable, append-only copies of source data. Helpful metadata includes:
– Source system and file or event identifier
– Ingestion timestamp and business effective date
– Record count and batch identifier
– File hash for idempotency
At this stage, errors are typically quarantined rather than corrected. If a load fails midway, it should be marked as failed and retried as a new batch. Duplicate deliveries are prevented using source hashes or file identifiers.
### Silver Layer
Silver transforms bronze data into a clean, reliable foundation. Common operations include:
– Parsing and enforcing correct data types
– Standardizing dates, currencies, and codes
– Deduplication
– Quarantining invalid records
– Joining reference data
Silver tables should maintain business-level detail and enforce clear rules. For example, an `order_total` must have a defined currency and numeric type, and an `order_id` must be unique.
### Gold Layer
Gold is built around business use cases. It typically contains:
– Aggregated metrics such as daily totals or counts
– Star schemas or data marts for fast queries
– Tailored datasets for finance, marketing, or operations
Because gold data is trusted and pre-aggregated, analytics and reporting run faster and with greater confidence.
## Tools You Can Use
You can implement a medallion architecture with many different technologies. Common choices include:
– **Databases and warehouses**: PostgreSQL, Snowflake, BigQuery, Redshift
– **Open table formats**: Delta Lake, Apache Iceberg, Hudi
– **Transformation tools**: SQL, dbt, Python, Spark
– **Cloud services**: AWS Glue, Athena, Step Functions, S3
This article uses **DuckDB** and plain SQL to keep the example simple and portable.
## A Practical Example: Retail Orders
Imagine a nightly CSV export from an online retailer. The data contains duplicates, parsing errors, and negative amounts. The goal is to produce daily sales by region and currency by 07:00 each morning.
The pipeline follows five stages:
1. Store each CSV unchanged in the bronze layer
2. Validate and standardize data in silver
3. Quarantine bad records
4. Aggregate to daily sales in gold
5. Prevent re-ingestion of the same source file
The complete code, project structure, and output are included in the original article, demonstrating how the medallion pattern works in practice.
## Frequently Asked Questions
### What is the difference between bronze and silver?
Bronze stores raw, unmodified source data. Silver applies cleaning and standardization rules, producing data that downstream systems can safely consume.
### Do I need a data warehouse to use medallions?
No. While a warehouse simplifies things, you can implement medallions using object storage (such as S3) and open table formats like Delta or Iceberg.
### How do I handle late or corrected data?
In bronze, append new data and record metadata. In silver, design your pipelines to be reproducible so that reprocessing yields the same results.
### What should I monitor in production?
Key checks include row counts, key uniqueness, quarantine rates, freshness, and schema drift. These guardrails help maintain trust across layers.
### Is the medallion pattern only for big data?
No. The same principles apply to small projects. The overhead is minimal, and the clarity and debuggability are valuable at any scale.
## Conclusion
The medallion architecture works because it makes data quality visible and explicit. Data from the source is not the same as validated data, and validated data is not automatically ready for analytics.
The example in this article was simplified, but the principles scale to production systems. As you move to larger platforms, you will need to decide on storage formats, orchestration tools, and automated checks—but the foundational layers remain the same.
By adopting a medallion structure, you turn your data platform into a series of well-defined, testable steps. That clarity pays off every time a dashboard shows an unexpected number and someone asks, “Why?”
