delta lake - PipelinePulse

Sign in Subscribe

delta lake

A collection of 6 posts

Medallion Architecture Explained (Bronze, Silver, Gold)

Medallion Architecture

Medallion Architecture Explained (Bronze, Silver, Gold)

The complete guide to medallion architecture in Databricks — bronze raw ingestion with Auto Loader, silver cleaning with MERGE upserts, gold business aggregations, orchestration, table maintenance, and 5 anti-patterns to avoid. Production code for every layer.

Apache Spark Structured Streaming for Beginners

structured streaming

Apache Spark Structured Streaming for Beginners

Build your first Spark Structured Streaming pipeline in Databricks. Covers readStream, writeStream, trigger modes, foreachBatch MERGE upserts, Auto Loader for file ingestion, and watermarks for late data — beginner-friendly with copy-paste code templates.

How to Debug Failing Databricks Pipelines

How to Debug Failing Databricks Pipelines

A systematic approach to debugging Databricks pipeline failures. Covers OOM errors, missing transaction logs, schema mismatches, MERGE duplicates, JDBC NaN issues, and streaming checkpoint corruption — with error decision trees, log reading guides, cluster config checklists, and performance fixes.

Delta Lake Time Travel: A Practical Guide

Delta Lake Time Travel: A Practical Guide

Query any previous version of a Delta table, restore data after accidental deletes, and audit changes over time. Covers VERSION AS OF, TIMESTAMP AS OF, RESTORE, DESCRIBE HISTORY, streaming replay, and the VACUUM trade-off — with SQL and PySpark examples.

Schema Evolution in Delta Lake: Handling Changes Gracefully

Schema Evolution in Delta Lake: Handling Changes Gracefully

Master Delta Lake schema evolution — mergeSchema, overwriteSchema, column mapping, type widening, and streaming schema changes. Includes a decision guide, 4 migration patterns, and 7 common gotchas from production experience.

PySpark Window Functions: The Complete Guide with Examples

PySpark Window Functions: The Complete Guide with Examples

Master PySpark window functions with real code examples. Covers row_number, rank, lag, lead, running totals, moving averages, sessionization, and more — plus 6 common gotchas that trip up even senior engineers.