PipelinePulse - PipelinePulse

Sign in Subscribe

PipelinePulse

Using AI Agents for Data Engineering: A Practical Guide

Using AI Agents for Data Engineering: A Practical Guide

How to actually use AI agents as a data engineer in 2026 — Claude, Copilot, Cursor, and Databricks Assistant compared honestly. Includes 5 prompt patterns that produce production code, a workflow for staying in control, and real examples from debugging to building silver transforms.

Medallion Architecture Explained (Bronze, Silver, Gold)

Medallion Architecture

Medallion Architecture Explained (Bronze, Silver, Gold)

The complete guide to medallion architecture in Databricks — bronze raw ingestion with Auto Loader, silver cleaning with MERGE upserts, gold business aggregations, orchestration, table maintenance, and 5 anti-patterns to avoid. Production code for every layer.

Apache Spark Structured Streaming for Beginners

structured streaming

Apache Spark Structured Streaming for Beginners

Build your first Spark Structured Streaming pipeline in Databricks. Covers readStream, writeStream, trigger modes, foreachBatch MERGE upserts, Auto Loader for file ingestion, and watermarks for late data — beginner-friendly with copy-paste code templates.

Data Observability: Tools and Practices for 2026

data observability

Data Observability: Tools and Practices for 2026

The 5 pillars of data observability, DIY monitoring with SQL templates, tool comparison (Great Expectations, Soda Core, Monte Carlo, Databricks Lakehouse Monitoring), and a maturity ladder for knowing what to build when. Start from zero and scale up.

Databricks Jobs vs Workflows: When to Use What

Databricks Jobs vs Workflows: When to Use What

Single-task jobs vs multi-task workflows in Databricks — when to use each, how to set up DAGs with parallel execution, job clusters vs all-purpose, task dependencies, retry configs, and scheduling. Complete guide with JSON templates and a decision framework.

How to Debug Failing Databricks Pipelines

How to Debug Failing Databricks Pipelines

A systematic approach to debugging Databricks pipeline failures. Covers OOM errors, missing transaction logs, schema mismatches, MERGE duplicates, JDBC NaN issues, and streaming checkpoint corruption — with error decision trees, log reading guides, cluster config checklists, and performance fixes.

Delta Lake Time Travel: A Practical Guide

Delta Lake Time Travel: A Practical Guide

Query any previous version of a Delta table, restore data after accidental deletes, and audit changes over time. Covers VERSION AS OF, TIMESTAMP AS OF, RESTORE, DESCRIBE HISTORY, streaming replay, and the VACUUM trade-off — with SQL and PySpark examples.

Schema Evolution in Delta Lake: Handling Changes Gracefully

Schema Evolution in Delta Lake: Handling Changes Gracefully

Master Delta Lake schema evolution — mergeSchema, overwriteSchema, column mapping, type widening, and streaming schema changes. Includes a decision guide, 4 migration patterns, and 7 common gotchas from production experience.

Data Contracts for Data Engineers: Getting Started in 2026

Data Contracts for Data Engineers: Getting Started in 2026

Learn what data contracts are, why they matter in 2026, and how to implement them in your pipelines with YAML specs and PySpark validation. Includes a complete contract template, enforcement code, CI/CD testing patterns, and a tool comparison — no expensive tooling required.

PySpark Window Functions: The Complete Guide with Examples

PySpark Window Functions: The Complete Guide with Examples

Master PySpark window functions with real code examples. Covers row_number, rank, lag, lead, running totals, moving averages, sessionization, and more — plus 6 common gotchas that trip up even senior engineers.

Databricks vs Snowflake: Honest Comparison for Data Engineers [2026]

Databricks vs Snowflake: Honest Comparison for Data Engineers [2026]

An honest comparison of Databricks and Snowflake across SQL experience, data engineering, ML, governance, and cost — from a working data engineer with no vendor bias.

Best AI Tools for Data Engineers in 2026 [Honest Reviews]

Best AI Tools for Data Engineers in 2026 [Honest Reviews]

Honest reviews of AI tools I actually use as a data engineer — coding assistants, data platforms, quality monitoring, and infrastructure tools. No sponsored listicle, just real takes.

SCD Type 2 Implementation in Databricks [Step-by-Step Guide]

SCD Type 2 Implementation in Databricks [Step-by-Step Guide]

How to implement SCD Type 2 in Databricks — change detection with null-safe comparisons, MERGE expiration, version inserts, and a complete reusable PySpark function.

How to Handle Null Values in Spark DataFrames [Complete Tutorial]

How to Handle Null Values in Spark DataFrames [Complete Tutorial]

Every null-handling pattern for Spark DataFrames — detection, filling, dropping, null-safe joins, and production-ready frameworks with PySpark and SQL examples.

Databricks MERGE INTO: Complete Guide with Real Examples [2026]

Databricks MERGE INTO: Complete Guide with Real Examples [2026]

Everything you need to know about MERGE INTO in Databricks — basic upserts, conditional updates, soft deletes, and performance tips from production experience.

Data Quality Checks Every Pipeline Should Have [2026 Guide]

Data Quality Checks Every Pipeline Should Have [2026 Guide]

Six categories of data quality checks with SQL and PySpark examples, plus a reusable Python framework to run them automatically after every pipeline load.

Delta Table OPTIMIZE, Z-ORDER, and VACUUM Explained [2026 Guide]

Delta Table OPTIMIZE, Z-ORDER, and VACUUM Explained [2026 Guide]

A practical guide to Delta Lake's three essential maintenance commands, with production-ready scripts and scheduling tips.

How to Run a Scheduled Python ETL Pipeline on a VPS [Step-by-Step Guide]

How to Run a Scheduled Python ETL Pipeline on a VPS [Step-by-Step Guide]

Not every data pipeline needs Databricks or Airflow. Sometimes you just need a Python script that runs on a schedule, pulls data from an API, transforms it, and loads it into a database. No orchestration framework. No managed platform. Just a $6/month server that quietly does its job. I

How to Fix Duplicate Records in Delta Tables [2026 Guide]

How to Fix Duplicate Records in Delta Tables [2026 Guide]

Your pipeline looks clean. Your row counts match. But your downstream reports show inflated numbers, and your stakeholders are asking questions you can't answer yet. Sound familiar? Duplicate records in Delta tables are one of the most common — and sneakiest — data quality issues in production pipelines. They don&