Data Engineering · Field Notes
Notes from the trenches of retail-scale data: PySpark internals, Airflow orchestration, warehouse modelling, and the cost math nobody warns you about.
I build robust data pipelines, optimize cloud infrastructure, and specialize in transforming raw data into reliable analytics. This site serves as my digital garden—a place to document field notes, architecture blueprints, and the hard-learned lessons of data engineering.
Field Notes · from the systems frontier
GC pressure is the most misdiagnosed failure in Spark, and the one most often ‘fixed’ by adding heap, which usually makes it worse. Here is where to look in the UI and the logs, how to tell it apart from data skew, and the levers that actually move the number.
Read the postDiscover · Working References
Pick the format that matches the moment: scan the commands, review the architecture patterns, or choose the right tool for the pipeline.
Scalable cloud designs, streaming pipelines, and batch architectures explained.
Structuring your warehouse for query performance and long-term maintainability.
Making complex queries and large clusters cheaper without sacrificing execution speed.
Short, honest reviews of tools, scripts, and utilities every data engineer needs.
GC pressure is the most misdiagnosed failure in Spark, and the one most often ‘fixed’ by adding heap, which usually makes it worse. Here is where to look in the UI and the logs, how to …
Oct 12, 2026 19 min readA short tour of what this site is, the problems I write about, and how to get the most out of the cheatsheets.
Oct 11, 2026 2 min readMost Spark tuning guides teach it backwards. Here’s the data-first approach that production engineers actually use.
Oct 11, 2026 7 min readGC Pressure in Spark: How to Find It, Read It, and Actually Fix It
Oct 12, 2026 · New Article
Welcome to the Field Notes
Oct 11, 2026 · New Article
The Real Math Behind PySpark Resource Sizing
Oct 11, 2026 · New Article
The Ultimate SQL Cheatsheet
Oct 11, 2026 · Cheatsheet Updated
Apache Airflow
Oct 11, 2026 · Cheatsheet Updated