A data engineer exists so the rest of an organization can trust the numbers. Raw\ndata arrives messy, late, duplicated, and lying — from databases, event streams,\nAPIs, and hand-edited spreadsheets — and somebody must turn it into tables an\nanalyst, a model, or an executive can rely on without a footnote. The gap between\n"the data is somewhere" and "we have a correct, timely answer" is bridged by\nplumbing that holds when the source schema changes at 3 a.m. unannounced.
\n","wordCount":81},{"heading":"Core Mission","id":"core-mission","markdown":"Move data from where it is produced to where it is needed — correct, complete, on\ntime, and reproducible — so the same input always yields the same trustworthy\noutput, run after run.","html":"Move data from where it is produced to where it is needed — correct, complete, on\ntime, and reproducible — so the same input always yields the same trustworthy\noutput, run after run.
\n","wordCount":31},{"heading":"Primary Responsibilities","id":"primary-responsibilities","markdown":"The visible work is writing pipelines; the actual work is defending data\ncorrectness across systems never designed to agree. A data engineer designs\ningestion from operational sources without melting them; models the\nwarehouse/lakehouse so a question maps to a query, not a forensic investigation;\nwrites idempotent, replayable transformation logic (ELT or ETL); enforces quality\nwith tests and contracts; orchestrates the dependency graph so the daily run\nfinishes before dashboards refresh; manages schema evolution so one upstream change\ndoesn't break twelve downstream jobs; and owns compute cost that can bill five\nfigures a month. When a number looks wrong, the data engineer is asked why — and \"I\ndon't know\" won't hold for long.","html":"The visible work is writing pipelines; the actual work is defending data\ncorrectness across systems never designed to agree. A data engineer designs\ningestion from operational sources without melting them; models the\nwarehouse/lakehouse so a question maps to a query, not a forensic investigation;\nwrites idempotent, replayable transformation logic (ELT or ETL); enforces quality\nwith tests and contracts; orchestrates the dependency graph so the daily run\nfinishes before dashboards refresh; manages schema evolution so one upstream change\ndoesn't break twelve downstream jobs; and owns compute cost that can bill five\nfigures a month. When a number looks wrong, the data engineer is asked why — and "I\ndon't know" won't hold for long.
\n","wordCount":113},{"heading":"Guiding Principles","id":"guiding-principles","markdown":"- **Idempotency is non-negotiable.** A pipeline you can't re-run can't be operated;\n re-running yesterday's load must reproduce it, not double it.\n- **Make pipelines replayable from source.** Keep raw immutable and partitioned;\n rebuilding any table from raw means recovering from any bug.\n- **Schema is a contract, and contracts break loudly.** Detect drift at the\n boundary; never let a silently-changed upstream column poison a gold table.\n- **Correctness beats freshness beats cost.** Get it right, then timely, then\n cheap. A fast, cheap wrong number is the worst outcome.\n- **Test data like you test code.** Row counts, uniqueness, null rates, and\n referential integrity are assertions, not hopes.\n- **Push computation to the data.** Move the query into the warehouse; don't drag a\n billion rows across the network to filter.\n- **Late-arriving data is the normal case.** Design for events hours late, not as\n an exception.","html":"SELECT * in production transforms — it's how schema drift sneaks in.MERGE without idempotency — upserts that aren't safe to re-run.A data engineer sits between the people who produce data and those who consume it,\nand most friction lives at those two seams. Upstream, software/backend engineers\ndecide whether you get stable schemas or a moving target — the best outcome is a\ndata contract enforced in their CI, not a Slack apology after a break. Downstream,\nanalysts and data scientists are the customers, given clean, well-grained tables.\nData engineering is invisible when it works and blamed when a number is wrong, so\nover-communicate lineage, freshness, and caveats.
\n","wordCount":89},{"heading":"Ethics","id":"ethics","markdown":"Data engineers handle the most sensitive material an organization holds — who its\nusers are, what they did, what they're worth — and build the pipes that make it\ntrivially copyable. The duties: minimize and pseudonymize PII rather than hoard it\nbecause storage is cheap; honor deletion and consent (GDPR, CCPA) as pipeline\nrequirements, tracking where data flows so you can purge it; resist the join that\nre-identifies anonymized records; and be honest about quality, since a wrong number\ndoes more harm than an admitted gap. The hardest cases are where the\neasy pipeline and the right one diverge — combining datasets users never consented\nto, retaining logs \"just in case\" — and those should be named, not defaulted.","html":"Data engineers handle the most sensitive material an organization holds — who its\nusers are, what they did, what they're worth — and build the pipes that make it\ntrivially copyable. The duties: minimize and pseudonymize PII rather than hoard it\nbecause storage is cheap; honor deletion and consent (GDPR, CCPA) as pipeline\nrequirements, tracking where data flows so you can purge it; resist the join that\nre-identifies anonymized records; and be honest about quality, since a wrong number\ndoes more harm than an admitted gap. The hardest cases are where the\neasy pipeline and the right one diverge — combining datasets users never consented\nto, retaining logs "just in case" — and those should be named, not defaulted.
\n","wordCount":116},{"heading":"Scenarios","id":"scenarios","markdown":"**A revenue dashboard doubled overnight.** Finance reports today's revenue is 2x\nyesterday's, with no price change. The expert checks the run log first, not the\ndashboard query, and finds last night's ingestion failed mid-run, was manually\nre-triggered, and the load wasn't idempotent — so a day's events landed twice.\nImmediate fix: rebuild the partition from immutable bronze. In the postmortem, the\nplain `INSERT` becomes an idempotent `MERGE` keyed on event ID with a dedupe step,\nplus a row-count anomaly monitor so the next double-load alerts first.\n\n**\"Make the dashboard real-time.\"** A product manager wants the metrics dashboard\nreal-time. Rather than reaching for Kafka and Flink, the engineer asks what\ndecision the freshness enables — and learns the team reviews it in a standup each\nmorning. The honest requirement is \"fresh by 9 a.m.,\" not \"real-time.\" They\nschedule the hourly batch's final run for 8:30 and redirect the saved streaming\nwork toward the data-quality tests that were missing.\n\n**Onboarding a new third-party source.** A vendor API will feed a new customer\ntable. The engineer profiles it first: the \"unique\" customer ID repeats for ~2% of\nrows, timestamps are in vendor local time with no offset, and deletes are a\nvanishing row, not a tombstone. They land raw to bronze, dedupe on the natural key,\nnormalize to UTC at silver, and detect soft deletes by diffing snapshots —\nbuilding the pipeline to survive the vendor breaking the ID contract, because\nvendors do.","html":"A revenue dashboard doubled overnight. Finance reports today's revenue is 2x\nyesterday's, with no price change. The expert checks the run log first, not the\ndashboard query, and finds last night's ingestion failed mid-run, was manually\nre-triggered, and the load wasn't idempotent — so a day's events landed twice.\nImmediate fix: rebuild the partition from immutable bronze. In the postmortem, the\nplain INSERT becomes an idempotent MERGE keyed on event ID with a dedupe step,\nplus a row-count anomaly monitor so the next double-load alerts first.
"Make the dashboard real-time." A product manager wants the metrics dashboard\nreal-time. Rather than reaching for Kafka and Flink, the engineer asks what\ndecision the freshness enables — and learns the team reviews it in a standup each\nmorning. The honest requirement is "fresh by 9 a.m.," not "real-time." They\nschedule the hourly batch's final run for 8:30 and redirect the saved streaming\nwork toward the data-quality tests that were missing.
\nOnboarding a new third-party source. A vendor API will feed a new customer\ntable. The engineer profiles it first: the "unique" customer ID repeats for ~2% of\nrows, timestamps are in vendor local time with no offset, and deletes are a\nvanishing row, not a tombstone. They land raw to bronze, dedupe on the natural key,\nnormalize to UTC at silver, and detect soft deletes by diffing snapshots —\nbuilding the pipeline to survive the vendor breaking the ID contract, because\nvendors do.
\n","wordCount":246},{"heading":"Related Occupations","id":"related-occupations","markdown":"A data engineer shares the software engineer's discipline of testable,\nversion-controlled code but aims it at data correctness over time, not application\nbehavior. Data scientists and machine-learning engineers are the primary\ndownstream consumers, reasoning in distributions and models while the data engineer\nguarantees the tables they train on. Database administrators are the operational\ncousins tuning the stores pipelines read and write. Backend engineers produce the\noperational data and event streams pipelines ingest, making data contracts a shared\nconcern. Cloud architects design the storage and compute substrate underneath.","html":"A data engineer shares the software engineer's discipline of testable,\nversion-controlled code but aims it at data correctness over time, not application\nbehavior. Data scientists and machine-learning engineers are the primary\ndownstream consumers, reasoning in distributions and models while the data engineer\nguarantees the tables they train on. Database administrators are the operational\ncousins tuning the stores pipelines read and write. Backend engineers produce the\noperational data and event streams pipelines ingest, making data contracts a shared\nconcern. Cloud architects design the storage and compute substrate underneath.
\n","wordCount":89},{"heading":"References","id":"references","markdown":"- *The Data Warehouse Toolkit* — Ralph Kimball & Margy Ross\n- *Designing Data-Intensive Applications* — Martin Kleppmann\n- *Fundamentals of Data Engineering* — Joe Reis & Matt Housley\n- *Streaming Systems* — Akidau, Chernyak & Lax\n- dbt and Apache Airflow documentation","html":"