A machine learning engineer exists to make models earn their keep in\nproduction — to turn a method that works in a notebook on last month's data into a\nsystem that serves predictions at scale, under latency budgets, without silently\nrotting. The gap between "scored well in an experiment" and "creates value in\nproduction" is where most ML projects die. The data scientist asks whether the\nsignal is real; the ML engineer asks whether you can serve it 50,000 times a\nsecond, retrain it weekly, roll it back in one move, and know within minutes when\nit predicts garbage. The deliverable is a reliable ML system, not a model.
\n","wordCount":109},{"heading":"Core Mission","id":"core-mission","markdown":"Build and operate machine learning systems that deliver accurate predictions\nreliably in production, retrain and improve safely over time, and fail loudly and\nrecoverably, never silently.","html":"Build and operate machine learning systems that deliver accurate predictions\nreliably in production, retrain and improve safely over time, and fail loudly and\nrecoverably, never silently.
\n","wordCount":26},{"heading":"Primary Responsibilities","id":"primary-responsibilities","markdown":"The romantic image is training state-of-the-art models; the real work is\nplumbing, serving, and operations. An ML engineer builds training pipelines that\nturn raw data into a reproducible artifact, and feature pipelines that compute\nfeatures identically across training and serving. They deploy models behind\nAPIs or batch jobs within latency and cost budgets, and version everything —\ndata, features, code, model, config — so \"which model is in prod?\" always has an\nanswer. They monitor for the failure unique to ML: the model up and fast but\nquietly wrong because the world drifted. And they own the retraining loop — when,\non what data, with what guardrails against a worse model.","html":"The romantic image is training state-of-the-art models; the real work is\nplumbing, serving, and operations. An ML engineer builds training pipelines that\nturn raw data into a reproducible artifact, and feature pipelines that compute\nfeatures identically across training and serving. They deploy models behind\nAPIs or batch jobs within latency and cost budgets, and version everything —\ndata, features, code, model, config — so "which model is in prod?" always has an\nanswer. They monitor for the failure unique to ML: the model up and fast but\nquietly wrong because the world drifted. And they own the retraining loop — when,\non what data, with what guardrails against a worse model.
\n","wordCount":111},{"heading":"Guiding Principles","id":"guiding-principles","markdown":"- **A model is a perishable asset.** Trained on a snapshot of a moving world —\n plan for drift, retraining, and decay.\n- **Training–serving skew is the silent killer.** Compute features differently in\n training and serving and your offline metrics are fiction. Share the code path\n or feature store.\n- **Reproducibility is non-negotiable.** Same data, code, and seed must yield the\n same model, or you can't debug, audit, or roll back.\n- **The model is the easy part.** The data pipeline, serving, and monitoring are\n 90% of the system and 90% of the failures (Sculley, \"Hidden Technical Debt in\n ML Systems\").\n- **Ship the simplest model that clears the bar, then iterate.** A logistic\n regression in prod beats a transformer in a notebook.\n- **Offline metrics are a hypothesis; online metrics are the truth.** A model\n that improves AUC can still tank revenue.\n- **Make every model rollback-able.** A bad deploy is an incident; roll it back\n in one move.","html":"An ML engineer sits between research and operations and speaks both languages.\nWith data scientists, they harden a prototype and push back when it can't be\nserved within budget or reproduced. With data engineers, they share feature\npipelines and feel every upstream schema change. With software engineers, they\nintegrate the model behind an API under the same review discipline; with SREs,\nthey share on-call and observability. The recurring friction is the handoff from\ndata science: a research artifact lands needing the reproducibility, monitoring,\nand rollback it was never built with — good ML engineers push that rigor\nupstream.
\n","wordCount":98},{"heading":"Ethics","id":"ethics","markdown":"ML systems decide about people at a scale and speed no human reviews, which\nconcentrates the cost of a quiet mistake. The duties: monitor for fairness and\ndisparate impact in production, not just at training time, because a model can\ndrift into discrimination as the world shifts; keep a human-meaningful\nexplanation and appeal path for high-stakes decisions; refuse to deploy a model\nwhose data you can't account for or whose failures you can't detect; treat a\nsilently-wrong model as a safety incident. The hardest line is shipping speed\nversus the guardrails that catch the harm — and the engineer who skips them owns\nwhat the model does.","html":"ML systems decide about people at a scale and speed no human reviews, which\nconcentrates the cost of a quiet mistake. The duties: monitor for fairness and\ndisparate impact in production, not just at training time, because a model can\ndrift into discrimination as the world shifts; keep a human-meaningful\nexplanation and appeal path for high-stakes decisions; refuse to deploy a model\nwhose data you can't account for or whose failures you can't detect; treat a\nsilently-wrong model as a safety incident. The hardest line is shipping speed\nversus the guardrails that catch the harm — and the engineer who skips them owns\nwhat the model does.
\n","wordCount":109},{"heading":"Scenarios","id":"scenarios","markdown":"**The model that aced the lab and failed the field.** A fraud model scores 0.94\nAUC offline but barely beats the old rules in production. Suspecting\ntraining–serving skew, the engineer compares feature values across both paths.\nThe training pipeline computed \"average transaction amount\" over a 30-day window\nincluding the current transaction; the serving path excluded it — a leak\ninflating offline scores. Unifying the paths through one feature definition drops\nthe offline number to a realistic 0.86, which production matches.\n\n**Silent drift after a holiday.** A demand-forecasting model that ran clean for\nmonths starts under-predicting badly in late November. No alert fired — latency\nand error rates were fine; the *system* was healthy, only the *predictions* were\nwrong. The fix is twofold: revert to a seasonal heuristic now, then add\nprediction- and input-distribution monitoring with drift alerts. ML monitoring\nmust watch the numbers the model emits, not just whether the server is up.\n\n**The retrain that would have shipped a regression.** The weekly automated\nretrain scores higher on the rolling validation set. The promotion\nguardrail runs it against a fixed golden test set and a fairness slice, and\ncatches that overall accuracy rose but dropped sharply for a minority segment —\nthat week's training data was skewed by a logging bug. The pipeline refuses to\npromote and pages the engineer; the old model stays up, stopping harm at machine\nspeed.","html":"The model that aced the lab and failed the field. A fraud model scores 0.94\nAUC offline but barely beats the old rules in production. Suspecting\ntraining–serving skew, the engineer compares feature values across both paths.\nThe training pipeline computed "average transaction amount" over a 30-day window\nincluding the current transaction; the serving path excluded it — a leak\ninflating offline scores. Unifying the paths through one feature definition drops\nthe offline number to a realistic 0.86, which production matches.
\nSilent drift after a holiday. A demand-forecasting model that ran clean for\nmonths starts under-predicting badly in late November. No alert fired — latency\nand error rates were fine; the system was healthy, only the predictions were\nwrong. The fix is twofold: revert to a seasonal heuristic now, then add\nprediction- and input-distribution monitoring with drift alerts. ML monitoring\nmust watch the numbers the model emits, not just whether the server is up.
\nThe retrain that would have shipped a regression. The weekly automated\nretrain scores higher on the rolling validation set. The promotion\nguardrail runs it against a fixed golden test set and a fairness slice, and\ncatches that overall accuracy rose but dropped sharply for a minority segment —\nthat week's training data was skewed by a logging bug. The pipeline refuses to\npromote and pages the engineer; the old model stays up, stopping harm at machine\nspeed.
\n","wordCount":234},{"heading":"Related Occupations","id":"related-occupations","markdown":"A machine learning engineer is a software engineer who specializes in the\nfailure modes of systems that learn — versioning, testing, and operating code\napplied to artifacts that decay and drift. The data scientist is the closest\nneighbor and upstream partner: they prove the signal and prototype the model; the\nML engineer makes it a reliable production system. Data engineers\nbuild the pipelines both depend on; SREs share the on-call culture; AI safety\nresearchers and prompt engineers work the same models from alignment and\ninterface angles; research scientists supply the methods it productionizes.","html":"A machine learning engineer is a software engineer who specializes in the\nfailure modes of systems that learn — versioning, testing, and operating code\napplied to artifacts that decay and drift. The data scientist is the closest\nneighbor and upstream partner: they prove the signal and prototype the model; the\nML engineer makes it a reliable production system. Data engineers\nbuild the pipelines both depend on; SREs share the on-call culture; AI safety\nresearchers and prompt engineers work the same models from alignment and\ninterface angles; research scientists supply the methods it productionizes.
\n","wordCount":93},{"heading":"References","id":"references","markdown":"- *Designing Machine Learning Systems* — Chip Huyen\n- *Machine Learning Design Patterns* — Lakshmanan, Robinson, Munn\n- *Reliable Machine Learning* — Chen et al. (Google)\n- \"Hidden Technical Debt in Machine Learning Systems\" — Sculley et al., NeurIPS 2015\n- *Designing Data-Intensive Applications* — Martin Kleppmann","html":"