Example · Data Engineer
Being the name on the 3 a.m. page protects you more than any pipeline you have written.
28 of your 40 hours still need you
- 16 stay yours
- 12 faster with AI
- 12 tools could take
A typical week at the middle pace. Your plan uses your own tasks.
Is data engineering safe from AI?
By our task-by-task count, 28 of a typical 40-hour week as a data engineer still need a person. Tools could already take about 12 hours. Turning a clear spec into pipeline code, configuration and tests is about 30% of your week, and a frontier model with your repo open does that today. If you have had one write the pipeline and dbt models for a new source, you have seen it get most of the way.
Read more
The two parts of the job many data engineers like least, the 3 a.m. page and the regulated file that goes out the door, are the two most protected things you do. The spec-to-code work you may be fastest at is the work a model does best. A model cannot be the named person when the number is wrong at 3 a.m., own the file that goes to a regulator or partner, or know which of a source system's status codes lie.
That can put you in a better spot than most people early in software. Two outside reasons explain it. In regulated industries a named human is answerable for the data you ship, and many security teams will not let a model near production. The second reason may not last. The plan turns protection that comes from policy into protection that comes from your own work.
The hard truth
The plumbing you build is becoming a product vendors sell.
Moving and modeling data used to be custom work. Vendors keep packaging it, and in October 2025 two of the biggest agreed to merge.
Sources (1)
- Fivetran and dbt Labs announced an all-stock merger, with about $600 million in combined annual recurring revenue and 10,000+ customers. Technical.ly, 2025-10
What stays yours
Being the person who is paged, and who says what happened afterwards
A runbook can rerun a task. It cannot be the named responder finance escalates to, or write a postmortem a partner will accept. The rotation exists because someone must answer.
Regulated and partner outputs: a credit-bureau furnishing file, a partner-bank reconciliation, finance close feeds
In consumer lending, Reg V (12 CFR 1022.42), the federal rule on furnishing credit data, requires written procedures for the accuracy and integrity of what is furnished to the bureaus, and a partner bank's regulators hold it responsible for your feeds under the 2023 interagency third-party guidance. A wrong file harms consumers and leaves a paper trail, so someone has to sign for it.
Knowing which of the source database's status codes lie
A model reading the schema sees a clean enum. Knowing that CLOSED sometimes means refinanced, and that refinances double-count on the loan key, is in your head, and a wrong answer looks correct until the reconciliation breaks.
"What breaks if we drop this column?"
Lineage tools show edges. They do not show that the finance close, a regulated file and one analyst's forgotten notebook all depend on that column in different ways, or that the app team's PM (product manager) will accept a two-week delay if you ask on Tuesday.
Deciding who may see which columns
The accountability here comes from whoever owns the security program (in financial services, the Safeguards Rule's Qualified Individual). Access reviews and masking policies are a control someone attests to. A model can propose a policy, but it cannot attest to it.
What's next for a data engineer?
Plan A is your best move inside the work you do now. Most sessions in the 13-week plan build toward it.
- A
Own reliability for the whole platform
Why it fits
Why you. Things break on your watch, so you know what breaks, and usually nobody keeps a page log. If you want to stop being the on-call person forever, this is the way out.
The gap. You need to learn SLO design (written reliability targets), error budgets and postmortem practice. You also need to show finance that "the number by 8" is a service level with a cost. The free Google SRE (site reliability engineering) books cover the mechanics.
The trade-off. Your name goes on the outages, including the ones an agent caused. On a small team this is a half-role; it only works paired with the regulated-output reviewer move. And if the company hires a Head of Platform from outside, "reliability" can read as keeping the lights on.
Other moves that fit
These are also good bets in this field. Read them if Plan A doesn't suit you.
Be the reviewer of record for the regulated outputs
Why it fits
Why you. If you already run the regulated file, you know where it goes wrong. Compliance knows your name. Nobody has written the control down.
The gap. You need controls language and deep knowledge of the file specification. For credit furnishing, that is the Credit Reporting Resource Guide from the CDIA, the credit reporting industry group,, which compliance almost certainly licenses. You also need a working relationship with the compliance officer.
The trade-off. It ties you to a regulated domain; a company that does not furnish or report does not have this seat. Regulation keeps the role but sets no number of seats. Usually there is one, and it is yours only if you get there before a vendor settles the question. On paper it is the least glamorous work in the department.
Own the platform for your core source system (the hedge)
Why it fits
Why you. Every quirk you know about the source database is an unwritten data contract that the app team keeps breaking.
The gap. You need to learn data-contract tools and product thinking, and find an app-team lead willing to sign.
The trade-off. Once your context is written down, it protects the company more than you. After the contract exists, the join-key check runs without you. Week 8 bets that authoring it earns the seat first.
What's a plan B for data engineers?
Plan B is a job AI is creating that these skills already fit. One session a week in the plan trains you for it, so it is ready if your job shrinks or you want a change.
BPlan B, picked for this role
LLM evaluation data scientist
You already build the checks that catch bad data before anyone sees it, and scoring model outputs runs on the same pipelines and SQL.
Other options
AI cost and vendor analyst
Your warehouse and compute bills are already yours to explain, so tracking model and token spend is a short step with some finance practice.
Context engineer and knowledge curator
Knowing which source tables are trustworthy and how they join is the core of curating what an AI agent is allowed to know.
In your own plan you can switch to any of the 47 roles.
The first two weeks
The first 2 of 13 weeks. Most weeks have three sessions of 45, 60 and 90 minutes, plus one 40-minute Plan B session, give or take a few minutes. Weeks 4 and 10 are lighter because they assume you're on call, so move them to match your rotation, and Weeks 11 to 13 drop the 90-minute session. By week 13 you'll have a six-month page log with draft SLOs, a written control document for your regulated file, and a one-page memo asking to own platform reliability.
- Week 1: Measure your week
What you do
- ATrack one week of your work against the task list on this page. Write down roughly how many hours went to each task.
- AExport the last six months of incidents from PagerDuty, your paging tool. Label each one by cause (source outage, schema change, bad deploy, capacity or unknown) and by whether a rerun fixed it.
- AWrite down every quirk of the source database that you carry in your head, one line each. Next to each one, note which downstream report or file it affects.
- BTake Anthropic's free AI Capabilities and Limitations course. Write down where AI-written SQL is most likely to go wrong.
- Week 2: Test an AI tool safely
What you do
- APick last month's most boring ticket. Ask an approved AI model to write the DAG (the pipeline's task graph) and the dbt models (the SQL transforms) using made-up rows, and time how long it takes.
- AList every mistake the model made. Sort each one into two groups: needed production data to get right, or wrong anyway.
- ARead chapter 4 (on SLOs, written reliability targets) and chapter 15 (on postmortems) of the free Google SRE book. Write down three ideas you could use on your team.
- BDo the first two sections of the Claude Code in Action course. Have it write dbt tests for a made-up orders table, then review each test.
Three courses from the plan
Checked 2026-09-22.
- Site Reliability Engineering; The Site Reliability WorkbookGoogle, free online at sre.google/books · Free
Yes, Weeks 2, 3 and 6. It is the highest-value reading in this report, and it is free.
- FTC Safeguards Rule: What Your Business Needs to KnowFederal Trade Commission · Free
Yes, Week 4, if you work in financial services.
- 12 CFR 1022.42 and Appendix E (Reg V furnisher accuracy and integrity)eCFR · Free
Yes, Week 5, if your company furnishes to the credit bureaus.
Risks beyond AI: moderatePolicy and cycle risk
Two risks here have nothing to do with what models can do. First, part of your protection is your employer's security policy (no model on production data, no agent with credentials), and policy changes in one meeting. Second, headcount in sectors like lending tracks interest rates and funding. An unfilled junior seat is as likely to come from that cycle as from the AI vision doc.
Where this could be wrong
- The shares are guesses until Week 1. If your log shows much more spec-to-code work, the share of your week tools can do passes 40% and pressure on entry-level jobs crosses 35 out of 100.
- A good part of the work that holds up is protected by policy, and policy is a decision. If security approves an in-boundary model next quarter, re-run this report; diagnosis and triage move first.
- We assumed a regulated employer with an output someone must sign for. If your company has no regulated file, the reviewer-of-record move does not exist and owning the source platform is the plan. If your company is public, SOX IT controls, from the federal financial reporting law, add a second anchor and this report is too pessimistic.
- Our estimate of how well AI diagnoses new kinds of failures is uncertain in both directions. With log and lineage access, agents diagnose most known-pattern failures today; on a change-data-capture (CDC) gap in a database whose status codes lie, the evidence is thin.
Also matched: data engineer, analytics engineer, etl developer. Research checked 2026-09-22. Page updated .