A data engineer builds and maintains the plumbing that gets data from the systems that produce it to the places where it is used: connections, pipelines, storage, scheduling and monitoring. Where an analyst answers the question, an engineer makes sure the data is there, on time and correct, every morning without anyone touching it.
A data engineer builds and maintains the plumbing that gets data from the systems producing it to the places it is used: connections, pipelines, storage, scheduling and monitoring. Where an analyst answers the question, an engineer makes sure the data is there — on time, complete and correct — every morning without anyone touching it.
The work is invisible when it goes well, which is why it is chronically under-resourced. It becomes visible the first Monday a report is empty because an API changed over the weekend and nothing was watching.
Three things, in roughly this order. Ingestion — getting data out of ERP, CRM, webshop and spreadsheets, on a schedule, without hammering the source system. Transformation — turning raw exports into tables that mean something, with the business rules written down rather than living in someone's head. And reliability — monitoring, alerting and retries, so a silent failure becomes a message rather than a wrong decision three weeks later.
An analyst consumes data; an engineer produces it. The clearest test is what each is blamed for: the analyst for a wrong conclusion, the engineer for a missing or late table. In practice the two roles argue productively at the boundary — the analyst wants a column that does not exist yet, the engineer wants to know what it should mean before adding it, and that conversation is where most data quality actually comes from.
Rarely as a full-time hire below fifty people, and often as a few weeks of work. The trigger is not data volume but the number of moving connections: one system feeding one report needs no engineer, while five systems feeding a shared view of the customer needs someone who owns the joins. Many SMEs get this done as a project and then run it internally, which is a reasonable and cheap arrangement.
More than most AI projects admit. A retrieval system or an agent is only as good as the freshness and correctness of what it reads, and keeping that true is data engineering work under a different name. Projects that skip it produce a demo that is right in March and quietly wrong by September — the failure mode described in AI on your own data.
The stack varies, but the categories do not.
Broadly the same lineage, wider modern scope.
Yes, and most SMEs should. Start with the smallest connection that answers a real question.
Want this applied in your business? See how we take it to production:
We build this AI in production — at fixed prices, with one named expert. Start with a free consultation.
Book a free consultation →