OT Data for AI: What to Export From the Historian First

Most AI projects on plant data are decided by the first export, long before anyone trains a model. What has to come out of the historian is the recorded values with their quality flags and UTC timestamps, the compression settings of every tag, the asset path that names each tag, and the maintenance, batch and lab records for the same assets and months. An export of interpolated averages in local time, without an asset key, cannot be repaired afterwards.

This post is for the IT manager, OT or automation manager and data lead at a Dutch or Flemish plant of 250 to 5,000 people, in food ingredients, chemicals, tank storage or machine building. Crux Digits does the data work from export to working model next to your OT team and system integrator; we do not run the OT network, the historian or the plant IT.

What should a plant export from the historian before an AI project?

Four things: recorded values with quality flags, the asset model behind each tag, event and batch context, and maintenance and lab labels. Only the first two live in the historian: AVEVA PI System, which AVEVA took over with OSIsoft in March 2021, Inductive Automation Ignition or a Siemens WinCC system. The rest sits in the MES, in SAP PM or Ultimo, and in the LIMS.

In PI the asset model is Asset Framework (AF). AVEVA's licensing schedule describes PI Data Archive as the component of PI Server that stores the time series. An AF element template carries attribute templates whose PI point data reference creates or points to the tag, so the element path says that P101_TT03 is the bearing temperature of pump P101 on line 2. Ask for that path with every tag; without it, the data team spends weeks decoding tag names with your process engineers.

The use case sets the scope. Predictive maintenance needs vibration, temperature, current and run hours per asset plus failure history (predictive maintenance). OEE loss analysis needs states, counts and stop reasons per line, covered on our OEE per line page. Quality prediction needs process values plus lab results, and energy management needs meter tags at the right resolution. Pick one asset group and one question first.

How do PI compression and exception settings delete the signal a model needs?

PI stores a value only when it deviates enough from the previous one, so the deviation settings decide which signal survives. Exception reporting runs in the interface, before data reaches the Snapshot. Compression runs again on the way into the archive, with its own deviation (CompDev), minimum time (CompMin) and maximum time (CompMax).

AVEVA's guidance is a compression deviation of 1 or 2 percent of span for most flows, pressures and levels, 1 or 2 degrees for temperatures, an exception deviation at least a factor 2 smaller, and a maximum time of about one shift, for example 8 hours. That suits trending and reporting, not a model looking for a bearing temperature that creeps up half a degree in a week: with a 2 degree compression deviation, that drift is stored as a near-straight line. AVEVA states that Data Archive never generates events; it only filters them. What was filtered out is gone.

Ignition does the same under other names. Its tag history settings have a Sample Mode (On Change, Periodic or Tag Group), a Deadband Mode, a Historical Deadband and a Deadband Style; the Analog style uses the deadband as a compression threshold in a modified sliding-window algorithm. So the first question to your OT team is not "can we have the data" but "what are CompDev, ExcDev and Step for these tags, or the deadband settings, and when did they last change".

Should the export hold recorded values or interpolated values?

Recorded values, because an interpolated value is calculated by the query and was never measured. PI's Step attribute decides how a query fills the gap. With Step off, 101.0 at 12:00:00 and 102.0 at 12:01:00 return 101.5 at 12:00:30; with Step on, 101.0. AVEVA advises no stepping for continuous signals such as flow meters and stepping for lab data and batch charge weights.

Resample in your own pipeline, where the rule is written down. PI Web API's Stream GetRecorded returns compressed values with a UTC timestamp and the flags Good, Questionable and Substituted. Two defaults catch scripts out: the start time defaults to one day back and maxCount to 1000 values, so a careless script returns one day, or 1000 values, of a two-year request. In OPC UA every DataValue carries a value, a statusCode, a sourceTimestamp and a serverTimestamp; drop the status code and you drop the fact that the sensor was faulty.

How do time zones and daylight saving break a plant data export?

EU summer time ended at 01:00 GMT on Sunday 25 October 2026, so yesterday the local hour from 02:00 to 03:00 happened twice. Directive 2000/84/EC fixes the change for the last Sundays of March and October. An export in local time without an offset therefore holds an hour of duplicate timestamps every October and a missing hour every March, and every join to the MES or SAP PM by timestamp puts events in the wrong hour or drops them.

The sources already work in UTC. OPC UA requires the sourceTimestamp in UTC, the Sparkplug specification requires UTC milliseconds since the epoch, and PI Web API returns timestamps with a Z suffix. The risk is at the edges: PI Web API reads a time string without offset in the time zone of its own server, MES shift calendars are local, and the malfunction start in SAP PM is whatever the planner typed. Rule: every field in UTC, ISO 8601 with offset, converted to local time once, in the report.

Where do the failure labels and the batch context come from?

Failures live in SAP PM or Ultimo as maintenance notifications, not in the historian, so labels need their own export. In SAP PM the breakdown data is the malfunction start and end plus the Breakdown flag, and the system calculates downtime from them. The notification hangs on a functional location or a piece of equipment. Ultimo, the Netherlands-based asset management software IFS agreed to acquire in 2022, holds the same kind of record. You need a mapping from functional location or equipment number to AF element, and it rarely exists until someone builds it.

These are weak labels. The malfunction start is when the failure was noticed, not when the bearing began to fail, and corrective, preventive and inspection orders share one table, so export the notification type and the breakdown flag. A LIMS result has a sample time, a result time and a sample point; only the sample time lines up with process data. The same holds for CCP measurements behind HACCP deviations.

Batch context lets a model tell production from cleaning. An AF event frame has a name, start time, end time, attributes and referenced AF elements, so a batch, a CIP cycle or a stop becomes a searchable record. Without event frames, MES order and batch tables do the job. Without either, a model trained on a filling line learns that cleaning-in-place is an anomaly.

How does plant data leave the OT network without opening a path in?

Through a conduit your OT team controls, using the zones and conduits of IEC 62443 and a DMZ as NIST describes. ANSI/ISA-62443-3-2 defines a conduit as a logical grouping of communication channels that share common security requirements connecting two or more zones, and requires a risk assessment and target security level for each. NIST SP 800-82 Rev. 3 routes all communication between enterprise and operations through services in a DMZ, and names a data diode at that boundary as an option that lets traffic leave the OT network but not enter.

In practice the path is one of three: a historian replica in the DMZ, AVEVA's PI to CONNECT agent into CONNECT data services, or an edge layer such as Ignition, Litmus Edge or Siemens Industrial Edge with WinCC Unified publishing MQTT with Sparkplug B, which is also ISO/IEC 20237. Sparkplug reports by exception, so it filters too, and its NBIRTH message lists every metric with name, datatype and current value: a useful tag inventory. A first historical export needs a read-only, one-way copy, never a new inbound path into the control network.

How much data is a realistic first export?

Worked example, with stated assumptions: 1,500 tags on one line, 18 months on a 10-second grid is about 7.1 billion values. That is 1,500 tags times 8,640 samples a day times 548 days. As CSV rows of tag, timestamp, value and quality at an assumed 60 bytes each, roughly 426 GB; through Stream GetRecorded at the default maxCount of 1000, over 7 million requests. Parquet stores less; how much less depends on the signals.

So measure first: export one week of recorded values for the full tag list and count the rows, which shows what compression kept and what 18 months will weigh. Then start narrow. 150 tags on one asset group for 12 months shows whether the data supports the question, and twelve months always include both clock changes and a full seasonal cycle.

What goes in the export specification, and why do first exports fail?

The specification names the tags, their historian settings, recorded values in UTC, the context records, the label sources, an asset mapping and a manifest. The tag list with AF path, descriptor, units, span and zero. Per tag, exception and compression settings and Step, or Ignition's sample mode and deadband settings, with the date of the last change. Recorded values with quality flags or OPC UA status codes, never interpolated or averaged. UTC timestamps in ISO 8601 with offset.

Then the context: event frames, or MES order, batch, product and recipe records with start and end; MES stop reasons and the shift calendar; SAP PM or Ultimo notifications for the same assets with type, malfunction start and end, breakdown flag, functional location and equipment number; LIMS samples with sample point, sample time, result time, result and limits. Add a mapping from AF element to functional location, MES line and LIMS sample point, a log of sensor replacements and calibrations, and a manifest with row counts per tag per month, so nobody can miss a truncation.

The three reasons a first export is useless, in the order we check them. One: interpolated or averaged values on a coarse grid with unknown compression, so nobody can tell measured from calculated. Two: tags without an asset key, so failure and lab records cannot be joined. Three: local timestamps without offset, so the October hour is doubled, the March hour is missing, and joins to SAP PM drift by an hour for half the year. The pipeline that loads and checks all this is data engineering, and it is most of the project.

Who does what between the OT team, the integrator and Crux Digits?

Your OT team and integrator own the historian, its settings, the conduits and the export path; we touch none of them. Your IT department or ICT partner owns the landing zone and access, maintenance owns SAP PM or Ultimo, and quality owns the LIMS.

Crux Digits does the data work from export to working model. We write the export specification for your plant, check the first export against it (row counts, gaps, quality flags, UTC, compression per tag), build the asset mapping and label set with your engineers, then build and test the model and document what it can and cannot see. Where compression has removed the signal the use case needs, we say so before any model is built and hand your OT team a change request for the tags that matter. The code and the pipeline are yours.

For how we run applied AI projects in fixed steps, see applied AI and ML for mid-sized business.

Last updated: 26 October 2026.

Frequently asked questions