From SCADA Historian to Learning System: A Practical Path

Every process plant we have worked with has years of sensor history sitting in a SCADA historian. OSIsoft PI, Wonderware Historian, FactoryTalk Historian, Ignition Tag Historian: the brand differs but the pattern is the same. Thousands of tags, often a decade of data, automatically timestamped and stored. Engineers rarely have trouble finding data. The trouble starts when they try to use it for something the historian was never designed for.

A SCADA historian is a record-keeping tool. It answers questions like "what was the pressure at 14:32:07 on March 4th?" with high fidelity. A world model trainer needs something different: a clean, aligned, causally interpretable time series that represents how a machine transitions between states in response to control inputs. That gap between historian output and model input is what this post walks through.

What You Actually Get When You Export Historian Data

The first thing most people notice when they export historian data is the timestamps. They are not evenly spaced. PI archives by exception by default, meaning a tag only writes a new row when the value changes by more than a configured dead-band threshold. A pressure tag sitting at 4.2 bar for six hours might produce exactly one row. A vibration tag cycling through a range might write thousands of rows per minute. You cannot naively resample across the two without understanding which regime each tag sits in.

The second issue is the tag catalog itself. A typical historian installation accumulates tags over years of plant modifications, instrument upgrades, and control system migrations. Tags get renamed. Old tags stay active as read-only aliases. New sensors are added mid-history, leaving gaps in earlier years. Control strategy changes mean that a tag that used to be a manual setpoint becomes a cascade output, fundamentally changing what the value means. Before any preprocessing starts, you need a map of which tags are semantically stable over the time range you want to train on.

Dead-Band Filtering and What It Does to Dynamics

Dead-band filtering is useful for storage efficiency and for reducing noise from stable signals, but it creates a subtle problem for dynamic modeling. Consider a temperature tag with a 0.5 degree dead-band configured. When you resample to a regular grid by forward-filling the last-known value, you get a staircase signal. Every step change in that signal looks like an instantaneous jump followed by a perfectly flat period, regardless of what the underlying physical process was actually doing. If you train a world model on this representation without correction, the model learns that temperature steps instantly and holds, which is physically wrong.

The right approach depends on the signal type. For integrating signals like tank level or accumulated flow, forward-fill is appropriate because the physical quantity does hold its value between changes. For signals that represent continuous physical states (temperature, pressure), the correct treatment is to recognize that the forward-filled value is an approximation of the true value, not the true value, and to handle uncertainty around that region accordingly. In practice this often means excluding the flat post-step-change periods from the training set for those tags, or reweighting them during loss computation.

Resampling to a Regular Time Grid

A world model needs a consistent representation of system state at each time step. This means choosing a sampling interval and resampling all tags to it. Choosing that interval is a domain decision: it should be fine enough to capture the fastest dynamics you care about (typically the fastest control loop response), but not so fine that you amplify noise and inflate your dataset size without adding information. For most process applications we have worked with, a 1-second to 10-second interval is the right range.

Resampling sounds mechanical but the edge cases compound quickly. What do you do with a 4-20mA sensor that was in a bad state (COMM_FAIL status code) for three hours? What do you do with a flow tag that has a legitimate zero value during a shutdown and a missing value during an instrument failure? The historian status codes matter here: most historians store a quality flag alongside the value, and that quality flag is what distinguishes "the sensor read zero because the process was at zero" from "the sensor read zero because the transmitter power was off." If you ignore quality codes, your training data will contain corrupted examples that teach the model systematically wrong dynamics.

Identifying Usable Time Periods

Not all of a historian's time range is useful for training a causal world model. You need periods where the plant was operating in a regime similar to what you want to predict, where the sensors were healthy, and where the control structure was stable (same loops closed, same strategy active). This means you need to find and exclude shutdowns, startup transients, maintenance windows, instrument calibrations, and periods where major process changes were being commissioned.

We do this with a combination of automated signal health checks and manual review with the process engineer who knows the plant history. The automated checks flag periods where too many tags have bad quality codes simultaneously, where key process variables are clearly outside operating range, or where the variance of normally-stable tags spikes anomalously. The manual review catches things the automated checks miss, like a three-week period where the plant was running a different raw material and the dynamics shifted enough that those runs should not be mixed with standard runs in the same training dataset.

Segmenting for Causal Learning

Once you have clean, aligned, continuous segments of operating data, the final step before training is segmentation into episodes that are useful for learning causal transitions. A causal world model learns from transitions: given this state at time t and this control action, what was the state at time t+n? For that to be informative, you need transitions that actually happened in the data. Steady-state periods where the operator made no adjustments for hours tell you almost nothing about how the system responds to a change.

This is why setpoint step records are valuable if the historian captures them. If you can identify precisely when a setpoint was changed and by how much, you have a labeled interventional example: before the change, state was X; operator set u from u0 to u1; the system transitioned to Y over the following T minutes. These are the gold-standard examples for causal model training. Periods between setpoint changes are still useful for learning steady-state behavior, but the transitions are what teach the model how the machine actually responds to control actions.

The Historian Is Not the Problem

We are not saying the historian is bad data. We are saying the historian was designed for record-keeping and for human operators checking recent values, not for training a causal model. Those two design objectives produce different artifacts: compression that saves storage but distorts dynamics, status codes that record instrument health but require explicit handling, timestamp granularity that matches HMI scan rates but not control loop responses. Recognizing these artifacts for what they are is what separates a data pipeline that produces a reliable world model from one that produces a model that looks fine in offline testing and then fails in live deployment.

The path from historian export to training-ready dataset is not glamorous work, but it is where most of the variance in final model quality actually comes from. The modeling architecture matters less than the training data quality for a problem with as much structure as equipment behavior prediction. Getting the data pipeline right is the job, and it is worth spending the time to do it carefully before training starts.