The Hidden Cost of Undocumented Process Changes

Process historians are reliable recorders of what sensors measured. They are not reliable recorders of why those measurements changed. The gap between those two things is the undocumented process change problem, and it has concrete costs that extend well beyond the inconvenience of missing log entries.

When a process variable shifts in the historian and there is no corresponding entry in the shift log or the DCS event journal, that shift becomes a permanent anomaly in the historical record. Future analysis, whether it is manual troubleshooting or machine learning, cannot distinguish between a shift caused by an operator action, a shift caused by equipment degradation, and a shift caused by a change in feed quality. Those three scenarios have different implications and different appropriate responses. Treating them as undifferentiated data is a compounding error that gets worse as the historical record grows.

How undocumented changes happen

The most common source is the manual setpoint adjustment made during a routine operator scan. The operator sees a process variable drifting toward a limit, bumps the setpoint on the controller, and the process stabilizes. This adjustment is technically a process change, but in the operational flow of a busy shift it does not rise to the level of a loggable event. The operator's mental model says: "I just brought it back to where it should be." The historian says: "Temperature setpoint changed from 167 to 170 at 14:23."

A second source is the informal optimization adjustment. Over time, operators build up knowledge about setpoint regions that work better than the designed operating point. A pressure setpoint that the original commissioning team set at 4.0 bar might be routinely run at 4.2 bar because experience has shown it reduces fouling on a downstream heat exchanger. These adjustments are never formally documented as process changes because they exist in the category of "how we actually run the plant," not "departures from the base case."

A third source is the tuning adjustment made during a brief equipment anomaly. A valve sticks, the operator switches to manual and adjusts the output to compensate, the valve unsticks, and the operator switches back to automatic with the setpoint in a slightly different position than it was before. The DCS event journal may capture the auto-to-manual and manual-to-auto transitions but not the setpoint value at each state. The historian shows the resulting process variable trajectory without an explanation of what caused it.

The cost to behavioral modeling

When you train a world model on historian data that contains undocumented setpoint changes, two things happen. First, the model sees apparent anomalies in process variable behavior at times when, according to the available control variable record, nothing changed. It will try to explain these anomalies through other available inputs, potentially learning spurious correlations between unrelated variables. Second, the model's estimate of how the process responds to a known control change becomes contaminated by the undocumented changes that happened in the vicinity of the training window for that response.

Concretely: if a flow rate setpoint was informally moved from 12.0 to 12.6 tonnes per hour three months before a documented step test at the same setpoint, the model may underestimate the process response to the step test because it is comparing against a steady state that was already different from the nominal. The contamination is subtle and does not produce obvious errors in cross-validation metrics; it shows up as reduced prediction accuracy in operating regimes that the operators know differ from the nominal but the model treats as equivalent.

Detection: what you can infer from the signal

The most reliable way to identify undocumented setpoint changes is to detect step-function-like discontinuities in process variable behavior that cannot be explained by the documented control variable history. A step change in a controlled variable that is not preceded or followed by a corresponding step in the documented setpoint record is a candidate for an undocumented manual action.

The detection algorithm we use looks for sharp transitions in process variable trajectories that have the following characteristics: the transition happens over a time scale shorter than the process time constant for that variable, the new level is sustained for at least several multiples of the time constant, and the magnitude of the transition is larger than the normal noise floor for that measurement. When all three conditions are met without a corresponding logged event, we flag the period as a candidate undocumented change.

False positives come from two sources: genuine equipment anomalies (a pump trips and its downstream pressure drops sharply) and measurement system issues (a transmitter recalibrates or shifts its zero). Distinguishing these from setpoint changes requires context. A step in one process variable accompanied by correlated steps in physically coupled variables suggests an equipment event. A step in one process variable with no corresponding motion in physically coupled variables is more likely a setpoint or local control loop change.

Correction: how we handle flagged periods in training

The conservatively correct treatment of a flagged undocumented change is to exclude the transition window from the training data. The period before the apparent change and the period after it can both be used as valid steady-state training data; the transition itself, and some margin around it, is treated as an unlabeled event that contaminates causal inference if included.

This exclusion approach is safe but wastes information. The transition itself, if correctly labeled as a setpoint change of known magnitude, would be valuable training data for learning the process response to that type of change. The question is whether the apparent magnitude of the change in the process variable can be inverted to estimate the setpoint change that caused it. In some cases it can, with acceptable uncertainty. In other cases, particularly where multiple correlated changes happened simultaneously, the inversion is too ambiguous to be useful.

A practical middle path is to flag the transitions and provide them to the process team for rapid verification. In practice, an experienced operator can often identify a flagged transition within minutes of looking at it: "That was the shift change on March 4th, we adjusted the pressure setpoint because the new feed batch had a different composition." With that annotation in place, the transition becomes labeled training data rather than an anomaly. The verification step adds time but substantially improves training data quality and, as a side effect, often surfaces operational patterns that were not previously documented anywhere.

The longer-term implication for process management

Undocumented process changes are not a data engineering problem in isolation. They reflect a broader gap between the official process parameter record and how the plant is actually run. This gap grows over time as experienced operators accumulate informal knowledge about optimal operating points and pass it on through informal channels rather than formal parameter updates.

A behavioral model that flags undocumented changes as part of its training data characterization inadvertently creates a mechanism for surfacing this gap. When the model identifies that the plant has been consistently running at a setpoint 0.3 bar above the documented nominal for the past eight months, that is information the process engineering team may not have had in one place before. Acting on it may mean formally updating the base case, or it may mean investigating whether the informal operating practice is actually optimal or just habitual.

We are not arguing that plants should instrument every informal operator adjustment at the level of a formal change management process. That would slow down normal operations and push operators away from the kind of adaptive responses that keep production on-spec between formal parameter reviews. What we are saying is that a system that can detect and flag the unlogged changes is genuinely useful, both for training data quality and for keeping the official process knowledge base from diverging too far from the real operating practice.

The historian is a faithful witness. It records what happened. Making sense of what caused it requires a bit more work than the historian was designed to provide.