Benchmarking Equipment World Model Accuracy: Our Methodology

The question we get asked most often before a pilot is: "How do you know if the model is good enough?" It is a fair question. We are asking engineers to trust a learned model of their machine for decisions that have real consequences. The answer requires a validation methodology that goes well beyond standard ML metrics like mean squared error, because MSE on a holdout set does not tell you whether the model is trustworthy for interventional prediction on live equipment.

This post describes how we actually evaluate a world model before we declare it ready for live prediction use.

Why Standard Holdout Validation Is Not Enough

The standard machine learning validation approach trains on a portion of the data, holds out the rest, and measures prediction error on the holdout set. For a world model that will be used for interventional prediction, this approach has a specific failure mode: it measures prediction accuracy on data drawn from the same observational distribution as the training data, but the model will be used to predict the effects of interventions that may not have occurred in the training data at all.

Consider a model trained on a compressor that historically ran at feed rates between 8 and 14 tonnes per hour. The operator wants to test a setpoint of 15.5 t/h, at the edge of the operating envelope. Holdout MSE on the 8-14 t/h range tells you nothing about model quality at 15.5 t/h. Standard validation also does not test whether the model has learned the causal direction correctly. A model that achieves low MSE by memorizing correlational patterns may produce exactly the wrong prediction for an intervention that breaks those correlational patterns.

Interventional Test Sets

The gold-standard evaluation for a causal world model is an interventional test set: a collection of recorded setpoint change events where the causal effect is known. We build these from the historian by identifying timestamped step changes in control setpoints and the subsequent response trajectories. Each event gives us the pre-change state, the intervention magnitude and direction, and the actual process response over the following time window.

We split these interventional examples strictly by time: all events before a cutoff date go into training, all events after go into the test set. We never use random shuffling for this split because random shuffling would allow the model to learn autocorrelations between adjacent time periods (the state before and after a step change often look similar), which would inflate apparent accuracy. Time-based splits preserve the realistic use case: the model is trained on the past and evaluated on the future.

Within the interventional test set, we stratify by intervention magnitude and direction. A step increase of 1 degree is a different prediction task than a step increase of 10 degrees, and a model might be accurate on small perturbations while failing badly on large ones. We want to see the accuracy curve across the intervention magnitude range so we can define the safe operating envelope for live prediction use.

Metrics That Matter

For each interventional test event, we compute several metrics. The primary metric is response trajectory RMSE: the root mean square error between the predicted and actual process response trajectories over the prediction horizon, computed separately for each output variable. We report this metric at multiple horizons (5 minutes, 15 minutes, 30 minutes) because accuracy degrades with horizon and the decay rate differs by process type.

We also compute calibration metrics for the uncertainty estimates. A well-calibrated 90% prediction interval should contain the true response approximately 90% of the time. We measure empirical coverage rates across the test set and plot calibration curves. Overconfident intervals (80% empirical coverage on a 90% nominal interval) indicate the model is underestimating its own uncertainty, which is the more dangerous failure mode for live use. Overconservative intervals (99% coverage on a 90% nominal interval) reduce the practical usefulness of the predictions but are not safety-critical.

A third metric class is directional accuracy: for each output variable and each test intervention, does the model correctly predict whether the variable increases, decreases, or stays approximately flat? This is a lower bar than trajectory RMSE but it is often the decision-relevant question. An operator who knows the direction of a response can make a safer decision even if the magnitude is uncertain. We report directional accuracy separately from magnitude accuracy because the two can diverge: a model that reliably identifies directions but overestimates magnitudes can be useful with appropriate guidance about interpretation.

Machine-Specific Acceptance Thresholds

We do not use universal accuracy thresholds across machines. A temperature prediction error of 2 degrees C is irrelevant for a batch reactor running at 800 degrees C and critical for a crystallization vessel operating at a 4-degree window above the nucleation temperature. The acceptance threshold for each output variable is set in collaboration with the process engineer who understands what resolution of prediction matters for their decisions.

This means the validation report we produce for a pilot is machine-specific and decision-specific. It shows the model's accuracy for the prediction tasks the operator actually cares about, not generic ML metrics. "For your temperature controller setpoint in the range you typically adjust it, the model predicts the steady-state response to within plus or minus 1.4 degrees C at 30 minutes, 90% of the time based on your last 6 months of setpoint step data" is the kind of statement the report produces. That statement is either satisfactory for the decisions the operator needs to make, or it is not.

Monitoring After Deployment

Validation before deployment answers the question "is this model good enough today?" It does not answer the question "will this model still be good enough in three months?" Equipment dynamics change: bearings wear, heat exchangers foul, catalysts deactivate, feed compositions shift with supplier changes. A model trained on last year's data may produce systematic errors on this year's operating conditions.

We address this with ongoing prediction residual monitoring. After each live prediction event (when the operator asks for a prediction and then makes the change), the system records the actual response and computes the prediction error. These residuals are tracked over time. When rolling residuals on a key output variable exceed the pre-deployment calibration error by a meaningful margin, the system flags the model for retraining review. This is not automatic retraining: the flag goes to the operator for a decision about whether the model behavior has genuinely changed or whether the recent operating window was atypical for other reasons. But it ensures the model is not quietly degrading without anyone noticing.

Benchmarking a world model is more work than evaluating a classification model. The extra effort is justified by the stakes: a prediction that misleads an engineer into making a harmful setpoint change is worse than no prediction at all. The methodology has to be designed around that asymmetry.