Continuous Improvement Toolkit
Public, plant-agnostic helpers for the NeqSim living-task runner
(neqsim task-living, task-cycle, task-solve, task-backtest; see the core
skill neqsim-continuous-task-improvement). Everything here runs without
enterprise access.
When to Use
- A living task must read a PI or IP.21 historian: use
TagreaderAdapter.
- A solve stage tunes a few bounded setpoints and each trial is a NeqSim run:
use
propose_next (GP + expected improvement). Pass its
expected_improvement as the candidate's predicted_delta so the stop rule
can declare converged when no trial is worth its cost.
- Model parameters drift (fouling factor, efficiency, meter bias): update them
every cycle with
enkf_update.
- Before trusting a calibration:
identifiability(J) — a rank below the number
of parameters means some combination is set by noise, not data.
Inputs
TagreaderAdapter: tagreader source, imstype, step_seconds and a
tags map (logical name -> historian tag), plus the runner's watermark window.
propose_next: previous trials X (list of setpoint vectors), results y,
bounds per variable, maximize, and an optional seed.
enkf_update: parameter ensemble (members x parameters), matching model
predictions (members x observations), the measured vector and obs_std.
identifiability: a sensitivity Jacobian (observations x parameters),
optional parameter names and parameter_scale.
Outputs
TagreaderAdapter.fetch(...) returns a SourceResult (ok, partial,
stale, failed, not_installed) and writes a CSV part file to the task's
continuous/data/ folder.
propose_next returns {"x", "expected_improvement", "predicted", "predicted_std", "best_so_far"}.
enkf_update returns the updated ensemble array.
identifiability returns singular_values, condition_number, rank,
n_parameters, an identifiable flag, unidentifiable_directions and a
per-parameter sensitivity_norm.
Engineering Method
- Bayesian optimisation: Gaussian-process surrogate (RBF kernel on the unit
cube) with the expected-improvement acquisition, maximised over random
candidates inside the bounds.
- Parameter tracking: stochastic ensemble Kalman filter with perturbed
observations, using ensemble cross-covariances between parameters and
predictions.
- Identifiability: singular-value decomposition of the scaled Jacobian; a
parameter combination whose singular value is below the noise level is not
determined by the data.
Python Usage Pattern
The sections below show the historian source, a solve stage and a parameter
update as used from a living task.
Historian source in cycle_plan.yaml
sources:
historian:
adapter: "continuous_improvement_toolkit.tagreader_adapter:TagreaderAdapter"
initial_lookback_days: 1
overlap_hours: 2
options:
source: "MY-PI-SERVER" # tagreader data source
imstype: "piwebapi" # or "aspenone"
step_seconds: 3600
tags: # logical name -> historian tag
suction_pressure_bara: "AREA.PT-0001"
discharge_pressure_bara: "AREA.PT-0002"
The runner resolves the dotted path, so no entry point is needed. Without
tagreader installed the source reports not_installed and the cycle is
degraded, not failed. Authentication failures set interaction_required.
Solve stage with Bayesian optimisation
from continuous_improvement_toolkit import propose_next
def run(ctx):
trials = ... # [(x, y), ...] kept in ctx.state_dir
step = propose_next([t[0] for t in trials], [t[1] for t in trials],
bounds=[(40.0, 60.0), (20.0, 35.0)], maximize=True, seed=len(trials))
y = evaluate_with_neqsim(step["x"]) # ProcessAutomation.evaluate(...)
return {"kpis": {"uplift": y},
"solve": {"value": max(y, best), "validated": True, "confidence": "medium",
"candidates": [{"action": "next BO trial", "p_success": 1.0,
"predicted_delta": step["expected_improvement"]}]}}
Parameter update and identifiability
from continuous_improvement_toolkit import enkf_update, identifiability
ensemble = enkf_update(ensemble, predictions, measured, obs_std=[0.2, 0.5])
report = identifiability(jacobian, names=["eta", "fouling"], parameter_scale=[0.01, 1e-4])
if not report["identifiable"]:
... # fix one parameter or add a measurement before calibrating
Validation Checklist
- Backtest the living task (
neqsim task-backtest) before trusting the drift or
solve loop on live data.
- Check the
SourceResult status each cycle; stale or partial data must not
drive a solve step.
- Run
identifiability before accepting an EnKF or calibration result.
- Confirm that a proposed setpoint is inside the bounds and was evaluated with
the NeqSim model (
validated: True) before reporting it.
Common Mistakes
- Mapping logical names to the wrong historian tags or units. Check the
tags
map against the P&ID.
- Reporting
expected_improvement as a realised gain. It is a prediction that
feeds the stop rule.
- Running the EnKF with too few members, which collapses the ensemble spread.
- Calibrating parameters that
identifiability reports as undetermined.
Limitations
- The GP uses a fixed RBF length scale on the unit cube; with fewer than ~5
trials the proposal is close to space-filling. It is a screening optimiser,
not a replacement for NeqSim's
AgenticProcessOptimizer on smooth problems.
- EnKF assumes approximately Gaussian errors; use at least 20 members.
- Nothing here validates a result — validation stays with the NeqSim model and
the backtest.
References
- Jones, D. R., Schonlau, M. and Welch, W. J. (1998). Efficient global
optimization of expensive black-box functions. J. Global Optim. 13, 455-492.
- Evensen, G. (2009). Data Assimilation: The Ensemble Kalman Filter, 2nd ed.
Springer.
- Page, E. S. (1954). Continuous inspection schemes. Biometrika 41, 100-115.
- tagreader-python: https://github.com/equinor/tagreader-python
1---2name: neqsim-continuous-improvement-toolkit3calculation_basis: "advisory"4version: "0.1.0"5description: "Building blocks for NeqSim living tasks: a tagreader PI/IP.21 adapter for cycle_plan.yaml, Gaussian-process Bayesian optimisation with expected improvement, an ensemble Kalman parameter update, and an SVD identifiability check. USE WHEN: a living task needs historian data without enterprise adapters, a solve loop must pick the next trial and its expected gain, parameters must update each cycle from measurements, or a calibration must be checked for unidentifiable parameters."6last_verified: "2026-09-25"7requires:8 python_packages: [numpy]9 java_packages: []10 env: []11 network: []12metadata:13 source: https://skillmd.com/skills/equinor/neqsim-continuous-improvement-toolkit14---1516# Continuous Improvement Toolkit1718Public, plant-agnostic helpers for the NeqSim living-task runner19(`neqsim task-living`, `task-cycle`, `task-solve`, `task-backtest`; see the core20skill `neqsim-continuous-task-improvement`). Everything here runs without21enterprise access.2223## When to Use2425- A living task must read a PI or IP.21 historian: use `TagreaderAdapter`.26- A solve stage tunes a few bounded setpoints and each trial is a NeqSim run:27 use `propose_next` (GP + expected improvement). Pass its28 `expected_improvement` as the candidate's `predicted_delta` so the stop rule29 can declare *converged* when no trial is worth its cost.30- Model parameters drift (fouling factor, efficiency, meter bias): update them31 every cycle with `enkf_update`.32- Before trusting a calibration: `identifiability(J)` — a rank below the number33 of parameters means some combination is set by noise, not data.3435## Inputs3637- `TagreaderAdapter`: tagreader `source`, `imstype`, `step_seconds` and a38 `tags` map (logical name -> historian tag), plus the runner's watermark window.39- `propose_next`: previous trials `X` (list of setpoint vectors), results `y`,40 `bounds` per variable, `maximize`, and an optional `seed`.41- `enkf_update`: parameter ensemble (members x parameters), matching model42 predictions (members x observations), the measured vector and `obs_std`.43- `identifiability`: a sensitivity Jacobian (observations x parameters),44 optional parameter `names` and `parameter_scale`.4546## Outputs4748- `TagreaderAdapter.fetch(...)` returns a `SourceResult` (`ok`, `partial`,49 `stale`, `failed`, `not_installed`) and writes a CSV part file to the task's50 `continuous/data/` folder.51- `propose_next` returns `{"x", "expected_improvement", "predicted",52 "predicted_std", "best_so_far"}`.53- `enkf_update` returns the updated ensemble array.54- `identifiability` returns `singular_values`, `condition_number`, `rank`,55 `n_parameters`, an `identifiable` flag, `unidentifiable_directions` and a56 per-parameter `sensitivity_norm`.5758## Engineering Method5960- Bayesian optimisation: Gaussian-process surrogate (RBF kernel on the unit61 cube) with the expected-improvement acquisition, maximised over random62 candidates inside the bounds.63- Parameter tracking: stochastic ensemble Kalman filter with perturbed64 observations, using ensemble cross-covariances between parameters and65 predictions.66- Identifiability: singular-value decomposition of the scaled Jacobian; a67 parameter combination whose singular value is below the noise level is not68 determined by the data.6970## Python Usage Pattern7172The sections below show the historian source, a solve stage and a parameter73update as used from a living task.7475## Historian source in `cycle_plan.yaml`7677```yaml78sources:79 historian:80 adapter: "continuous_improvement_toolkit.tagreader_adapter:TagreaderAdapter"81 initial_lookback_days: 182 overlap_hours: 283 options:84 source: "MY-PI-SERVER" # tagreader data source85 imstype: "piwebapi" # or "aspenone"86 step_seconds: 360087 tags: # logical name -> historian tag88 suction_pressure_bara: "AREA.PT-0001"89 discharge_pressure_bara: "AREA.PT-0002"90```9192The runner resolves the dotted path, so no entry point is needed. Without93`tagreader` installed the source reports `not_installed` and the cycle is94**degraded**, not failed. Authentication failures set `interaction_required`.9596## Solve stage with Bayesian optimisation9798```python99from continuous_improvement_toolkit import propose_next100101def run(ctx):102 trials = ... # [(x, y), ...] kept in ctx.state_dir103 step = propose_next([t[0] for t in trials], [t[1] for t in trials],104 bounds=[(40.0, 60.0), (20.0, 35.0)], maximize=True, seed=len(trials))105 y = evaluate_with_neqsim(step["x"]) # ProcessAutomation.evaluate(...)106 return {"kpis": {"uplift": y},107 "solve": {"value": max(y, best), "validated": True, "confidence": "medium",108 "candidates": [{"action": "next BO trial", "p_success": 1.0,109 "predicted_delta": step["expected_improvement"]}]}}110```111112## Parameter update and identifiability113114```python115from continuous_improvement_toolkit import enkf_update, identifiability116ensemble = enkf_update(ensemble, predictions, measured, obs_std=[0.2, 0.5])117report = identifiability(jacobian, names=["eta", "fouling"], parameter_scale=[0.01, 1e-4])118if not report["identifiable"]:119 ... # fix one parameter or add a measurement before calibrating120```121122## Validation Checklist123124- Backtest the living task (`neqsim task-backtest`) before trusting the drift or125 solve loop on live data.126- Check the `SourceResult` status each cycle; `stale` or `partial` data must not127 drive a solve step.128- Run `identifiability` before accepting an EnKF or calibration result.129- Confirm that a proposed setpoint is inside the bounds and was evaluated with130 the NeqSim model (`validated: True`) before reporting it.131132## Common Mistakes133134- Mapping logical names to the wrong historian tags or units. Check the `tags`135 map against the P&ID.136- Reporting `expected_improvement` as a realised gain. It is a prediction that137 feeds the stop rule.138- Running the EnKF with too few members, which collapses the ensemble spread.139- Calibrating parameters that `identifiability` reports as undetermined.140141## Limitations142143- The GP uses a fixed RBF length scale on the unit cube; with fewer than ~5144 trials the proposal is close to space-filling. It is a screening optimiser,145 not a replacement for NeqSim's `AgenticProcessOptimizer` on smooth problems.146- EnKF assumes approximately Gaussian errors; use at least 20 members.147- Nothing here validates a result — validation stays with the NeqSim model and148 the backtest.149150## References151152- Jones, D. R., Schonlau, M. and Welch, W. J. (1998). Efficient global153 optimization of expensive black-box functions. J. Global Optim. 13, 455-492.154- Evensen, G. (2009). Data Assimilation: The Ensemble Kalman Filter, 2nd ed.155 Springer.156- Page, E. S. (1954). Continuous inspection schemes. Biometrika 41, 100-115.157- tagreader-python: https://github.com/equinor/tagreader-python