Normal Behavior Model Using Autoencoders
One approach to tackling this problem is to create a model of normal behavior of the asset by training a model using only historical sensor data from all of the normal modes of operation of the asset. If an asset has never failed, this would include all of the past data. Any data from periods of abnormal or failure events will need to be excluded. A model that has learned the normal operation of an asset will be able to indicate when it is beginning to act abnormally which is often a sign of impending failure or suboptimal operation.
A Neural Network Autoencoder described on pages xx-yy is well suited to learn the normal behavior of an asset from historical sensor values from the asset. Remember that an autoencoder attempts to copy its Input to its Output through a constrained coding or latent layer that creates the desired encoding. The diagram of an autoencoder is repeated below. Since the autoencoder is learning X’ from X, the training data is self-labeled. All that is required is to remove any abnormal data from the training set.
The relevant sensors for the asset are the inputs X. The Encoder learns how to compress the input data into normal operating states encoded in the latent space H. The Decoder also learns how to decode the latent space H to reconstruct the inputs as X’. The latent space, H, needs to be as small as possible, but still large enough to represent all the important normal operating states. Statistical analysis of the data (e.g. Principle Component Analysis or PCA) can determine an appropriate value for H. The model is then trained to minimize the differences between X and X’ for all of the training data.
Once trained, live operational data can be fed to the model to predict a new X’. If the error between the predicted X’ and X is small, the asset is most likely operating in a normal operating state that is close to one of the states in the training data. As this prediction error increases, the likelihood that the asset is operating in a state not seen during the training data increases since the model is having difficulty reconstructing the input data. The X’ values with the largest prediction errors also provide important clues to human operators as indicators to what is abnormal about the current operating state which is critical for explainability and for identification of the actions that need to be taken to correct the abnormality.
Wind Turbine Example
Wind Turbines have become an important source of renewable energy and can be seen on the horizon in many places around the world. They also provide a relatively straightforward example for the application of Neural Network Autoencoders to predict pending failure events. When a Wind Turbine fails, it can take weeks to schedule the necessary crane and other equipment required to make the repairs. During that time, all the electricity (and revenue) that Wind Turbine could have produced is lost forever. Thus, predicting pending failure with sufficient warning is critical to maximizing the revenue from a farm of Wind Turbines.
A simple diagram of a Wind Turbine is shown below. They typically consist of three large rotor blades which are pointed into the prevailing wind. The rotors have airfoils similar to the wings on an airplane. The Bernoulli effect across the rotors pulls them around in a circle. This rotates a shaft within the Main Bearing. The Gear
Figure 3.1 Wind Turbine Generator Diagram.
Box translates the slower rotation (RPM) of the rotors to the higher RPM required for efficient electricity generation in the generator. Each of these components within the Wind Turbine can be a source of failure and need to be modeled to predict pending failure. Modeling normal behavior for the main bearing will be used as an example.
For this example, the main bearing temperature sensor will be the primary sensor used to indicate a pending problem with the main bearing. Below are graphs of the air temperature and wind speed near Oakley, Kansas for 2019 from publicly available NOAA weather data and is not from an actual wind farm (though wind farms are plentiful in western Kansas). The air temperature plot shows the annual seasonality trend of winter in January, through summer, and then back to winter in
December. The daily temperature cycle from cooler in the morning to warmer in the afternoon is also visible in this plot.
The wind speed is variable, but not obviously seasonal. A simple spreadsheet simulation of a wind turbine shows that the rotation speed of the turbine (RPM) follows the wind speed except when the wind speed exceeds the upper bound of the rotational capability of the turbine. The RPM is normalized between 0 and 10 for these graphs. The main bearing temperature follows the air temperature but is generally higher due to frictional heating when the turbine is spinning.
This graph plots all four inputs starting in late October 2019- The green and yellow lines show how the RPM tracks the wind speed until the maximum RPM capabilities of the Turbine is reached. The main bearing temperature in red tracks the air temperature but drifts higher when the RPM of the rotor increases the main bearing temperature due to frictional heating. When the high winds of a cold front come in, the main bearing temperature stays noticeably above the air temperature until after the front has passed and the wind speed returns to a more normal range. From there, the RPM decreases and the main bearing temperature again tracks the air temperature.
A Neural Network Autoencoder can be trained to learn all of these relationships simply from learning how to reconstruct these four inputs plus other relevant sensors on the wind turbine such as blade angles, nacelle temperature, vibration sensors, etc. Once the autoencoder has been trained, it can be used to predict these inputs using live data from the wind turbine. If the main bearing begins to suffer mechanical damage, which increases frictional heating, the model will continue to predict the blue line below, but the actual temperature will begin to deviate to the orange values, indicating the need for maintenance activity. Once repaired, the main bearing temperature returns to matching the predicted values.
Autoencoders, by definition, have the same inputs X as outputs X’. However, for normal behavior models of physical assets, some modifications are often useful in industrial applications. For example, in this wind turbine case, accurately predicting the air temperature and wind speed are not relevant to detecting pending issues with the turbine as the operator has no control over the wind or temperature. These important, but exogenous inputs can be provided as a set of inputs Y to the encoder that are not included in the outputs X’ that the decoder is attempting to reconstruct.
Likewise, time-delayed versions of some of the X inputs can be included in Y, allowing the model to learn time-dependencies in the data. For example, the change in RPM is not instantaneous with a change in wind speed due to the momentum of the large rotors. Likewise, the frictional temperature changes also lag behind the changes in RPM or changes in the air temperature and at different rates. Thus, the neural network encoder may have some subset ofXn, Xn l, Xn 2, ... Xn m as well as Y all being fed to the encoder and then used by the decoder to predict X’. This diagram illustrates this concept.
The Wind Turbine example is much simpler than most normal behavior models that would be created for predictive maintenance. A more typical asset would have tens of sensors in X’. In these cases, the signal of abnormal behavior may be contained in the reconstruction error of more than one sensor. Thus, some form of aggregate score using something like Mean Squared Error (MSE) or Hoteling score is used to create a single “abnormality” score. In all cases, the reconstruction error for each sensor is generally a good indicator to the operator of what action to take (e.g. the vibration or temperature is too high).
Given a reasonable set of normal training data, normal behavior models built from neural network autoencoders can be very good at detecting when an asset is behaving differently than it has in the past. However, these models cannot distinguish between abnormal behavior that requires maintenance and an asset that is now operating in a “new normal” state. The latter can happen after repair or maintenance in which new parts or lubrication have changed the relationships between the inputs. If the operator determines that the model is detecting a “new normal,” the model will need to be retrained with samples of this new data before it can become effective again. Periodic retraining is also useful to address the inevitable drift as mechanical parts wear and age.
This Wind Turbine example has shown how a normal behavior model can be developed for an industrial asset using historical sensor data and a Neural Network
Autoencoder (or variants). This model can be used with live sensor data to identify when the asset is deviating from its past normal operation and provide important clues about which sensors are deviating from normal. This information can be used to diagnose and take action on an asset that is in a suboptimal state or trending toward failure before the failure occurs. These types of normal behavior models are an important part of a preventive maintenance system.
Artasanchez A, Joshi P: Artificial Intelligence with Python, 2nd Edition, United Kingdon: Packt Publication; 2020.
Forbes: What is Deep Learning AI? A Simple Guide with 8 Practical Examples; n.d.
Graupe D: Principles of Artificial Neural Networks: Basic Designs to Deep Learning, Singapore: World Scientific Publishing Company; 2019- SAS: “Natural Language Processing (NLP): What It Is and Why It Matters;” n.d.