This means actual control, although most AI applications only make recommendations

For many of us, artificial intelligence is already a part of our everyday lives, whether in the form of smart heating systems that predictively control room temperature according to our wants and needs or smartphone apps that automatically book our next taxi ride. All around us, smart algorithms are already making our lives easier. In industrial applications, the use of AI is often still limited to system observations, predictions, and recommendations for action.

But what about the complex plants and machines produced by Siemens?

Here, too, there’s enormous potential, both for saving money and energy with the aid of clever and predictive control and for driving processes more sustainably. However, the particular challenges posed by the complexity or safety criteria associated with industrial systems as well as by a lack of application experience are still making it difficult to take advantage of the tremendous economic benefits. In this blog post, we’ll explain how this potential can be tapped with the help of AI and what expertise is required.

A typical goal of controlling a heating system, for example, might be to reduce the consumption of heating energy in a conference room.

Until now, such heating systems have mainly been controlled individually or via a building management system. Building management systems follow a preset temperature profile, meaning they always try to adhere to predefined target temperatures. The temperature in a conference room changes in response to environmental influences like sunlight or the number of people present. Simple (PI or PID) controllers are used to make constant adjustments so that the measured room temperature is as close to the target temperature values as possible.

Ensuring a specific and comfortable temperature typically competes with the goal of keeping the consumption of heating energy to a minimum. The subjective assessment of what temperature is perceived as comfortable also varies from person to person.

Even in this simple example, the number of possible control strategies grows large and there is no trivial and static solution that will accommodate all scenarios. In complex industrial plants, which are also subject to strict safety regulations, the range of possible control solutions grows exponentially.

“Internationally, this [building] sector is responsible for almost 40 percent of energy- and process-related [greenhouse gas] emissions, a number that will increase significantly by 2050 when the building inventory is expected to have doubled.” DENA 2021: DENA Gebäudereport 2021

Given the sector’s high energy consumption, it’s worth investing in smart systems that optimize control as an investment in more efficient operation.

Many AI systems are focused on providing forecasts and recommendations. But if the goal is an immediate effect, actual changes need to be made to the “manipulated variables.” Manipulated variables are controller outputs that can be actively changed. In the example of a heating system above, the heating water flow would be a manipulated variable and turning the thermostat up or down would change the manipulated variable. In the industrial environment, changes to manipulated variables are often performed as follows:

Predictions based on simulations or data provide greedy recommendations for adjustments or demonstrate the quantitative effect of changes based on the current state. Subsequent optimization calculations that calculate the effect of multiple and sequential manipulated variable changes using the predictive models – as is the case with model predictive controls (MPCs) – may be optimal in theory. They would, however, also require arbitrarily long series of optimizations (called rollouts) across consecutive system states (e.g. room temperatures).

In addition to the development of a predictive model, this MPC approach requires a significant amount of computational power as close to the machine as possible (edge deployment) to achieve low latencies between state observation and the implementation of manipulated variable changes. This also makes it more difficult to use cloud resources and limits the area of application to slow processes.

We believe that the best alternative is learning a control strategy by means of reinforcement learning (RL). Reinforcement learning is a machine learning method that has no explicit (learning) objective. Instead, an “agent” with as complete a knowledge of the system state as possible learns the manipulated variable changes that maximize a “reward” function defined by humans. Using algorithms from reinforcement learning, the agent, meaning the control strategy, can be trained from both current and recorded system data. This requires measurements for the manipulated variable changes that have been carried out, for the (resulting) changes to the system state over time, and for the variables necessary for calculating the reward.

With our product Gas Turbine Auto Tuner [GTAT, 2020], we demonstrated how this optimization task is performed as a learning problem on historical operational data – in other words, the task is solved in an offline setting. The time-consuming learning process corresponds to the repeated optimization of the MPC and can, for example, also be performed in a local data center or the cloud. Execution of the pre-optimized control strategy requires only a minimum of computing power during operation, which can typically be provided by any edge device.

Given the typically limited amount of available historical operational data, using this capability requires AI expertise and intensive research in reinforcement learning over many years. This is where the mere application of standard RL methods and common open-source packages falls short and fails. Typical challenges include:

The availability of historical operational data, especially in potentially attractive regions of the state and action space.
How to handle the uncertainty that results from a low data density and noisy or inaccurately measured values.
The duration and convergence of the RL process and
The interpretability and verifiability of the resulting control strategy.
Reliable behavior in unknown situations.

Once the optimized control strategy has been developed, productive use of this type of solution also requires technical answers to the following questions:

How can the learned control strategy be adapted to changing requirements?
How can the now improved behavior be used for renewed learning and how can the control strategy be updated easily?
How can the original and AI-based solutions harmonize and complement one another?

There are no out-of-the-box answers to these questions. They require in-depth fundamental research and well thought out practical implementations in collaboration with domain experts.

In the case of limited availability of historical operational data, particularly in desirable regions of the state and action space, one possibility is to directly implement exploration phases that involve more significant changes to relevant manipulated variables. An alternative is to supplement the operational data with simulation data which, for example, is outside the typical “operating range.” After a suitable operating time has elapsed, a preliminary and validated control policy can be updated using the data it itself generated using the same training process. This supports learning and applying the discovered dependencies in operating ranges that have already been optimized. In terms of software technology, this process can be facilitated by means of a good integration of policy training in the cloud in combination with a model container that establishes a clearly defined interface between the control system and the AI model. In addition to verifying measuring units (Celsius/Fahrenheit), such a model container can also serve to verify the sequence, number, or validity of input variables as well as the control policy’s signature or licensing.

A reliable and verifiable implementation of reinforcement learning can be described using the example of a heating system above: The existing heating characteristic curve can be replaced by a “band” within which the control policy can shift the existing target value up or down. The width of this band as well as the rate of change and other criteria can be limited and reliably and verifiably “implemented” in the existing control system. At the same time, this type of AI integration supports a good “collaboration” of the original strategy and the AI strategy because changes to the original and AI strategies can be combined directly.

In practice, problems are often the result of distorted or missing measured values that describe the state space. A rule-based or model-based verification of the state and action spaces makes it possible to confirm a “tuning confidence,” which is required for modifying the original control strategy.

If specific requirements are placed on the interpretability of the AI solution, the RL control strategy can also be selected from a set of interpretable models – in contrast to black box models [Hein, 2020]. In this context, it is interesting that even brand-new, data-based operational knowledge can be extracted and verified by domain experts when control policies are updated.

Thanks to 30 years of intensive research and development work, Siemens AG and Siemens Energy AG have been able to establish an extremely strong patent position based on this technology with over 30 patents. This is also reflected in the World Intellectual Property Organization’s ranking of top players in the fields of neural networks and AI. [WIPO 2019]

The described procedures and the existing references (GT-AT) clearly show that today’s AI-based control systems yield improved results, even in challenging, high-dimensional applications marked by uncertainty. At the same time, they can be implemented reliably and in a standardized process. So, it is worth the effort to begin modernizing hard-to-control or suboptimal and economically important open- and closed-loop control processes to gain a competitive and qualitative edge!

If you want to learn more about the potential for optimization or about our general expertise in the area of industrial artificial intelligence, contact us personally or contact our Siemens AI Lab.

References:

DENA 2021: DENA Gebäudereport 2021
https://www.dena.de/fileadmin/dena/Publikationen/PDFs/2021/dena-GEBAEUDEREPORT_2021_Fokusthemen_zum_Klimaschutz_im_Gebaeudebereich.pdf

GTAT 2020: https://www.siemens-energy.com/global/en/offerings/services/digital-services/gt-autotuner.html

Hein 2020: D. Hein, S. Limmer, and T.A. Runkler. 2020. Interpretable Control by Reinforcement Learning. IFAC-PapersOnLine 53, 2 (2020), 8082–8089. 21th IFAC World Congress. https://doi.org/10.1016/j.ifacol.2020.12.2277

WIPO 2020: https://www.wipo.int/edocs/pubdocs/en/wipo_pub_1055.pdf