Dynamic programming approach for one linear quadratic optimal control problem with uncertainty
DOI:
https://doi.org/10.17721/2706-9699.2026.1.02Keywords:
optimal control, parabolic differential equations, dynamic programming, polynomial chaos expansionAbstract
This article focuses on solving the linear-quadratic (LQ) optimal control problem for a parabolic partial differential equation (PDE) operating under parametric uncertainty. To manage this uncertainty, we model the unknown parameter via a probability distribution and minimize the expected value of the cost functional.
Applying a Dynamic Programming approach to this system yields an exact optimal state-feedback control law, which is governed by a infinite-dimensional Integro-Differential Riccati Equation (IDRE). Because solving this equation directly is computationally prohibitive, we employ a Spectral Galerkin method combined with a polynomial chaos expansion to approximate the stochastic parameter space. This mathematical transformation reduces the complex, stochastic IDRE into a standard Matrix Riccati Differential Equation (MRDE).
By reducing the stochastic PDE control problem to an MRDE that can be solved in advance, our framework avoids the severe computational bottlenecks typically associated with uncertain environments. This provides a highly efficient baseline for robust controller design and future reinforcement learning implementations.
References
1. Kapustian O., Laptiev O., Makarovych A. Averaging of Linear Quadratic Parabolic Optimal Control Problem. Axioms. 2025. Vol. 14. No. 7. P. 512.
2. Pesare A., Palladino M., Falcone M. Convergence results for an averaged LQR problem with applications to reinforcement learning. Mathematics of Control, Signals, and Systems. 2021. Vol. 33. No. 3. P. 379–411.
3. Alla A., Pacifico A., Palladino M., Pesare A. Online identification and control of PDEs via Reinforcement Learning methods. Advances in computational mathematics. 2024. Vol. 50, No. 4.
4. Sutton R. S., Barto A. G., Williams R. J. Reinforcement learning is direct adaptive optimal control. IEEE Control Systems. 1992. Vol. 12. No. 2. P. 19–22.
5. Sutton R. S., Barto A. G. Reinforcement Learning: An Introduction, 2nd ed. MIT Press, Cambridge, MA, 2018.
6. Recht B. A tour of reinforcement learning: The view from continuous control. Annual Review of Control, Robotics, and Autonomous Systems. 2019. Vol. 2. P. 253–279.
7. Pacifico A., Pesare A., Falcone M. A new algorithm for the LQR problem with partially unknown dynamics. Large-Scale scientific computing. Cham, 2022. P. 322–330.
8. Kirk D. E. Optimal control theory: an introduction. Dover Publications, 2004. 464 p.
9. Ghanem R. G., Spanos P. D. Stochastic finite elements: a spectral approach. Minneola, N.Y : Dover Publications, 2003. 222 p.
10. Xiu D., Karniadakis G. E. The Wiener-Askey polynomial chaos for stochastic differential equations. SIAM Journal on Scientific Computing. 2002. Vol. 24. No. 2. P. 614–644.
11. Gautschi W. Orthogonal Polynomials: Computation and Approximation. Oxford University Press, 2004.
12. Abou-Kandil H., Freiling G., Ionescu V., Jank G. Matrix Riccati Equations in Control and Systems Theory. Birkh¨auser Basel, 2003.
13. Kapustyan E. A., Nakonechnyi A. G. Optimal bounded control synthesis for a parabolic boundary-value problem with fast oscillatory coefficients. Journal of automation and information sciences. 1999. Vol. 31, no. 12. P. 33–44.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 А. В. Макарович, О. А. Капустян
This work is licensed under a Creative Commons Attribution 4.0 International License.
