Derivative-free optimization for custom loss functions
DOI:
https://doi.org/10.17721/2706-9699.2025.1.07Keywords:
Derivative-free optimization, machine learning, black-box optimizationAbstract
Derivative-free optimization (DFO) has emerged as a powerful technique for solving optimization problems where the gradient of the objective function is either unavailable, expensive to compute, or non-smooth. This article explores the application of DFO methods to optimize custom loss functions in machine learning and other fields. The paper also highlights the challenges and potential improvements in the current DFO approaches, offering insights for further research and practical applications.
References
Kumar S. GD doesn’t make the cut: Three ways that non-differentiability affects neural network training. arXiv:2401.08426. 2024.
Patel Y. Neural Network Training and Non-Differentiable Objective Functions. arXiv:2305.02024. 2023.
Audet C. Derivative-Free Optimization. Springer Handbook of Computational Intelligence. 2021.
Kim B., McKenzie D., Cai H., Yin W. Curvature-Aware Derivative-Free Optimization. Journal of Scientific Computing 2025. Vol. 103. P. 1–12.
Gross J. C., Parks G. T. Optimization by moving ridge functions: Derivative-free optimization for computationally intensive functions. arXiv:2007.04893. 2020.
Roberts L., Kirsch D., Hutter D. DFO-LS: Derivative-Free Optimizer for LeastSquares Minimization. Numerical Algorithms Group 2025.
Anggara D., Suarna N., Wijaya Y. Comparative analysis of Adam, SGD, and RMSprop optimizers performance on the H5 models. Network Engineering Research Operation 2023. Vol. 8. P. 53–64.
Huo Y. Convergence of Adam and RMSprop under Relaxed Smoothness Assumption. 2025.
Liu Y., Pan R., Zhang T. Large Batch Analysis for Adagrad Under Anisotropic Smoothness. arXiv:2406.15244. 2024.
Naaman D., Ahmed B., Ibrahim I. Optimization by Nature: A Review of Genetic Algorithm Techniques. Indonesian Journal of Computer Science 2025. Vol. 14. P. 1–14.
Chen D. Application of Improved Genetic Algorithms in Path Planning. 2024.
Uchida K., Yamaguchi T., Shirakawa S. Covariance Matrix Adaptation Evolution Strategy for Low Effective Dimensionality. arXiv:2412.01156. 2024.
Hansen N. The CMA Evolution Strategy: A Tutorial. arXiv:1604.00772. 2023.
Turner R., Eriksson D., McCourt M., Kiili J., Laaksonen E., Xu Z., Guyon I. Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge. arXiv:2104.10201. 2021.
Rio-Chanona E., Petsagkourakis P., Bradford E., Graciano J., Chachuat B. Real Time Optimization Meets Bayesian Optimization and Derivative-Free Optimization: A Tale of Modifier Adaptation. arXiv:2009.08819. 2021.
Galantai A. A Stochastic Convergence Result for the Nelder–Mead Simplex Method. Mathematics 2023. Vol. 11. P. 1–10.
Selvam M., Manickam R., Saravanan V. Nelder–Mead Simplex Search Method – A Study. Data Analytics and Artificial Intelligence 2022. P. 117–122.
Ragonneau T. PDFO: a cross-platform package for Powell’s derivative-free optimization solvers. Mathematical Programming Computation 2024. Vol. 16. P. 1–25.
Yavuz G. Senior Learning JAYA With Powell’s Method and Incremental Population Strategy. IEEE Access 2022. Vol. 99. P. 1–19.
Klyushin D., Petunin Y. A Nonparametric Test for the Equivalence of Populations Based on a Measure of Proximity of Samples. Ukrainian Mathematical Journal 2003. Vol. 55. P. 147–163.
