SG-RL researchers develop online MPC weight-adaptation policy on autonomous racing platforms; up to 70.6% fewer samples in tests

SG-RL researchers develop online MPC weight-adaptation policy on autonomous racing platforms; up to 70.6% fewer samples in tests

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter, Felix Jahncke, S´ebastien Gros, Davide Scaramuzza, and Johannes Betz Abstract—In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context- dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solu- tions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias–variance tradeoff: Reinforcement Learning (RL) optimizes realized closed- loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low- variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Re- inforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve sta- bility and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO’s best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments. Index Terms—Learning and Adaptive Systems; Robust/ Adap- tive Control of Robotic Systems; Optimization and Optimal Control; Reinforcement Learning for MPC I. INTRODUCTION Model Predictive Control (MPC) is a powerful framework for constrained optimal control of complex dynamical sys- tems [1]. Its closed-loop performance depends heavily on the formulation of the optimization problem, including the system model, constraints, and cost function. In many applications, the model and constraints are fixed to preserve safety guarantees. The cost function remains a primary mechanism for shaping controller closed-loop behavior and trading off competing objectives [2]. However, a single fixed set of cost-function weights is often insufficient when operating conditions change. Different scenarios may require different trade-offs between objectives, making the relationship between cost weights and closed-loop performance highly context dependent. This ob- servation has motivated the development of Weights-varying MPC approaches [3], [4], [5], which adapt the cost-function weights online while keeping the underlying model and con- straints unchanged. Such adaptation is particularly relevant in safety-critical and highly dynamic applications. A prominent example is the combined nonlinear longitudinal-lateral control problem in autonomous vehicle racing [6], where operating Observations Environment Update MPC System Cost Weights Rewards SG-RL (Ours) Guided RL update RL gradient MPC Solver Gradient Guidance Actor Fig. 1: Solver-Gradient Guided Reinforcement Learning (SG- RL) learns an online MPC weight-adaptation policy πϕ by combining environment-based policy-gradient learning ∇ϕJRL with model-based solver-gradient guidance ∇ϕJSG. The update preserves closed-loop return optimization while using solver sensitivities as bounded, model-informed guid- ance for improved sample efficiency and performance. near the handling limits requires different performance trade- offs across racetrack segments. Learning such policies is challenging because it induces a bilevel optimization structure [7]. Specifically, the return depends on the policy parameters only indirectly, through the numerical solution of a constrained MPC; decisions at one time step affect future states and therefore future MPC so- lutions, resulting in long-horizon credit-assignment problems; the MPC solver introduces implicit, nonlinear, and potentially non-smooth mappings that complicate gradient computation; and the policy must be learned in closed-loop interaction with the controlled system. Existing approaches for MPC cost-weight optimization can broadly be divided into two categories. The first computes a single fixed parameter vector offline, whereas the second learns a state-dependent weight-adaptation policy online. Clas- sical offline tuning methods, such as genetic algorithms and Bayesian optimization [8], [9], [10], [11], can identify useful fixed weights but do not address online adaptation to changing conditions. Since this work addresses online adaptation, we focus on the latter category. In the taxonomy of [12], this corresponds to a hierarchical Reinforcement Learning (RL)– MPC architecture in which the MPC is part of the actor and a neural policy supplies the MPC parameters. Within arXiv:2609.01061v1 [cs.RO] 1 Sep 2026

this setting, recent work distinguishes two complementary sources of learning signal: methods that build environment- based gradients from sampled closed-loop interactions (black- box learning), and methods that compute model-based solver gradients by differentiating through the MPC solver (white- box learning). The first family learns online cost-weight adaptation policies using environment-based gradients. The environment-based baselines considered here construct policy updates from sam- pled closed-loop returns, treating the MPC controller as part of the environment [3], [4], [13], [14]. RL has also been applied more broadly to learning fixed MPC parameters, such as terminal costs or prediction horizons [15], [14], while the concept of Weights-varying MPC introduced learning state-dependent cost-weight adaptation policies [3], [4]. More generally, MPC has been incorporated as a structured policy or value-function approximator within actor-critic architec- tures, deterministic and stochastic policy-gradient methods, economic MPC, learning-based MPC formulations, and joint system-identification and control frameworks [16], [17], [18], [19], [20], [21], [22], [23], [24]. These baselines benefit from long-horizon credit assignment and can use reward signals defined by broad classes of closed-loop objectives, including non-smooth or sparse terms. However, their updates still rely on sampled closed-loop interactions and inherit the usual approximation, finite-horizon, and estimator-bias issues of practical policy-gradient methods. The second family learns MPC parameters using model- based solver gradients obtained from differentiable optimiza- tion. Building on task-based learning through optimization problems [25], differentiable optimization layers [26], [27], and differentiable MPC [28], recent advances enable efficient backpropagation through increasingly general MPC formula- tions, including constrained nonlinear MPC [29], [30]. Until recently, such approaches focused almost exclusively on opti- mizing fixed MPC parameters through solver sensitivities, in- cluding DiffTune-MPC [31], differentiable actor learning [32], cost-matching approaches [33], and ZipMPC [34]. Only very recently, gradient-based policy learning (Gradient-Based Pol- icy Learning (GB-PL)) for nonlinear MPC [5] extended the differentiable nonlinear MPC to learn online cost-weight adap- tation policies. Our GB-PL baseline is built upon [5] and uses a recent backpropagation software feature in acados [29] to efficiently compute solver gradients for nonlinear constrained MPC. In this view, GB-PL plays a cost-matching role: it aligns a differentiable MPC surrogate loss with task-level performance terms on trajectories generated by the predictive model. Unlike environment-based RL updates, these methods exploit local first-order gradients of the differentiable surrogate under solver-regularity assumptions, resulting in substantially higher sample efficiency when the surrogate is informative. Despite their sample efficiency, differentiable-solver meth- ods optimize model-based surrogate objectives whose gradi- ents are only as accurate as the predictive model and therefore may become misleading under model mismatch. Furthermore, they generally lack the exploration mechanisms and long- horizon value estimation that make RL effective for optimizing the performance of the realized closed-loop. Conversely, our environment-based PPO baseline does not use the rich first- order information available from differentiable MPC solvers. The methods compared in this work therefore emphasize one of two complementary information sources: either sampled closed-loop returns or solver sensitivities. Environment-based gradients more directly reflect the realized closed-loop objec- tive but can suffer from high variance, whereas solver gradients provide lower-variance local guidance but inherit bias from the predictive model. Their complementary properties motivate the combination of both within a single learning framework. This gap motivates the central question of this paper: how can environment-based and model-based solver gradients be combined effectively for learning online MPC cost-weight adaptation policies? This question is related to a broader family of guided, struc- tured, and hybrid reinforcement-learning methods that com- bine model-free policy updates with model-based gradients, planning modules, auxiliary losses, or teacher signals [35], [36], [37], [38], [39], [40], [41], [42], [43]. These methods demonstrate that combining complementary gradient estima- tors can substantially improve learning efficiency. However, these works do not use differentiable constrained nonlinear MPC solver sensitivities as structured auxiliary gradients for policy optimization over online MPC cost weights. Within the differentiable MPC literature, solver sensitivities have also been used beyond direct parameter optimization. Adhau et al. [44] employ nonlinear-program sensitivities to ap- proximate action-value quantities in a Q-learning framework, thereby reducing the number of MPC evaluations required dur- ing learning. This approach, however, neither learns an online cost-weight adaptation policy nor uses solver sensitivities to guide policy optimization itself. Romero et al. [32] propagate solver gradients through a differentiable linear MPC layer embedded in a PPO actor for quadrotor control, while related hybrid schemes study differentiable simulation gradients [45] or use RL to provide NMPC reference guidance [46]. In con- trast, our differentiable nonlinear MPC-based work does not replace environment-based policy gradients with model-based solver gradients, nor does it guide the controller only through references. Instead of viewing solver gradients as an alternative optimization objective, we use them as auxiliary signals inside PPO. Consequently, the optimization objective remains the sampled closed-loop return, while solver sensitivities provide local model-informed search directions for online cost-weight adaptation with constrained nonlinear MPC. The contributions of this work are threefold:

1. Solver-gradient Guide for online MPC weight adap- tation. We introduce Solver-Gradient Guided Reinforcement Learning (SG-RL), a framework for online cost-weight adap- tation in constrained nonlinear MPC. Compared to RL, which learns from environment-based policy gradients estimated from sampled closed-loop returns, and differentiable-solver approaches that optimize model-based surrogate objectives using solver sensitivities, SG-RL uses solver-gradient guidance as an auxiliary signal for RL while still optimizing the realized closed-loop return (Sec. VI).
2. Four methods for gui