Suzuka

SG-RL researchers develop online MPC weight-adaptation policy on autonomous racing platforms; up to 70.6% fewer samples in tests Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter, Felix Jahncke, S´ebastien Gros, Davide Scaramuzza, and Johannes Betz Abstract—In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context- dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solu- tions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias–variance tradeoff: Reinforcement Learning (RL) optimizes realized closed- loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low- variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Re- inforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve sta- bility and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO’s best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments. Index Terms—Learning and Adaptive Systems; Robust/ Adap- tive Control of Robotic Systems; Optimization and Optimal Control; Reinforcement Learning for MPC I. INTRODUCTION Model Predictive Control (MPC) is a powerful framework for constrained optimal control of complex dynamical sys- tems [1]. Its closed-loop performance depends heavily on the formulation of the optimization problem, including the system model, constraints, and cost function. In many applications, the model and constraints are fixed to preserve safety guarantees. The cost function remains a primary mechanism for shaping controller closed-loop behavior and trading off competing objectives [2]. However, a single fixed set of cost-function weights is often insufficient when operating conditions change. Different scenarios may require different trade-offs between objectives, making the relationship between cost weights and closed-loop performance highly context dependent. This ob- servation has motivated the development of Weights-varying MPC approaches [3], [4], [5], which adapt the cost-function weights online while keeping the underlying model and con- straints unchanged. Such adaptation is particularly relevant in safety-critical and highly dynamic applications. A prominent example is the combined nonlinear longitudinal-lateral control problem in autonomous vehicle racing [6], where operating Observations Environment Update MPC System Cost Weights Rewards SG-RL (Ours) Guided RL update RL gradient MPC Solver Gradient Guidance Actor Fig. 1: Solver-Gradient Guided Reinforcement Learning (SG- RL) learns an online MPC weight-adaptation policy πϕ by combining environment-based policy-gradient learning ∇ϕJRL with model-based solver-gradient guidance ∇ϕJSG. The update preserves closed-loop return optimization while using solver sensitivities as bounded, model-informed guid- ance for improved sample efficiency and performance. near the handling limits requires different performance trade- offs across racetrack segments. Learning such policies is challenging because it induces a bilevel optimization structure [7]. Specifically, the return depends on the policy parameters only indirectly, through the numerical solution of a constrained MPC; decisions at one time step affect future states and therefore future MPC so- lutions, resulting in long-horizon credit-assignment problems; the MPC solver introduces implicit, nonlinear, and potentially non-smooth mappings that complicate gradient computation; and the policy must be learned in closed-loop interaction with the controlled system. Existing approaches for MPC cost-weight optimization can broadly be divided into two categories. The first computes a single fixed parameter vector offline, whereas the second learns a state-dependent weight-adaptation policy online. Clas- sical offline tuning methods, such as genetic algorithms and Bayesian optimization [8], [9], [10], [11], can identify useful fixed weights but do not address online adaptation to changing conditions. Since this work addresses online adaptation, we focus on the latter category. In the taxonomy of [12], this corresponds to a hierarchical Reinforcement Learning (RL)– MPC architecture in which the MPC is part of the actor and a neural policy supplies the MPC parameters. Within arXiv:2609.01061v1 [cs.RO] 1 Sep 2026 ...