---
title: "SG-RL researchers develop online MPC weight-adaptation policy on autonomous racing platforms; up to 70.6% fewer samples in tests"
sdDatePublished: "2026-09-08T16:29:00Z"
source: "https://rpg.ifi.uzh.ch/docs/Arxiv26_Zarrouki.pdf"
topics:
  - name: "artificial intelligence"
    identifier: "medtop:20001298"
  - name: "research and development"
    identifier: "medtop:20000208"
locations:
  - "Suzuka"
  - "Zürich"
  - "Monza"
  - "Modena"
---


SG-RL researchers develop online MPC weight-adaptation policy on autonomous racing platforms; up to 70.6% fewer samples in tests

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC

Accelerating Reinforcement Learning via MPC Solver-Gradient
Guidance for Weights-varying MPC
Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter, Felix Jahncke,
S´ebastien Gros, Davide Scaramuzza, and Johannes Betz
Abstract—In Model Predictive Control (MPC), cost-function
weights shape closed-loop behavior, yet changing conditions often
make fixed parametrizations suboptimal and motivate context-
dependent online adaptation. Learning such policies is difficult
because behavior depends implicitly on numerical MPC solu-
tions, producing nonlinear, potentially nonsmooth, long-horizon
dependencies on policy parameters. This creates a bias–variance
tradeoff: Reinforcement Learning (RL) optimizes realized closed-
loop return from environment samples but is sample-inefficient,
whereas Gradient-Based Policy Learning (GB-PL) uses low-
variance solver gradients from differentiable MPC to optimize
surrogate losses on predicted trajectories but can be biased
under model mismatch. We propose Solver-Gradient Guided Re-
inforcement Learning (SG-RL), a solver-sensitivity augmentation
for RL-based online MPC cost-weight adaptation. SG-RL keeps
sampled closed-loop return as the objective and uses bounded
solver-derived gradients as auxiliary guidance to improve sta-
bility and sample efficiency. We instantiate SG-RL in Proximal
Policy Optimization (PPO) with four modular algorithms that
inject solver-gradient guidance into actor-update scaling, policy
loss, advantage estimation, and value-function learning. On two
full-scale autonomous racing platforms with intentional model
mismatch, SG-RL reaches PPO’s best closed-loop return with
up to 70.6% fewer samples, outperforms GB-PL baselines by
at least 54% in closed-loop return, and generalizes zero-shot to
unseen environments.
Index Terms—Learning and Adaptive Systems; Robust/ Adap-
tive Control of Robotic Systems; Optimization and Optimal
Control; Reinforcement Learning for MPC
I. INTRODUCTION
Model Predictive Control (MPC) is a powerful framework
for constrained optimal control of complex dynamical sys-
tems [1]. Its closed-loop performance depends heavily on the
formulation of the optimization problem, including the system
model, constraints, and cost function. In many applications, the
model and constraints are fixed to preserve safety guarantees.
The cost function remains a primary mechanism for shaping
controller closed-loop behavior and trading off competing
objectives [2]. However, a single fixed set of cost-function
weights is often insufficient when operating conditions change.
Different scenarios may require different trade-offs between
objectives, making the relationship between cost weights and
closed-loop performance highly context dependent. This ob-
servation has motivated the development of Weights-varying
MPC approaches [3], [4], [5], which adapt the cost-function
weights online while keeping the underlying model and con-
straints unchanged. Such adaptation is particularly relevant in
safety-critical and highly dynamic applications. A prominent
example is the combined nonlinear longitudinal-lateral control
problem in autonomous vehicle racing [6], where operating
Observations
Environment
Update
MPC
System
Cost Weights
 Rewards
SG-RL (Ours)
Guided RL update
RL gradient
MPC Solver Gradient
Guidance
Actor
Fig. 1: Solver-Gradient Guided Reinforcement Learning (SG-
RL) learns an online MPC weight-adaptation policy πϕ
by combining environment-based policy-gradient learning
∇ϕJRL with model-based solver-gradient guidance ∇ϕJSG.
The update preserves closed-loop return optimization while
using solver sensitivities as bounded, model-informed guid-
ance for improved sample efficiency and performance.
near the handling limits requires different performance trade-
offs across racetrack segments.
Learning such policies is challenging because it induces
a bilevel optimization structure [7]. Specifically, the return
depends on the policy parameters only indirectly, through the
numerical solution of a constrained MPC; decisions at one
time step affect future states and therefore future MPC so-
lutions, resulting in long-horizon credit-assignment problems;
the MPC solver introduces implicit, nonlinear, and potentially
non-smooth mappings that complicate gradient computation;
and the policy must be learned in closed-loop interaction with
the controlled system.
Existing approaches for MPC cost-weight optimization can
broadly be divided into two categories. The first computes
a single fixed parameter vector offline, whereas the second
learns a state-dependent weight-adaptation policy online. Clas-
sical offline tuning methods, such as genetic algorithms and
Bayesian optimization [8], [9], [10], [11], can identify useful
fixed weights but do not address online adaptation to changing
conditions. Since this work addresses online adaptation, we
focus on the latter category. In the taxonomy of [12], this
corresponds to a hierarchical Reinforcement Learning (RL)–
MPC architecture in which the MPC is part of the actor
and a neural policy supplies the MPC parameters. Within
arXiv:2609.01061v1 [cs.RO] 1 Sep 2026

this setting, recent work distinguishes two complementary
sources of learning signal: methods that build environment-
based gradients from sampled closed-loop interactions (black-
box learning), and methods that compute model-based solver
gradients by differentiating through the MPC solver (white-
box learning).
The first family learns online cost-weight adaptation policies
using environment-based gradients. The environment-based
baselines considered here construct policy updates from sam-
pled closed-loop returns, treating the MPC controller as part
of the environment [3], [4], [13], [14]. RL has also been
applied more broadly to learning fixed MPC parameters, such
as terminal costs or prediction horizons [15], [14], while
the concept of Weights-varying MPC introduced learning
state-dependent cost-weight adaptation policies [3], [4]. More
generally, MPC has been incorporated as a structured policy
or value-function approximator within actor-critic architec-
tures, deterministic and stochastic policy-gradient methods,
economic MPC, learning-based MPC formulations, and joint
system-identification and control frameworks [16], [17], [18],
[19], [20], [21], [22], [23], [24]. These baselines benefit from
long-horizon credit assignment and can use reward signals
defined by broad classes of closed-loop objectives, including
non-smooth or sparse terms. However, their updates still rely
on sampled closed-loop interactions and inherit the usual
approximation, finite-horizon, and estimator-bias issues of
practical policy-gradient methods.
The second family learns MPC parameters using model-
based solver gradients obtained from differentiable optimiza-
tion. Building on task-based learning through optimization
problems [25], differentiable optimization layers [26], [27],
and differentiable MPC [28], recent advances enable efficient
backpropagation through increasingly general MPC formula-
tions, including constrained nonlinear MPC [29], [30]. Until
recently, such approaches focused almost exclusively on opti-
mizing fixed MPC parameters through solver sensitivities, in-
cluding DiffTune-MPC [31], differentiable actor learning [32],
cost-matching approaches [33], and ZipMPC [34]. Only very
recently, gradient-based policy learning (Gradient-Based Pol-
icy Learning (GB-PL)) for nonlinear MPC [5] extended the
differentiable nonlinear MPC to learn online cost-weight adap-
tation policies. Our GB-PL baseline is built upon [5] and uses
a recent backpropagation software feature in acados [29] to
efficiently compute solver gradients for nonlinear constrained
MPC. In this view, GB-PL plays a cost-matching role: it
aligns a differentiable MPC surrogate loss with task-level
performance terms on trajectories generated by the predictive
model. Unlike environment-based RL updates, these methods
exploit local first-order gradients of the differentiable surrogate
under solver-regularity assumptions, resulting in substantially
higher sample efficiency when the surrogate is informative.
Despite their sample efficiency, differentiable-solver meth-
ods optimize model-based surrogate objectives whose gradi-
ents are only as accurate as the predictive model and therefore
may become misleading under model mismatch. Furthermore,
they generally lack the exploration mechanisms and long-
horizon value estimation that make RL effective for optimizing
the performance of the realized closed-loop. Conversely, our
environment-based PPO baseline does not use the rich first-
order information available from differentiable MPC solvers.
The methods compared in this work therefore emphasize one
of two complementary information sources: either sampled
closed-loop returns or solver sensitivities. Environment-based
gradients more directly reflect the realized closed-loop objec-
tive but can suffer from high variance, whereas solver gradients
provide lower-variance local guidance but inherit bias from
the predictive model. Their complementary properties motivate
the combination of both within a single learning framework.
This gap motivates the central question of this paper: how
can environment-based and model-based solver gradients be
combined effectively for learning online MPC cost-weight
adaptation policies?
This question is related to a broader family of guided, struc-
tured, and hybrid reinforcement-learning methods that com-
bine model-free policy updates with model-based gradients,
planning modules, auxiliary losses, or teacher signals [35],
[36], [37], [38], [39], [40], [41], [42], [43]. These methods
demonstrate that combining complementary gradient estima-
tors can substantially improve learning efficiency. However,
these works do not use differentiable constrained nonlinear
MPC solver sensitivities as structured auxiliary gradients for
policy optimization over online MPC cost weights.
Within the differentiable MPC literature, solver sensitivities
have also been used beyond direct parameter optimization.
Adhau et al. [44] employ nonlinear-program sensitivities to ap-
proximate action-value quantities in a Q-learning framework,
thereby reducing the number of MPC evaluations required dur-
ing learning. This approach, however, neither learns an online
cost-weight adaptation policy nor uses solver sensitivities to
guide policy optimization itself. Romero et al. [32] propagate
solver gradients through a differentiable linear MPC layer
embedded in a PPO actor for quadrotor control, while related
hybrid schemes study differentiable simulation gradients [45]
or use RL to provide NMPC reference guidance [46]. In con-
trast, our differentiable nonlinear MPC-based work does not
replace environment-based policy gradients with model-based
solver gradients, nor does it guide the controller only through
references. Instead of viewing solver gradients as an alternative
optimization objective, we use them as auxiliary signals inside
PPO. Consequently, the optimization objective remains the
sampled closed-loop return, while solver sensitivities provide
local model-informed search directions for online cost-weight
adaptation with constrained nonlinear MPC.
The contributions of this work are threefold:
1. Solver-gradient Guide for online MPC weight adap-
tation. We introduce Solver-Gradient Guided Reinforcement
Learning (SG-RL), a framework for online cost-weight adap-
tation in constrained nonlinear MPC. Compared to RL, which
learns from environment-based policy gradients estimated
from sampled closed-loop returns, and differentiable-solver
approaches that optimize model-based surrogate objectives
using solver sensitivities, SG-RL uses solver-gradient guidance
as an auxiliary signal for RL while still optimizing the realized
closed-loop return (Sec. VI).
2. Four methods for gui