Persistent Safety Set Guided Offline Safe Reinforcement Learning

IJCAI 2026

Ayan Choudhury^*, Janaka Chathuranga Brahmanage^*, Akshat Kumar, Praveen Paruchuri

^* Ayan Choudhury and Janaka Brahmanage contributed equally to this work.
International Institute of Information Technology, Hyderabad · Singapore Management University

📄 Paper </> Code

Value propagation with vs. without persistent safety. The figure shows a trajectory that briefly leaves the feasible region. Prior methods allow value propagation through unsafe states by enforcing safety only at policy extraction. Our approach enforces safety within the Bellman update, blocking value propagation through unsafe transitions.

Abstract

Offline safe reinforcement learning learns high-return policies that satisfy hard safety constraints using only a pre-collected dataset. This setting is challenging due to the inability to explore, and the risk of propagating value errors through unsafe state-space regions. To address this, first, we characterize the safe state region by developing a framework for learning control barrier functions (CBFs) using a novel generalized Bellman operator, yielding a persistent safety set, from which the agent can remain safe indefinitely. Second, we show that several existing safety set estimation methods (e.g., reachability-constrained RL) can be formulated within our CBF learning framework, highlighting its generality. We further propose a new CBF that ensures safety under environment dynamics uncertainty, unlike standard CBFs designed for deterministic settings. Third, we propose a new reward maximization algorithm that effectively exploits our learned persistent safety set for reward critic estimation. Empirical results on standard benchmarks show that our approach achieves state-of-the-art safety with fewer constraint violations while maintaining competitive returns.

Main Contributions

Generalized Bellman operator for CBF learning: A framework for learning discrete-time CBFs from offline data via a generalized Bellman operator. The operator is a contraction, and its fixed point is the maximal forward-invariant safe set — a valid persistent safety certificate. Several existing CBF formulations (maximum, smooth, additive) are special cases of it.
Robust CBFs under stochastic dynamics: An asymmetric-aggregation (expectile / percentile) update that approximates the worst-case future violation without a transition model, satisfying the robust discrete CBF condition where standard deterministic CBFs fail.
Forward-invariant safety enforcement: The learned CBF is used as an explicit safety critic inside offline actor–critic learning, so both value estimation and policy improvement stay in barrier-certified safe regions. This prevents reward leakage — value propagating through unsafe transitions — instead of enforcing safety only at policy extraction time.
Empirical validation: State-of-the-art constraint satisfaction with competitive returns on Safety Gym, Bullet Gym, and MetaDrive offline safe-RL benchmarks; near-zero cost across tasks where baselines that filter only at policy extraction violate constraints.

Reward Leakage

Reward critic vs safety critic distributions

Reward critic Q_r(s,a) vs. safety critic Q_h(s,a) distributions for Swimmer Velocity. The red dashed line at Q_h(s,a) = 0 marks the safety boundary: points to the left are safe, those to the right are unsafe. Left: CBF_smooth prevents reward leakage — Q_r(s,a) is low when Q_h(s,a) > 0. Right: FISOR shows reward leakage — Q_r(s,a) stays high even for unsafe pairs.