NeurIPS 2026

DexDICE: Reconciling Safety and Performance via Dual-Expert Offline Imitation Learning

Seokin Seo1,2 · Sung Kuk Shyn2 · Minji Seo1,3 · Yisoo Lee1 · Jongmin Lee4 · Hongseok Yang5 · Kee-Eung Kim2

1 Korea Institute of Science and Technology (KIST) 2 Korea Advanced Institute of Science and Technology (KAIST) 3 Hyundai Motor Company 4 Yonsei University 5 Korea Institute for Advanced Study (KIAS)
Paper illustration of safe and performant demonstrations and the safe-support-constrained policy.
DexDICE learns high-performing behavior within the estimated support of safe demonstrations.

Abstract

Safe offline imitation learning often requires either a hand-designed cost function or demonstrations that are both safe and high-performing. DexDICE instead learns from two complementary sources: safe demonstrations that may be suboptimal and performant demonstrations that may violate safety guidelines. It aligns with performant behavior while constraining the policy to the estimated support of safe behavior. The paper analyzes safety for the ideal support-constrained solution and evaluates the practical method on a re-curated DSRL benchmark and a physical mobile-robot navigation task trained from human demonstrations.

Method

DexDICE learns from safe demonstrations \(\mathcal{D}_{\mathrm{S}}\), performant demonstrations \(\mathcal{D}_{\mathrm{P}}\), and a larger unlabeled dataset \(\mathcal{D}_{\mathrm{U}}\), with state-action distributions \(d_{\mathrm{S}}\), \(d_{\mathrm{P}}\), and \(d_{\mathrm{U}}\).

Why a support constraint, not distribution matching

A divergence regularizer such as \(D_{\mathrm{KL}}(d \,\|\, d_{\mathrm{S}})\) pulls the policy toward the safe expert, including its suboptimality. DexDICE uses only the support of the safe demonstrations: it restricts where the policy can act, not how often.

\[ \min_{\pi} \; \mathbb{D}\big(d_\pi \,\|\, d_{\mathrm{P}}\big) \quad \text{s.t.} \quad \mathrm{supp}(d_\pi) \subseteq \mathrm{supp}(d_{\mathrm{S}}) \]
Match the performant distribution \(d_{\mathrm{P}}\) while staying inside the support of the safe distribution \(d_{\mathrm{S}}\).

Support-constrained DICE

DICE (stationary DIstribution Correction Estimation) casts policy optimization as a convex problem over stationary state-action distributions \(d\) rather than policies. It learns the correction ratio \(w(s,a) := d(s,a)/d_{\mathrm{U}}(s,a)\), which reweights the offline data toward the target distribution, so training needs only samples from a static dataset and no environment interaction.

DexDICE adds the safe-support constraint to this framework. Because \(d\) sums to one, the constraint is a single linear equality:

\[ \mathrm{supp}(d) \subseteq \mathrm{supp}(d_{\mathrm{S}}) \;\Longleftrightarrow\; \sum_{s,a} \mathbb{1}_{d_{\mathrm{S}}}(s,a)\, d(s,a) = 1 , \]

where \(\mathbb{1}_{d_{\mathrm{S}}}\) is the indicator of \(\mathrm{supp}(d_{\mathrm{S}})\). Adding the Bellman flow constraints gives

\[ \begin{aligned} \min_{d \ge 0} \;\; & D_{\mathrm{KL}}(d \,\|\, d_{\mathrm{P}}) \\ \text{s.t.} \;\; & \sum_{s,a} \mathbb{1}_{d_{\mathrm{S}}}(s,a)\, d(s,a) = 1, \\ & (\mathcal{B}_* d)(s) = (1-\gamma)\,\rho_0(s) + \gamma\,(\mathcal{T}_* d)(s) \quad \forall s . \end{aligned} \]

With multipliers \(\lambda\) (support) and \(\nu(s)\) (Bellman flow), the correction ratio \(w\), and \(r_{\mathrm{P}} := d_{\mathrm{P}}/d_{\mathrm{U}}\), the Lagrangian involves only expectations over \(d_{\mathrm{U}}\):

\[ L(w,\nu,\lambda) = (1-\gamma)\,\mathbb{E}_{s\sim\rho_0}[\nu(s)] - \lambda + \mathbb{E}_{(s,a)\sim d_{\mathrm{U}}}\big[\, w(s,a)\,\big(A_{\nu,\lambda}(s,a) - \log w(s,a)\big) \big], \]
\[ A_{\nu,\lambda}(s,a) := \underbrace{\gamma\,(\mathcal{T}\nu)(s,a) - \nu(s)}_{\text{temporal difference}} + \underbrace{\log r_{\mathrm{P}}(s,a)}_{\text{performance proxy}} + \underbrace{\lambda\, \mathbb{1}_{d_{\mathrm{S}}}(s,a)}_{\text{safe support}} . \]

The maximization over \(w\) has a closed form, leaving a minimization over \(\nu\) and \(\lambda\):

\[ w^*_{\nu,\lambda}(s,a) = \exp\!\big(A_{\nu,\lambda}(s,a) - 1\big), \]
\[ \min_{\nu,\lambda}\; \mathbb{E}_{(s,a)\sim d_{\mathrm{U}}}\big[\exp\!\big(A_{\nu,\lambda}(s,a) - 1\big)\big] + (1-\gamma)\,\mathbb{E}_{s\sim\rho_0}[\nu(s)] - \lambda . \]

Intuition: The term \(\lambda\,\mathbb{1}_{d_{\mathrm{S}}}\) up-weights \(w^*\) inside \(\mathrm{supp}(d_{\mathrm{S}})\) by a factor \(e^{\lambda}\), and \(\log r_{\mathrm{P}}\) steers it toward performant behavior. The policy is extracted from \(w^*\) by weighted behavioral cloning.

Estimating the safe support without unsafe labels

The unsafe distribution \(d_{\mathrm{N}}\) is never observed, so DexDICE trains a safe-vs-unlabeled discriminator \(\hat r_{\mathrm{S}} \approx d_{\mathrm{S}}/d_{\mathrm{U}}\). Writing \(d_{\mathrm{U}} = \eta\, d_{\mathrm{S}} + (1-\eta)\, d_{\mathrm{N}}\) with class prior \(\eta\) gives

\[ \frac{d_{\mathrm{N}}(s,a)}{d_{\mathrm{S}}(s,a)} = \frac{1}{1-\eta}\,\frac{d_{\mathrm{U}}(s,a)}{d_{\mathrm{S}}(s,a)} - \frac{\eta}{1-\eta} . \]

The two ratios are monotonically related, so \(\hat r_{\mathrm{S}}\) ranks pairs in the same order as the safe-vs-unsafe ratio (rank consistency) and can be thresholded without estimating \(\eta\). The threshold is set by a target false-negative rate \(\kappa\):

\[ \tau_\kappa := \mathrm{Quantile}_\kappa\big(\{\hat r_{\mathrm{S}}(s,a) : (s,a)\in\mathcal{D}_{\mathrm{S}}\}\big), \qquad \hat{\mathbb{1}}_{d_{\mathrm{S}}}(s,a) := \mathbb{1}\big\{\hat r_{\mathrm{S}}(s,a) \ge \tau_\kappa\big\} . \]

Fewer than a \(\kappa\) fraction of safe demonstration pairs are rejected. All benchmark results use \(\kappa = 0.01\).

Pipeline summary
  1. Discriminators\(\hat r_{\mathrm{S}} \approx d_{\mathrm{S}}/d_{\mathrm{U}}\), \(\hat r_{\mathrm{P}} \approx d_{\mathrm{P}}/d_{\mathrm{U}}\)
  2. \(\kappa\)-quantile CutoffEstimated safe support \(\hat{\mathbb{1}}_{d_{\mathrm{S}}}\)
  3. DICE CorrectionOptimize \(\nu, \lambda\) to obtain \(w^*\)
  4. Policy ExtractionWeighted BC

Key theoretical result

Under some assumptions, if \(\kappa\) exceeds the boundary mass \(\kappa_0\) by a finite-sample margin, \(\kappa > \kappa_0 + O(|\mathcal{D}_{\mathrm{S}}|^{-1/2})\), the ideal support-constrained policy \(\pi_\kappa\) avoids the reachable unsafe region \(\mathbf{N}\):

\[ d^{\pi_\kappa}(\mathbf{N}) = 0 \quad \text{with probability at least } 1-\delta . \]

This guarantee holds for the ideal policy whose support lies entirely within the estimated safe support.

Experiments

DSRL Benchmark: Re-curated to Dual-Expert IL Setting

With a single fixed \(\kappa\) = 0.01 across all 38 tasks, DexDICE has the highest group-mean return among methods whose group-mean cost satisfies the constraint. It is safe on 31 tasks and has the best safe return on 19, including ties within one standard error.

Group-wise results on the re-curated DSRL benchmark
Task groupDexDICE returnBest group-safe baselineDexDICE costSafe tasks
Safety Gymnasium0.370.27 DemoDICE-S0.7117/21
Bullet Safety Gym0.590.44 FBC, CDT0.637/8
MetaDrive0.500.33 DemoDICE-S0.697/9

Return and cost are normalized group means (safe when cost ≤ 1). “Best group-safe baseline” is the highest-return baseline among those satisfying that group-mean cost threshold. Safe-task counts use each task's cost constraint.

Tasks with the best safe return (out of 38)
  • DexDICE19Safety Gymnasium (9) PointButton1, PointButton2, PointCircle1, PointGoal2, CarButton1, CarButton2, CarPush2, Walker2dVelocity, AntVelocityBullet Safety Gym (5) BallRun, AntRun, CarCircle, DroneCircle, AntCircleMetaDrive (5) easymean, easydense, mediummean, mediumdense, hardsparse
  • DemoDICE-S9Safety Gymnasium (4) PointCircle1, PointCircle2, CarButton1, AntVelocityMetaDrive (5) easydense, mediumsparse, mediummean, hardsparse, hardmean
  • TraC9Safety Gymnasium (3) HopperVelocity, HalfCheetahVelocity, Walker2dVelocityBullet Safety Gym (3) BallRun, CarRun, DroneRunMetaDrive (3) mediumsparse, mediummean, hardsparse
  • CDT8Safety Gymnasium (2) PointGoal2, CarGoal1Bullet Safety Gym (2) BallRun, DroneRunMetaDrive (4) easysparse, easymean, mediumsparse, mediummean
  • FBC5Safety Gymnasium (1) HopperVelocityBullet Safety Gym (2) CarRun, BallCircleMetaDrive (2) easysparse, hardsparse
  • DemoDICE-U5Safety Gymnasium (5) PointButton1, PointPush1, PointPush2, CarGoal1, CarGoal2
  • FISOR-IL4Safety Gymnasium (3) PointGoal1, PointPush1, CarPush1Bullet Safety Gym (1) CarRun
  • DWBC-S3Safety Gymnasium (2) CarGoal1, SwimmerVelocityBullet Safety Gym (1) CarRun
  • JFBC3Safety Gymnasium (1) CarPush1Bullet Safety Gym (1) CarRun
  • BC1MetaDrive (1) harddense

Safe: mean normalized cost within the threshold (cost ≤ 1). Best: highest return among safe methods on a task, including ties within one standard error. Hover over or tap a bar to list the tasks.

Controlled Experiments: Robustness to Data Quantity and Quality

On six Safety Gymnasium tasks, we vary the number of safe or performant demonstrations and the fraction of safe-high demonstrations in each set. Each plot shows average return against average cost; the green region satisfies the cost constraint, and red points are DexDICE.

Average return versus cost as the number of safe demonstrations varies.
Varying the number of safe demonstrations
Average return versus cost as the number of performant demonstrations varies.
Varying the number of performant demonstrations
Average return versus cost as safe-demonstration quality varies.
Varying safe-demonstration quality
Average return versus cost as performant-demonstration quality varies.
Varying performant-demonstration quality

DexDICE generally maintains safety and the highest return among safe methods. Performance weakens with only two safe trajectories or at very low performant-set quality. Select a plot to open it at full size.

Real-world Mobile-robot Navigation

Policies were trained from human teleoperation data and evaluated over ten physical trials per method. DexDICE reached the goal in all ten trials with no measured unsafe-distance time steps.

Top-down illustration of the mobile-robot arena, showing the start, goal, and two obstacles.
Mobile-robot navigation arena from the paper.
Real-robot results, evaluated on 10 trials per method
MethodSuccessTime to goalViolation rate
DexDICE10/1016.9 s0.00%
DemoDICE-U8/1015.9 s0.69%
BC-Safe8/1036.0 s3.81%
FISOR-IL8/1016.9 s4.89%
FBC6/1015.8 s6.38%

Success means reaching the goal without collision. Time to goal is measured on successful trials. Violation rate is the fraction of time steps within the LiDAR-based unsafe-distance threshold.

Robot experiment videos

0:00 / 0:00
All clips start at departure and share one timeline. Shorter clips hold their final frame, and playback restarts after the longest clip ends.
DemoDICE-U
BC-Safe
FISOR-IL
FBC

Takeaways

  1. Support, not distribution. Safe demonstrations constrain where the policy acts, and performant demonstrations determine what it does there. This avoids inheriting the suboptimality of the safe expert.
  2. No unsafe labels, one knob. A safe-vs-unlabeled discriminator and a single quantile level \(\kappa\) estimate the safe support without cost labels or a class prior.
  3. Safe and performant in practice. With a fixed \(\kappa = 0.01\), DexDICE satisfies the cost constraint on 31 of 38 DSRL tasks, and on a physical mobile robot it reaches the goal in 10 of 10 trials with a 0.00% violation rate.

BibTeX

@inproceedings{seo2026dexdice,
  title     = {DexDICE: Reconciling Safety and Performance via Dual-Expert Offline Imitation Learning},
  author    = {Seo, Seokin and Shyn, Sung Kuk and Seo, Minji and Lee, Yisoo and Lee, Jongmin and Yang, Hongseok and Kim, Kee-Eung},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}