Abstract
We study how to upper-bound the expected episodic cost of a candidate reinforcement-learning policy A relative to a trusted baseline B, using rollouts but no environment model. On a finite observation alphabet where expected per-step cost is policy-invariant given the symbol, J(A) ≤ J(B) + T TV(d_(A), d_(B)), with no i.i.d. assumption over time. For nondecreasing costs in an ordinal risk level, a one-sided Kolmogorov–Smirnov certificate restricted to cost-onset thresholds can be sharper; its finite-sample empirical-Bernstein slack has no $\sqrt K$ term. We show that cost sufficiency is necessary, cost labels on candidate symbols do not help, and a K-dependent sampling term is necessary for the total-variation certificate at T = 1. On Constrained CartPole, where cost sufficiency holds exactly, no finite-sample certificate was violated in 405 evaluations. At 2,000 episodes, the monotone empirical-Bernstein bound is 4.3–8.8 cost units for policies with realised cost below 0.8 (ceiling 200). On Safety-Gymnasium, a cost-relevant alphabet approximately restores sufficiency on PointGoal1, while a generic k-means alphabet does not. In a retrospective 24-cell validation split, no bound falls below the independent reference mean’s 95% lower confidence limit; the median bound is 4.53 times reference cost, so this is a robustness check rather than a preregistered validation. On one additional PointGoal1 training seed held out from rule development, the candidate mean was 70.96 and the bound was 273.80; this is a case study, not a coverage validation. Across five candidates on that seed, all-threshold bounds were 288--500 and exceeded each independent reference interval; they were looser than pilot-selected bounds. Pilot-size bootstraps on CarGoal1 show that the selected window matches the 100-episode fit in all resamples with 25–100 episodes, but onset selection varies and 50-episode bounds exceed the cost ceiling. CartPole bootstrap screening flags all agents exceeding margins of 1, 5 or 10 cost units, with high false-rejection rates at small sample sizes. A variance-adaptive empirical-Bernstein importance-sampling bound, like the range-only bound, is vacuous at horizon 200 for the policies studied.
Editorial notes
Points the editor recorded on accepting the paper. They are published with the article; the authors were not required to address them before publication.
- Section 2 (Procedure, step 4) still says to fix the onset set and η from baseline pilot data; state there that the all-threshold, η=0 rule is the default and that a pilot-selected set needs a structural justification, and fix the 'orProposition' typo.
- Section 4 (second limitation bullet) should state that the all-threshold rule is now the recommended default and cite its measured looseness on seed 3 (median 340.91 versus 224.03).
- Section 3 cites the seed-3 candidate table as 'Table 3', while the response letter says Table 4 and Section 5 assigns Table 4 to the Safety-Gymnasium results; number all tables consistently.
- Section 5 should name the script and output file that produce the new seed-3 noised-candidate and all-threshold table in the v3 tag, and reconcile the 'experiments/v2' path with the v3 release.
- Section 3 ('Fresh-seed case study') should state explicitly that the threshold-0 result (520.99) uses the 0.05 residual margin, whereas the all-threshold default (499.97) uses η=0, so readers do not read the two as conflicting.
1. Introduction
Constrained reinforcement learning methods such as CPO (Achiam et al. 2017) and Lagrangian PPO (Ray et al. 2019) reduce constraint violations in training, but the resulting policy is evaluated by rolling it out, and a handful of rollouts does not say how its cost relates to that of a controller that is already trusted. A common practical situation is the following. An incumbent controller (a human operator, a legacy controller, or an earlier agent) has a measured expected cost . A new agent is proposed. We have simulator rollouts of both, and we would like an upper bound on that holds before acts in the real system.
This paper studies a family of such bounds and states exactly when each is valid. Our technical tools are elementary; the contributions are the statements, the finite-sample treatment and the measurement of when they are informative:
-
A transfer bound with exact tightness (Theorem 1, Proposition 3). Under a cost-sufficiency assumption, with no i.i.d. assumption; for fixed the right-hand side is the supremum over all admissible cost functions.
-
A sharper bound for monotone costs (Theorem 4, Proposition 8). If cost is nondecreasing in an ordinal risk level and can only begin to rise at known onset thresholds, the bound becomes a one-sided Kolmogorov–Smirnov distance on the risk level, whose finite-sample slack has no term and, with empirical-Bernstein radii, is small when the risky occupancy is small.
-
Limits (Proposition 5, Proposition 11, Proposition 10): cost sufficiency cannot be dropped, cost labels on the agent’s symbols do not help, and a -dependent sampling term is necessary for the total-variation certificate at , with a gap to our slack.
-
An empirical study, including how the alphabet affects the sufficiency assumption, how loose each certificate is, whether it is ever violated, three training seeds on PointGoal1 and one on CarGoal1, and a comparison with importance sampling (Section 3).
We do not claim that the bounds are tight in practice, that they improve on constrained-RL training, or that they remove the need for a cost function: the certificate is cost-agnostic only in the sense that it holds simultaneously for all cost functions in the stated class.
Related work.
Bounds on the change in expected return in terms of total variation between policies underlie conservative policy iteration (Kakade and Langford 2002), TRPO (Schulman et al. 2015) and CPO (Achiam et al. 2017); these constrain training updates. Occupancy-matching analyses in imitation learning also relate expected-cost differences to divergences between occupancy measures (Ho and Ermon 2016); Proposition 3 is the standard integral-probability-metric characterization of total variation (Müller 1997). We do not claim either inequality as new.
The closest work on post-design certification is Vincent et al.’s 2023 preprint and 2024 journal version (Vincent et al. 2023, 2024), which give finite-sample, distribution-free bounds on a policy’s expected cost, VaR, CVaR and sparse failure probability from that policy’s own i.i.d. simulator rollouts, and correct for selecting among candidate policies. Their method directly certifies absolute performance and requires evaluating trajectory costs in a simulator whose uncertainty is representative. Our question is relative: can unlabeled rollouts of a candidate be compared with a trusted baseline when its action probabilities are unavailable? We use baseline costs and candidate/baseline occupancies, under the explicit cost-sufficiency assumption. The methods are complementary; when candidate costs are available and simulator assumptions are acceptable, direct rollout bounds are a simpler alternative.
Risk-sensitive GAIL (RS-GAIL) matches sets of distorted occupancies that encode an expert’s and learner’s risk profiles, using Jensen–Shannon or Wasserstein objectives to train an imitation policy (Lacotte et al. 2019). It is a learning objective for risk-sensitive imitation, not a finite-sample post-deployment bound on expected cost relative to a baseline. Our monotone certificate likewise uses risk-level occupancies, but its purpose is to upper-bound the cost of an already-trained policy; it does not train a policy or compare CVaR risk profiles.
SPoRt (Cloete et al. 2025) is also a close model-free safety result. It bounds a new policy’s probability of violating a temporal property from a safe base policy’s violation bound and a maximum policy ratio, and uses projection during training to preserve that bound. SPoRt addresses event-probability safety and requires policy-ratio control; our result addresses expected additive cost, uses occupancy samples rather than action probabilities, and is post hoc. High-confidence off-policy evaluation (Thomas et al. 2015) instead bounds importance-weighted returns from behavior-policy trajectories and requires action probabilities. Shielding (Alshiekh et al. 2018), Lyapunov methods (Chow et al. 2018) and control barrier functions (Cheng et al. 2019) provide related guarantees through models or explicit unsafe sets. We therefore claim novelty only for the finite-sample occupancy certificates under their stated assumptions and the empirical study of when those assumptions and certificates are useful.
2. Setting and main bound
Setting.
Episodes have a fixed horizon . A discretisation maps each observation to one of symbols in a finite alphabet ; episodes that terminate early are padded with an absorbing symbol so that every episode yields a sequence . Let be the per-episode visitation frequency of episode . The occupancy measure of a policy is . The cost of an episode is with and .
Assumption 1 (Cost sufficiency). There is a function with for every policy , every and every with positive probability. This holds in particular when is a deterministic function of the discretised observation .
No other assumption on the dynamics is made: no Markov property, no i.i.d. structure over time, no stationarity.
Theorem 1 (Cost transfer). Under Assumption 1,
Corollary 2 (Jensen–Shannon form). , hence .
The JSD form is never tighter than the TV form; we include it because JSD is a common divergence for comparing histograms, and we report both in the experiments.
Proposition 3 (Exact tightness over cost functions). For any two occupancy measures on , , attained by .
Proposition 3 says that if nothing is known about the cost function except Assumption 1 and the range , the bound of Theorem 1 cannot be improved: for every pair some admissible cost makes it an equality. It does not say that the bound is tight for the cost function of a particular task.
Theorem 4 (Monotone-cost transfer). Let be an ordinal risk level, a set of onset thresholds and , all fixed before any rollout of . Suppose Assumption 1 holds with , where is nondecreasing and its total rise outside is at most : . Then
where takes the maximum over all thresholds , and . If and , the step function at a maximising attains the bound.
Theorem 4 interpolates between knowing nothing and knowing the cost. With equal to all thresholds it only assumes that cost never decreases with the risk level; with equal to the top threshold and it says that the cost is incurred only in the highest risk level, and the bound becomes the excess occupancy of that level. The slack allows a small rise elsewhere at the price of . Since is the distance between the tail functions of the two risk-level distributions and the class of nondecreasing costs is the class that defines first-order stochastic dominance (Shaked and Shanthikumar 2007; Müller and Stoyan 2002), Theorem 4 is a quantitative form of stochastic dominance of ’s risk level over ’s, and is the integral probability metric generated by monotone step functions. Unlike the TV bound, its finite-sample slack (Proposition 8) has no term.
A robust default for onset selection.
When the cost-onset thresholds are not known from task structure, use all thresholds, and . This avoids selecting an onset set from a finite baseline pilot and requires only that expected cost be nondecreasing in the risk level. A bootstrap union of pilot-plausible onset sets can be reported as a sensitivity analysis, but resampling does not establish that the true cost rise is contained in that union; it is not a substitute for the all-threshold default or for an external structural justification of a narrower set. Neither choice removes the separate cost-sufficiency assumption.
Proposition 5 (Cost sufficiency cannot be dropped). For every there are policies on a two-symbol discretisation with (so ) and when Assumption 1 is dropped.
Corollary 6 (Composition). For a third policy (e.g. an intermediate version of the agent), , with and computable from separate pairs of rollout sets. Total variation satisfies the triangle inequality, so this is never smaller than the direct bound and is useful only when and cannot be rolled out in the same simulator.
Finite samples.
Let be the pooled empirical occupancy measures from and independent episodes. Within an episode the symbols are dependent, so we use episodes, not time steps, as the independent units.
Proposition 7 (Finite-sample certificate). Let costs per step lie in , be the mean cost of the baseline episodes, and
Under Assumption 1, with probability at least ,
The sampling term scales as : certifying a close agent needs episodes, which is why the alphabet should be small. The cost of is not used, so the certificate can be computed from unlabeled rollouts of .
Proposition 8 (Finite-sample monotone certificate). Under the assumptions of Theorem 4, let be the empirical occupancy measures from independent episodes, if and otherwise, and . With probability at least ,
Proposition 9 (Variance-adaptive form). Under the assumptions of Theorem 4, let and be the per-episode tail fractions, , and for a sample let with the unbiased sample variance. Write , with and as in Proposition 8. With probability at least ,
where the last maximum is over all thresholds.
Proposition 10 (Sample-size lower bound for the TV certificate, ). Let , so that an episode is a single symbol, and let a procedure receive independent samples from each of on symbols and output with for every pair (a valid certificate at confidence ). If, in addition, whenever (the certificate is informative for identical policies), then for a universal constant , the second term for .
Proposition 10 concerns certificates that upper-bound the total variation distance, i.e. the certificate of Theorem 1; it shows that the -dependence of that sampling term cannot be removed, but it leaves a gap. The monotone certificate bounds a different, smaller functional (a maximum over at most events) and is not covered. Requiring the slack of Proposition 7 to be at most needs , while the lower bound is ; for fixed the gap is a factor of order . We do not know whether it can be closed for certificates (as opposed to testers). The proposition is stated for , a special case of our setting; for the observations within an episode are dependent and we do not claim a lower bound.
Proposition 11 (Cost labels do not help). Let per-step costs be binary and augment the alphabet with the realised cost bit, so that Assumption 1 holds trivially. For the empirical occupancies, , where . Hence the plug-in bound is at least the direct estimate , and the finite-sample bound of Proposition 7 is at least , the Hoeffding bound computed directly from the labelled episodes of (which does not depend on ).
Proposition 11 explains why we do not try to restore sufficiency by labelling symbols with realised costs: with labels the certificate is never better than evaluating directly, so it is useful only when computed without cost labels of , and the unlabelled alphabet must then carry enough information about the cost, as Proposition 5 requires. In Section 3 we construct such an alphabet from baseline-pilot data alone.
- Procedure. (1) Fix the discretisation and the horizon before looking at rollouts of (fitting on separate baseline rollouts keeps the guarantee valid). (2) Roll out and for independent episodes and count symbols per episode. (3) Compute , and . (4) Report the bound of Proposition 7, together with the ceiling and whether the bound is below it. For the monotone certificate, additionally fix the risk level , the onset set and the slack from baseline pilot data (steps 1–2), and use Proposition 8 orProposition 9 in step 4.
3. Experiments
The experiments are designed to measure (i) whether the certificates are ever violated when their assumptions hold, (ii) how large they are relative to the trivial ceiling and to the realised cost, (iii) how the choice of alphabet affects the cost-sufficiency assumption, and (iv) how they compare with importance sampling. All numbers are produced by the scripts in the experiments/v2 directory (Section 5); the tables are generated from their output files and every number quoted in the text is re-derived from those files by check_numbers.py.
Reported quantities.
is the realised mean cost of on an independent set of episodes (10000 for CartPole, 300 for Safety-Gymnasium). TV, JSD and KS are plug-in bounds with the total variation (Theorem 1), the Jensen–Shannon form (Corollary 2) and the one-sided Kolmogorov–Smirnov distance over all thresholds (Theorem 4); restricts to onset thresholds . These plug-in bounds are not guarantees. Fin. is Proposition 7 (TV) or Proposition 8 (“Hoeff.”), and EB is Proposition 9; all use (failure probability at most ). is with the baseline’s per-symbol cost rate (the known cost function in CartPole); it uses what the cost-agnostic bounds avoid and serves as a reference. The ceiling is (200 for CartPole, 1000 for Safety-Gymnasium); a bound above it carries no information.
Constrained CartPole (assumptions hold exactly)
Setup.
CartPole-v1, horizon . The alphabet is a grid over pole angle and angular velocity plus an absorbing symbol for a fallen pole (). The cost is a function of the symbol: a ramp in the absolute angle of the bin centre (zero below rad, one above rad), and one for the absorbing symbol, so Assumption 1 holds exactly. The risk level is the absolute angle bin (0–4) with the absorbing symbol as level 5; the cost is nondecreasing in it, so Theorem 4 applies with equal to all five thresholds. The baseline is ; the agents are the baseline with a random action with probability , controllers with a smaller angular-velocity gain (0.1, 0.25) and a controller with a bias term (lazy). For each agent we draw 10000 episodes for certificates and an independent 10000 for ; certificates use episodes per policy on disjoint blocks (20, 20 and 5 repetitions).
| Agent | TV | JSD | KS | TV fin. 500 | KS EB 500 | TV fin. 2000 | KS Hoeff. 2000 | KS EB 2000 | ||
|---|---|---|---|---|---|---|---|---|---|---|
| eps0.02 | 0.1 | 0.1 | 1.4 | 11.8 | 0.3 | 81.1 | 17.5 | 41.0 | 21.8 | 4.7 |
| eps0.05 | 0.2 | 0.2 | 3.6 | 20.1 | 1.1 | 83.0 | 18.7 | 43.1 | 22.7 | 5.7 |
| eps0.1 | 0.7 | 0.6 | 8.7 | 32.3 | 3.9 | 87.8 | 22.1 | 48.3 | 25.4 | 8.8 |
| eps0.2 | 4.7 | 4.8 | 28.6 | 62.2 | 19.6 | 107.7 | 40.2 | 68.2 | 41.1 | 25.7 |
| eps0.3 | 20.1 | 19.8 | 62.8 | 97.5 | 51.1 | 141.9 | 74.6 | 102.4 | 72.7 | 58.7 |
| eps0.5 | 95.2 | 95.2 | 142.2 | 166.3 | 131.8 | 221.3 | 155.9 | 181.7 | 153.3 | 139.7 |
| gain0.1 | 0.0 | 0.0 | 0.8 | 9.1 | 0.1 | 80.7 | 16.9 | 40.4 | 21.6 | 4.3 |
| gain0.25 | 0.1 | 0.1 | 0.4 | 1.0 | 0.1 | 80.7 | 17.1 | 40.0 | 21.6 | 4.4 |
| lazy | 0.5 | 0.5 | 29.8 | 32.2 | 1.8 | 109.0 | 19.6 | 69.4 | 23.4 | 6.6 |
CartPole: realised cost and bounds (mean over disjoint samples). The baseline’s cost is . TV, JSD and KS are plug-in at ; the other columns are finite-sample bounds at the stated with . Ceiling: 200.
Validity.
Over the 405 evaluations (27 agent-by- cells; repetitions within a cell use disjoint samples, cells share baseline blocks and are not independent) the TV and JSD plug-in bounds, the finite-sample TV bound, the finite-sample monotone bound (Hoeffding) and the empirical-Bernstein bound were never violated. The plug-in KS bound, which carries no guarantee, was violated in 11 of 405 evaluations, all for agents whose cost is within of the baseline’s and where the bound is dominated by the noise in .
Power to detect excess cost.
We evaluated the EB certificate as a screening rule for the margin : a policy is not certified when its upper bound exceeds the independent reference . We classified policies as truly unsafe using independent 10,000-episode reference pools, and estimated screening rates from 3,000 episode-bootstrap resamples of the separate rollout pools. For cost units, the rule flagged every truly unsafe agent at each (estimated power 1.00). This came with false-rejection rates among safe agents of 1.00 at and ; at the false-rejection rate was 1.00, 0.75 and 0.25 for margins 1, 5 and 10, respectively. At the 10-unit margin, for example, and are unsafe by 20.1 and 95.1 units; three of four safe agents are certified, while the agent (4.6 units above baseline) is not. The certificate has power in this constructed policy family, but at smaller margins its looseness makes the screen uninformative.
Failure when cost sufficiency is violated.
A one-symbol construction isolates the role of Assumption 1: let both policies occupy the same symbol at every step, with , but let a hidden policy-dependent variable make the baseline cost zero and the candidate cost one at every step. With , the empirical TV is zero and the finite-sample TV upper bound is 42.8, below the true . The candidate’s conditional cost given the observed symbol differs from the baseline’s, so Assumption 1 fails; this is not a violation of the theorem. The example makes the failure mode concrete: occupancy cannot detect cost changes hidden by the alphabet.
Non-vacuousness.
The finite-sample TV bound is below the ceiling for 5, 8 and 9 of the 9 agents at , and . The monotone empirical-Bernstein bound is below the ceiling for 8, 9 and 9 agents (range 84.5–232.9 at , 16.9–155.9 at , 4.3–139.7 at ). For the six agents with and it is 4.3–8.8, which is 5 to 11 times below the TV bound; the remaining gap to comes from the baseline-cost term and from the tail-fraction radii.
Price of agnosticism.
The cost-aware estimate is within 0.4 cost units of at . The plug-in TV bound exceeds by a factor of 6–18 for the noise agents with and about 60 for lazy, while the plug-in KS bound exceeds it by a factor of 3–6 for those noise agents and 3.6 for lazy: assuming monotone cost removes most of the looseness for these agents, and what remains is the price of not knowing the cost function.
Comparison with importance sampling (CartPole)
Importance sampling (IS) is the basis of high-confidence off-policy evaluation (Thomas et al. 2015). It needs the action probabilities of both policies and data from the baseline only, whereas our certificates need rollouts of both and no action probabilities. To compare on equal terms we use baseline with noise (), agents with (so both policies have full support and explicit probabilities, ), and baseline episodes.
| SNIS estimate | ESS (of 2000) | max weight | IS EB upper | TV bound | Finite-sample | ||
|---|---|---|---|---|---|---|---|
| 0.05 | 0.21 | 0.78 | 2.2 | 11.9 | 79.6 | 119.1 | |
| 0.1 | 0.71 | 1.04 | 3.5 | 9.7 | 73.8 | 113.4 | |
| 0.2 | 4.94 | 4.49 | 65.8 | 5.0 | 53.3 | 92.9 |
CartPole, importance sampling versus the occupancy certificates, behavior episodes. SNIS: self-normalised IS estimate of . ESS: effective sample size. IS EB upper: ordinary importance-sampling estimate plus an empirical-Bernstein radius at one-sided , using the known range . TV bound: plug-in; Finite-sample: Proposition 7.
The self-normalised IS point estimates are close to (absolute errors 0.57, 0.33, and 0.45 cost units) although the effective sample size is only 2.2, 3.5 and 65.8 of 2000; these point estimates carry no guarantee. We also evaluated a variance-adaptive high-confidence bound: ordinary per-trajectory importance-weighted returns are bounded by , so the empirical-Bernstein inequality (Maurer and Pontil 2009) gives a valid upper bound without truncating weights. At and 95% one-sided confidence, the bounds are , and cost units for , respectively. All exceed the ceiling of 200, so this variance-adaptive bound is also vacuous here, though smaller than the range-only Hoeffding radius (– cost units). This result is specific to the chosen behavior/target policies and does not establish that all variance-adaptive IS bounds are vacuous. Truncating weights can bias an upper cost estimate downward and therefore does not by itself give a valid upper bound. Our certificates are also above : plug-in TV bounds are 53.3–79.6 and finite-sample bounds 92.9–119.1. They bound excess cost relative to the baseline without action probabilities; IS can instead estimate the candidate’s absolute cost when those probabilities are known.
Safety-Gymnasium (assumptions hold approximately)
Setup.
We trained PPO-Lagrangian (Ji et al. 2024) with initial multipliers for 100k steps each on SafetyPointGoal1-v0 (three training seeds) and SafetyCarGoal1-v0 (one seed) (Ji et al. 2023). For each task and seed the policy is the baseline , and the agents are the other four policies and four noisy versions of the baseline (Gaussian action noise ): 8 agents per (task, seed), 32 cells in total. The baselines are not equally safe: , and for the three PointGoal1 seeds and for CarGoal1, so for some seeds many agents are safer than the baseline. Horizon . Certificates use episodes per policy and an independent 300 episodes.
Two alphabets.
Both are fixed from a baseline pilot (50 episodes, half of them with action noise , a value not used for any agent, because the noiseless baselines rarely incur cost) before any agent is rolled out. (a) -means: clusters of the whole observation. (b) Cost-relevant: the window of 16 consecutive observation dimensions whose maximum best predicts the per-step cost on the pilot (window 28 for the three PointGoal1 seeds, with pilot ; window 40 for CarGoal1, ), cut into 8 risk levels at pilot quantiles and crossed with 3 -means clusters of the remaining dimensions (). The onset thresholds are those between consecutive levels at which the pilot cost rate rises by at least 0.05; they are , , for the three PointGoal1 seeds and for CarGoal1. The slack of Theorem 4 is the total pilot rise outside plus a margin of , which gives (0.055 for seed 2). What was fixed before what. The alphabet, onset set and of each run are computed from baseline pilot episodes only, before the agents of that run are analysed. They are not independent of our development, however: PointGoal1 seed 0 was the development run, on which the window-selection rule was designed after inspecting which observation block correlates with cost in the rollouts of a non-baseline agent (); and the rule for replaced a fixed set (chosen on seed 0) after that set failed on CarGoal1. The margin in was likewise chosen after seeing the data. The alphabets of the other three original runs are therefore held out from the design, but nothing is held out for the onset rule or for the margin. We now also report a new PointGoal1 training seed (seed 3), trained after freezing this complete selection rule; this is the only Safety-Gymnasium result here that holds out the task seed for the window, onset and margin together.
Validation split.
We designate PointGoal1 seed 0 as the development run and evaluate the unchanged v1 selection procedure on the other two PointGoal1 seeds and CarGoal1 (24 agent-by-run cells). The pilot-fit, certificate and independent truth pools remain disjoint within each run. All 24 empirical-Bernstein upper bounds exceed their 300-episode reference means; none falls below the reference mean’s 95% lower confidence limit. The median upper-bound/reference-cost ratio is 4.53. This split is retrospective because those results were already collected for the original paper; we therefore treat it as a robustness check, not a preregistered validation.
Fresh-seed case study.
We trained the baseline and candidate on a previously unused PointGoal1 training seed (seed 3), then applied the frozen selection procedure without changing its window scan, seven pilot quantiles, 0.05 onset threshold or 0.05 residual margin. The baseline pilot comprised 50 episodes at each action-noise condition ( and 1.5) and selected window 28, onset set , and . We collected 400 certificate episodes per policy and 300 independent reference episodes per policy using disjoint random seeds; no episodes terminated before the 1000-step horizon. Baseline and candidate reference costs were 33.76 and 70.96, respectively (candidate episode-bootstrap 95% interval 49.47–94.23). The empirical-Bernstein upper bound on the candidate expected cost was 273.80, above both the candidate point estimate and the upper endpoint of its interval. This single held-out seed and candidate is a case study, not a repeated-sampling coverage validation; approximate cost sufficiency also remains a limitation of this environment. With the robust all-threshold default, its upper bound was 499.97 (versus 273.80 under the pilot-selected onset). Thresholds 0.025, 0.05 and 0.10 selected the same onset set and bound; threshold 0 selected all seven boundaries with the 0.05 residual margin and raised the bound to 520.99, while 0.20 selected and gave 328.16.
Noisy-baseline candidates and onset robustness.
Without further training, we added action noise to the held-out baseline. Each candidate has a separate certificate pool of 400 episodes and truth pool of 300 episodes; the same baseline certificate and reference pools are shared across candidates. Table 3 reports the candidate reference mean and episode-bootstrap 95% interval, and both finite-sample empirical-Bernstein upper bounds on . For all five seed-3 candidates (including the trained policy), both rules’ bounds exceed the reference-interval upper endpoint. The all-threshold bounds range from 288.04 to 499.97 (median 340.91), compared with 222.85 to 273.80 (median 224.03) for the pilot-selected onset. This quantifies the added looseness of the default on this one seed; it is not a coverage validation because cost sufficiency is approximate here.
| Candidate | Reference mean (95% CI) | Pilot-selected upper | All-threshold upper |
|---|---|---|---|
| Trained | 70.96 (49.47, 94.23) | 273.80 | 499.97 |
| Baseline + | 39.85 (28.62, 52.65) | 233.24 | 315.65 |
| Baseline + | 41.41 (31.54, 53.14) | 223.08 | 340.91 |
| Baseline + | 40.53 (31.62, 50.38) | 224.03 | 354.39 |
| Baseline + | 41.98 (33.13, 51.95) | 222.85 | 288.04 |
Held-out PointGoal1 training seed 3. Each row reports a candidate reference mean (95% episode-bootstrap interval) and 95% empirical-Bernstein upper bounds on its expected total cost. The pilot-selected column uses onset and ; the robust default uses all seven thresholds and . The trained candidate and four noisy-baseline candidates use 300 independent truth episodes and 400 certificate episodes each. The baseline pools are shared across rows.
Sensitivity to the onset threshold.
Holding each released alphabet and evaluation pool fixed, we varied the rise threshold over and . For each of the three validation runs, the selected onset set was identical throughout this interval: on PointGoal1 seed 1, on seed 2, and on CarGoal1. The resulting EB bounds and point-coverage counts were unchanged. Setting the threshold to zero selected all seven thresholds and increased the median EB bounds from 314 to 394 on seed 1, from 179 to 475 on seed 2, and from 284 to 456 on CarGoal1. Thus the chosen 0.05 threshold is stable over this range, while including every threshold materially loosens the certificate.
Sensitivity to pilot size.
We collected a fresh CarGoal1 pilot with 50 baseline episodes in each condition (no action noise and ), then bootstrapped episodes to compare total pilot sizes 10, 25, 50 and 100 (20 resamples per size). Relative to the 100-episode reference fit, the selected observation window matched in 85% of 10-episode fits and every fit with at least 25 episodes; the median absolute error of the seven risk-bin edges was 0.021, 0.009, 0.009 and 0.006 observation units, respectively. At size 50 and rise threshold 0.05, 13/20 resamples selected onset , 5/20 selected , and the remaining two selected other sets. The median certificate was about 1,314 at pilot size 50 and 1,311 at size 100, above the ceiling of 1,000 because this sensitivity run used only 50 fresh certificate episodes per policy. These bootstrap results quantify selection variability for this task; they are not a coverage test. On the new PointGoal1 seed-3 pilot, the selected window matched the 100-episode fit in 19/20 bootstrap fits at total size 10 and 20/20 at sizes 25, 50 and 100. Median absolute edge errors were 0.0197, 0.0143, 0.0078 and 0.0047. At total size 50, the onset set was in 8/20 fits, in 8/20, and another set in 4/20. Thus the selected window was stable in this pilot, while the onset selection remained sensitive to which episodes were sampled.
| Agent | (k-means) | (risk) | TV | KS | KS | TV fin. | KS Hoeff. | KS EB | |
|---|---|---|---|---|---|---|---|---|---|
| lam0.1 | 73.1 | 10.2 | 83.4 | 485 | 482 | 115 | 894 | 369 | 276 |
| lam0.5 | 72.8 | 9.8 | 86.8 | 503 | 491 | 118 | 911 | 372 | 296 |
| lam1.0 | 85.1 | 9.9 | 102.5 | 603 | 603 | 140 | 1012 | 395 | 317 |
| lam2.0 | 115.1 | 10.8 | 116.8 | 502 | 502 | 153 | 911 | 408 | 323 |
| basenoise0.1 | 8.8 | 6.2 | 9.2 | 68 | 46 | 12 | 477 | 266 | 155 |
| basenoise0.3 | 7.1 | 6.9 | 6.7 | 79 | 34 | 8 | 488 | 263 | 149 |
| basenoise0.6 | 6.3 | 9.0 | 8.4 | 189 | 68 | 12 | 598 | 267 | 150 |
| basenoise1.0 | 9.0 | 11.1 | 9.2 | 325 | 75 | 13 | 734 | 268 | 153 |
PointGoal1, training seed 0 (baseline cost ), episodes per policy, ceiling 1000. is shown for both alphabets. TV and KS (all thresholds) are plug-in; KS is the plug-in bound of Theorem 4 with , ; the last three columns are finite-sample bounds (Hoeff. and EB with the same , ). Other seeds and CarGoal1: Appendix C.
Sufficiency depends on the alphabet.
With the -means alphabet the cost-aware estimate is off by a median relative error of 0.38 over the 24 PointGoal1 cells (it is 9.8–10.8 for agents whose realised cost is 73–115 in seed 0), indicating that Assumption 1 fails; with the cost-relevant alphabet the median relative error is 0.17 (83–117 for the same agents). On CarGoal1 the -means alphabet was already adequate (median relative error 0.06) and the cost-relevant one was slightly worse (0.15). Relative errors are computed against . The sufficiency assumption is therefore a property of the alphabet and the pair of policies. It is not directly testable from the baseline alone. As a direct check on the hypothesis of Theorem 4, which ignores the three clusters inside each risk level, we compute for each cell the drift between the per-level cost rates of and of the baseline on their certificate pools: its median absolute value is 2% of , its largest absolute value 18.0 cost units (PointGoal1 seed 1), and it is positive (the agent costs more than the baseline’s rates predict) in 19 of 32 cells. A weaker check is available on the pilot, between its two policies (noiseless and ): the largest difference in per-level cost rate, over levels with at least 20 visits for both, is 0.31 and 0.27 for PointGoal1 seed 1 and CarGoal1 (at the transition levels), 0.06 for seed 2, and cannot be computed for seed 0 beyond level 4 because the noiseless pilot visits the top levels too rarely; it does not predict the drift of the other agents.
Validity.
Across the 32 cells, none of the finite-sample bounds (TV, monotone Hoeffding and monotone EB, all-threshold and pilot-selected) was violated. This is not a strong test: the four baseline pools are shared by the cells of a run, and the finite-sample slack (127–258 cost units for the EB certificate over its plug-in counterpart) is large compared with the sampling error of and of the occupancies. Ground-truth costs are heavy-tailed (per-episode standard deviations up to 227 for a mean of 82), so itself has a standard error of 1.3–12.7 cost units, and the certificate-pool and ground-truth means of the baselines agree within ; none of the plug-in violations below exceeds two standard errors of the ground truth. Plug-in bounds, which carry no guarantee, were violated: the pilot-selected plug-in in 3 of 32 cells (6 of 32 with ), the TV plug-in with the -means alphabet in 1 of 32 (CarGoal1) and with the cost-relevant alphabet in none. If the onset set is fixed at instead of being selected from the pilot, the plug-in is violated in 14 of the 32 cells (the finite-sample bounds still hold, which we attribute to their slack, not to validity): a misspecified onset set silently voids the guarantee, which is why it must come from baseline data. Because cost sufficiency is only approximate in Safety-Gymnasium (median relative error 0.17 with the cost-relevant alphabet), these non-violations are empirical diagnostics and do not establish formal coverage there. A guarantee applies only when sufficiency is justified, for example by a known cost function of the symbol.
Non-vacuousness and looseness.
With the pilot-selected onset set the empirical-Bernstein bound is below the ceiling of 1000 in all 32 cells (137–406); the TV finite-sample bound on the same alphabet is below it in 30 of 32. The plug-in bound is within a median factor of 1.4 (PointGoal1) and 1.2 (CarGoal1) of , while the empirical-Bernstein certificate is a median factor of 6.3 (PointGoal1) and 4.1 (CarGoal1) above it, against 15.1 and 7.7 for the TV finite-sample bound. The remaining looseness is the sampling slack at and, for agents safer than the baseline, the one-sided nature of the bound. Monotonicity check. The baseline’s own per-level cost rates were nondecreasing in all four runs, and the agents’ empirical per-level rates were nondecreasing in 31 of 32 cells (the exception is one agent of PointGoal1 seed 1, ); this checks the assumption of Theorem 4 only on the data at hand.
4. Limitations
The operational use is one-sided screening against an expected-cost budget : if the finite-sample upper bound is at most , the policy is certified below that budget under the assumptions; otherwise the result is inconclusive. It does not certify that a candidate is safer than the baseline or provide a hard per-episode safety guarantee. In our CartPole screening experiment the test detected all policies above the chosen margins, but at it still rejected 25% of policies that were within a 10-unit margin. In Safety-Gymnasium, bounds below the 1000-unit horizon ceiling can exclude sufficiently large expected costs, but their 4–6-fold looseness over realised cost limits decisions near tighter budgets.
-
Assumption 1 is the substantive assumption and Proposition 5 shows it cannot be removed. It is a property of the alphabet and the pair of policies, not of the task alone: with a generic -means alphabet it failed on PointGoal1 (median relative error of the cost-aware estimate 0.38) and an alphabet built from baseline pilot data restored it approximately (0.17), but not exactly. It can be checked only with cost labels for some episodes of the agent (we report per-cell drift on certificate pools); Proposition 11 explains why cost labels cannot be used inside the certificate itself, and a pilot comparison between two baseline-like policies is only a weak proxy.
-
The monotone-cost certificate adds assumptions: cost is nondecreasing in the risk level, and the onset set contains every threshold at which cost can begin to rise. We select and from baseline pilot data, and the selection rule was itself developed on the same task family (Section 3); a misspecified set voids the guarantee, as the fixed set shows (plug-in bound violated in 14 of 32 cells). The all-threshold version avoids but is looser.
-
Looseness. The certificates are one-sided bounds on the excess over the baseline: they cannot certify that an agent is safer than , and for baselines that are themselves unsafe (PointGoal1 seed 1, ) they are mostly a statement about . At the finite-sample EB certificate was a median 6.3 (PointGoal1) and 4.1 (CarGoal1) times the realised cost on Safety-Gymnasium, and the sampling term of the TV certificate decays only as .
-
Lower bound. Proposition 10 is for and leaves a gap of order to our slack; we do not know whether it can be closed for certificates.
-
Scope. One low-dimensional environment where all assumptions hold exactly, three training seeds on PointGoal1 and one on CarGoal1, all with the same short training recipe (100k steps, PPO-Lagrangian), and agents that are siblings of the baseline. We do not evaluate real deployments or compare with deployment-time safety mechanisms such as shields, because the certificate is not an alternative to them. The importance-sampling comparison is on a single environment with noise policies.
-
Novelty. The main inequalities are elementary consequences of the definitions of total variation and of the Kolmogorov–Smirnov distance; the contribution is their careful statement for pre-deployment certification, the finite-sample treatment and the measurements above.
5. Reproducibility
Code, the unit tests (tests/test_cert.py) and the experiment scripts (experiments/v2) are in the public repository https://github.com/npow/relative-safety-certificate, tag v3, which also contains the summary files behind every table and the rollout data. cartpole.py gen|analyse regenerate Table 1; is_comparison.py regenerates Table 2; train_ppolag.py and safetygym2.py fit|onset|gen|analyse regenerate Table 4 and Appendix C; revision_study.py, is_eb.py and pilot_sensitivity.py reproduce the revision analyses saved in experiments/v2/revision; make_tables.py writes the LaTeX tables and check_numbers.py verifies the reported quantities against the summary and revision output files. Pinned versions are in experiments/v2/requirements.txt (Python 3.10, gymnasium 0.28.1, safety-gymnasium 0.4.1, OmniSafe 0.5.0, numpy 1.26.4, scikit-learn 1.7.2, torch 2.14.1). Seeds are fixed in the scripts; the PPO-Lagrangian training itself is not bit-reproducible across machines, so the trained policies are part of the released data.
6. Proofs
Proof of Theorem 1. By the tower property and Assumption 1, . Summing over and using gives . This is an identity; no independence or Markov structure is used. For any and probability vectors , for , which is at most . Apply with ; . ◻
Proof of Corollary 2. Let . By Pinsker’s inequality, and likewise for , with natural logarithms. Since , , i.e. . ◻
Proof of Proposition 3. The upper bound is the last step of the proof of Theorem 1. For , , because the positive and negative parts of have equal mass. ◻
Proof of Proposition 5. Take two symbols and the same fixed sequence of symbols under both policies, so that . Let the cost be under and under (the cost depends on a hidden state that the discretisation does not reveal); with for a fraction of the steps, and . ◻
Proof of Corollary 6. Apply Theorem 1 twice, or equivalently use the triangle inequality . This requires Assumption 1 for , and jointly. ◻
Proof of Proposition 7. Baseline cost. Episode costs lie in and are independent, so by Hoeffding’s inequality with probability . We then bound using this and Theorem 1.
Occupancy. Let , a function of independent episodes. Changing one episode of changes by at most in total variation, because the per-episode frequency vector is a probability vector; the same holds for with . McDiarmid’s inequality (McDiarmid 1989) with gives with probability . By the triangle inequality, pointwise, so . For each symbol, is the mean of i.i.d. variables with mean , so and, by Cauchy–Schwarz, . Combining and taking a union bound over the two events gives the claim. ◻
Proof of Theorem 4. Write , and . By Theorem 1’s identity, . Abel summation, using , gives . The increments are nonnegative and sum to at most ; those inside contribute at most and those outside , whose total is at most , at most . The inequalities hold because is the difference of the two measures on a single event. For and the step function at a maximising with , the increments are zero except , so the sum equals . ◻
Proof of Proposition 8. The baseline term is as in Proposition 7. For each and each episode the tail fraction , and these are i.i.d. across episodes with mean . Hoeffding’s inequality gives, for each threshold that enters ( if , all thresholds otherwise), with probability at least , and likewise. A union bound over the thresholds and the two policies has total failure probability at most , so simultaneously, and hence and, if , the same for (which is at most one). Combining with the baseline event gives probability at least . ◻
Proof of Proposition 9. Per-episode tail fractions and normalised costs lie in and are i.i.d. across episodes. Theorem 4 of Maurer and Pontil (Maurer and Pontil 2009) states that for i.i.d. , , the mean of is within of its expectation with probability at least (in each direction). We apply it to the upper deviation of and the lower deviation of for each at level and to the upper deviation of the baseline cost at level ; the union bound over events at level costs . Then for all thresholds that enter, simultaneously, and Theorem 4 applies (for the events are taken over all thresholds, so that is controlled as well). ◻
Proof of Proposition 10. Declare “equal” if and “far” otherwise. If , the output is “equal” with probability at least by the second assumption. If , then by validity with probability at least , so the output is “far”. This is a tester that distinguishes from with error probability at most from samples of each distribution. By Chan, Diakonikolas, Valiant and Valiant (Chan et al. 2014), any such tester needs samples, the second term in the range stated there. ◻
Proof of Proposition 11. Let be the event that the cost bit is one. Then , and . Hence . In Proposition 7, , which gives the second claim. ◻
7. Experimental details
CartPole: policies, discretisation and cost ramp as in Section 3; the steps after a failing step are padded with the absorbing symbol. Safety-Gymnasium: PPO-Lagrangian with 2000 steps per epoch, cost limit 25, , 100k steps, latest saved checkpoint, deterministic actor mean (plus Gaussian noise where stated); episodes have fixed length 1000. The symbol of the observation before a step is paired with the cost returned by that step, per-step costs are at most one (asserted on the recorded data) and every episode has exactly steps (asserted). The alphabet pilot consists of 10 jobs of 5 baseline episodes (every 5th step recorded), half noiseless and half with ; the window and the risk-level edges are fixed from it, and the onset set from a re-run of the same pilot. The pilot, certificate and ground-truth episodes use disjoint seeds. Ground-truth uses 300 independent episodes per agent, so its own standard error is not negligible for heavy-tailed costs (the baseline of PointGoal1 seed 1 has a certificate-pool mean of 64 against a ground-truth mean of 82, a difference of standard errors).
8. Safety-Gymnasium: all tasks and seeds
| Task, seed | Agent | (k-means) | (risk) | TV | KS | TV fin. | KS Hoeff. | KS EB | ||
|---|---|---|---|---|---|---|---|---|---|---|
| PointGoal1, seed 0 | lam0.1 | 3.7 | 73.1 | 10.2 | 83.4 | 485 | 115 | 894 | 369 | 276 |
| PointGoal1, seed 0 | lam0.5 | 3.7 | 72.8 | 9.8 | 86.8 | 503 | 118 | 911 | 372 | 296 |
| PointGoal1, seed 0 | lam1.0 | 3.7 | 85.1 | 9.9 | 102.5 | 603 | 140 | 1012 | 395 | 317 |
| PointGoal1, seed 0 | lam2.0 | 3.7 | 115.1 | 10.8 | 116.8 | 502 | 153 | 911 | 408 | 323 |
| PointGoal1, seed 0 | basenoise0.1 | 3.7 | 8.8 | 6.2 | 9.2 | 68 | 12 | 477 | 266 | 155 |
| PointGoal1, seed 0 | basenoise0.3 | 3.7 | 7.1 | 6.9 | 6.7 | 79 | 8 | 488 | 263 | 149 |
| PointGoal1, seed 0 | basenoise0.6 | 3.7 | 6.3 | 9.0 | 8.4 | 189 | 12 | 598 | 267 | 150 |
| PointGoal1, seed 0 | basenoise1.0 | 3.7 | 9.0 | 11.1 | 9.2 | 325 | 13 | 734 | 268 | 153 |
| PointGoal1, seed 1 | lam0.1 | 82.4 | 63.6 | 60.6 | 46.0 | 113 | 65 | 522 | 319 | 295 |
| PointGoal1, seed 1 | lam0.5 | 82.4 | 37.0 | 65.8 | 33.7 | 183 | 64 | 592 | 319 | 255 |
| PointGoal1, seed 1 | lam1.0 | 82.4 | 39.7 | 61.9 | 37.1 | 145 | 65 | 554 | 320 | 281 |
| PointGoal1, seed 1 | lam2.0 | 82.4 | 32.3 | 65.4 | 17.6 | 208 | 64 | 617 | 319 | 210 |
| PointGoal1, seed 1 | basenoise0.1 | 82.4 | 92.9 | 64.3 | 62.2 | 163 | 75 | 572 | 330 | 333 |
| PointGoal1, seed 1 | basenoise0.3 | 82.4 | 88.4 | 65.9 | 72.3 | 267 | 106 | 676 | 360 | 364 |
| PointGoal1, seed 1 | basenoise0.6 | 82.4 | 92.0 | 70.0 | 78.2 | 430 | 104 | 839 | 359 | 360 |
| PointGoal1, seed 1 | basenoise1.0 | 82.4 | 90.7 | 74.4 | 74.1 | 593 | 86 | 1002 | 340 | 337 |
| PointGoal1, seed 2 | lam0.1 | 6.7 | 68.8 | 15.1 | 59.6 | 573 | 97 | 982 | 352 | 238 |
| PointGoal1, seed 2 | lam0.5 | 6.7 | 32.9 | 14.0 | 35.9 | 425 | 62 | 834 | 317 | 207 |
| PointGoal1, seed 2 | lam1.0 | 6.7 | 84.7 | 19.0 | 88.4 | 501 | 118 | 910 | 373 | 271 |
| PointGoal1, seed 2 | lam2.0 | 6.7 | 26.5 | 15.6 | 33.4 | 346 | 49 | 755 | 305 | 191 |
| PointGoal1, seed 2 | basenoise0.1 | 6.7 | 8.0 | 7.9 | 9.5 | 340 | 10 | 749 | 266 | 137 |
| PointGoal1, seed 2 | basenoise0.3 | 6.7 | 9.8 | 8.6 | 10.7 | 334 | 12 | 743 | 267 | 140 |
| PointGoal1, seed 2 | basenoise0.6 | 6.7 | 12.3 | 12.1 | 15.9 | 321 | 22 | 730 | 277 | 154 |
| PointGoal1, seed 2 | basenoise1.0 | 6.7 | 26.7 | 17.7 | 19.4 | 358 | 34 | 767 | 290 | 167 |
| CarGoal1, seed 0 | lam0.1 | 58.7 | 164.5 | 60.0 | 134.3 | 390 | 199 | 799 | 454 | 406 |
| CarGoal1, seed 0 | lam0.5 | 58.7 | 62.1 | 61.2 | 58.4 | 207 | 71 | 616 | 326 | 274 |
| CarGoal1, seed 0 | lam1.0 | 58.7 | 115.5 | 62.0 | 103.6 | 369 | 148 | 778 | 403 | 360 |
| CarGoal1, seed 0 | lam2.0 | 58.7 | 102.8 | 61.3 | 86.8 | 263 | 117 | 672 | 372 | 327 |
| CarGoal1, seed 0 | basenoise0.1 | 58.7 | 64.2 | 60.7 | 47.7 | 83 | 61 | 491 | 315 | 241 |
| CarGoal1, seed 0 | basenoise0.3 | 58.7 | 64.4 | 60.8 | 70.3 | 92 | 79 | 501 | 334 | 285 |
| CarGoal1, seed 0 | basenoise0.6 | 58.7 | 57.8 | 61.0 | 66.0 | 129 | 69 | 538 | 324 | 273 |
| CarGoal1, seed 0 | basenoise1.0 | 58.7 | 57.1 | 61.2 | 69.6 | 157 | 80 | 566 | 334 | 282 |
All 32 (task, seed, agent) cells, episodes per policy. and : ground-truth costs of baseline and agent. for both alphabets. TV: plug-in, cost-relevant alphabet. KS: plug-in with the pilot-selected onset set. TV fin., KS is the plug-in bound with the pilot-selected and (0.05, or 0.055 for seed 2); KS Hoeff. and EB: finite-sample bounds with the same , .
9. References
- Achiam, Joshua, David Held, Aviv Tamar, and Pieter Abbeel. 2017. “Constrained Policy Optimization.” Proceedings of the 34th International Conference on Machine Learning (ICML).
- Alshiekh, Mohammed, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. 2018. “Safe Reinforcement Learning via Shielding.” Proceedings of the AAAI Conference on Artificial Intelligence.
- Chan, Siu-On, Ilias Diakonikolas, Gregory Valiant, and Paul Valiant. 2014. “Optimal Algorithms for Testing Closeness of Discrete Distributions.” Proceedings of the 25th ACM-SIAM Symposium on Discrete Algorithms (SODA).
- Cheng, Richard, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. 2019. “End-to-End Safe Reinforcement Learning Through Barrier Functions for Safety-Critical Continuous Control Tasks.” Proceedings of the AAAI Conference on Artificial Intelligence.
- Chow, Yinlam, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. 2018. “A Lyapunov-Based Approach to Safe Reinforcement Learning.” Advances in Neural Information Processing Systems.
- Cloete, Jacques, Nikolaus Vertovec, and Alessandro Abate. 2025. “SPoRt – Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL.” Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, 4976–84. https://doi.org/10.24963/ijcai.2025/554.
- Ho, Jonathan, and Stefano Ermon. 2016. “Generative Adversarial Imitation Learning.” Advances in Neural Information Processing Systems.
- Ji, Jiaming, Borong Zhang, Jiayi Zhou, et al. 2023. “Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark.” Advances in Neural Information Processing Systems, Datasets and Benchmarks Track.
- Ji, Jiaming, Jiayi Zhou, Borong Zhang, et al. 2024. “OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research.” Journal of Machine Learning Research 25 (285): 1–6.
- Kakade, Sham, and John Langford. 2002. “Approximately Optimal Approximate Reinforcement Learning.” Proceedings of the 19th International Conference on Machine Learning (ICML), 267–74.
- Lacotte, Jonathan, Mohammad Ghavamzadeh, Yinlam Chow, and Marco Pavone. 2019. “Risk-Sensitive Generative Adversarial Imitation Learning.” Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, Proceedings of machine learning research, vol. 89: 2154–63.
- Maurer, Andreas, and Massimiliano Pontil. 2009. “Empirical Bernstein Bounds and Sample Variance Penalization.” Proceedings of the 22nd Conference on Learning Theory (COLT).
- McDiarmid, Colin. 1989. “On the Method of Bounded Differences.” In Surveys in Combinatorics. Cambridge University Press.
- Müller, Alfred. 1997. “Integral Probability Metrics and Their Generating Classes of Functions.” Advances in Applied Probability 29 (2): 429–43. https://doi.org/10.2307/1428011.
- Müller, Alfred, and Dietrich Stoyan. 2002. Comparison Methods for Stochastic Models and Risks. Wiley.
- Ray, Alex, Joshua Achiam, and Dario Amodei. 2019. Benchmarking Safe Exploration in Deep Reinforcement Learning.
- Schulman, John, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. “Trust Region Policy Optimization.” Proceedings of the 32nd International Conference on Machine Learning (ICML).
- Shaked, Moshe, and J. George Shanthikumar. 2007. Stochastic Orders. Springer Series in Statistics. Springer.
- Thomas, Philip S., Georgios Theocharous, and Mohammad Ghavamzadeh. 2015. “High-Confidence Off-Policy Evaluation.” Proceedings of the AAAI Conference on Artificial Intelligence 29: 3000–3006. https://doi.org/10.1609/aaai.v29i1.9541.
- Vincent, Joseph A., Aaron O. Feldman, and Mac Schwager. 2023. Guarantees on Robot System Performance Using Stochastic Simulation Rollouts. https://arxiv.org/abs/2309.10874.
- Vincent, Joseph A., Aaron O. Feldman, and Mac Schwager. 2024. “Guarantees on Robot System Performance Using Stochastic Simulation Rollouts.” IEEE Transactions on Robotics 40: 3984–4002. https://doi.org/10.1109/TRO.2024.3444070.
Disclosure of AI use
Claude (Anthropic; Claude Code with Sonnet and Opus-class models) was used throughout the original work. OpenAI Codex was used for the first-round revision analyses, related manuscript changes, and response letter. These tools assisted with code, experiments, writing, and simulated peer review. The human author directed the research, reviewed the results, and is responsible for the content. All reported numbers are produced by the linked scripts.