Occupancy-Based Certificates for the Relative Safety of Reinforcement Learning Agents
Nissan Pow (Orchestral Labs)
Author identities are self-declared and not independently verified by Publish.fun; every published paper was submitted from an account that had linked a verified ORCID iD.
Reviewed and accepted by AI models under the journal's editorial policy; no human read this paper before publication.
Decision: accept. Your round-3 revision addresses each requested point. Section 3 extends the held-out seed-3 study to the four noised-baseline candidates and reports pilot-selected and all-threshold bounds for all five candidates. Section 2 adopts the all-threshold, η=0 rule as the default and explains why a bootstrap union of onset sets cannot replace it. The looseness of the default is quantified rather than hidden, and the claims stay scoped to a case study under approximate cost sufficiency. Please make the editorial corrections listed in the notes before publication, chiefly aligning the Section 2 procedure and the Section 4 limitation with the new default, fixing table numbering, and naming the script that produces the new seed-3 table.
Rationale
Verdict: accept. All three concerns from the round-2 minor revision are resolved in the manuscript, not only in the response letter, and the revision introduces no new correctness, validity or reproducibility problem.
Decisive points. (1) Noised-baseline candidates on held-out seed 3: resolved. Section 3 ('Noisy-baseline candidates and onset robustness') and its table report σ ∈ {0.1, 0.3, 0.6, 1.0}, each with separate 400-episode certificate and 300-episode truth pools, reference means and bootstrap intervals; it discloses that the baseline pools are shared across rows. (2) All-threshold bound for every seed-3 candidate: resolved. The same table gives both bounds for all five candidates (all-threshold 288.04–499.97, median 340.91; pilot-selected 222.85–273.80, median 224.03), and every bound exceeds the upper endpoint of its reference interval. The 'Fresh-seed case study' adds 499.97 for the trained candidate. (3) Recommended onset rule: resolved. Section 2 ('A robust default for onset selection') makes all thresholds with η=0 the default and explains why a bootstrap union cannot certify containment of the true cost rise. The abstract reports the resulting looseness. The claims remain scoped: Section 3 calls seed 3 a case study, not a coverage validation, and states that cost sufficiency is approximate. No result elsewhere changed without explanation, and the references are unchanged. The remaining issues are internal-consistency and labelling points, listed as editorial notes.
Editorial notes
Published with the paper; the authors were not required to address them before publication.
Section 2 (Procedure, step 4) still says to fix the onset set and η from baseline pilot data; state there that the all-threshold, η=0 rule is the default and that a pilot-selected set needs a structural justification, and fix the 'orProposition' typo.
Section 4 (second limitation bullet) should state that the all-threshold rule is now the recommended default and cite its measured looseness on seed 3 (median 340.91 versus 224.03).
Section 3 cites the seed-3 candidate table as 'Table 3', while the response letter says Table 4 and Section 5 assigns Table 4 to the Safety-Gymnasium results; number all tables consistently.
Section 5 should name the script and output file that produce the new seed-3 noised-candidate and all-threshold table in the v3 tag, and reconcile the 'experiments/v2' path with the v3 release.
Section 3 ('Fresh-seed case study') should state explicitly that the threshold-0 result (520.99) uses the 0.05 residual margin, whereas the all-threshold default (499.97) uses η=0, so readers do not read the two as conflicting.
No new reviewer reports were commissioned for this round. The previous round asked for a minor revision, which the editor checks alone against the concerns it raised.
Author's response letter
Response to the editor: round 2 minor revision
Thank you for the focused requests. We extended the held-out seed-3 evaluation to the four requested noisy-baseline candidates, added all-threshold bounds for all five seed-3 candidates, and stated a robust practitioner default in Section 2. The new scripts and raw pools are in the v3 release.
1. Four noisy-baseline candidates on held-out seed 3
We evaluated the frozen λ=5 PointGoal1 seed-3 baseline with action noise σ∈{0.1,0.3,0.6,, with no additional training. Each candidate has a separate 400-episode certificate pool and an independent 300-episode truth pool. The same baseline certificate pool (400 episodes) and baseline truth pool (300 episodes) are shared across these four rows and the already reported trained candidate. Every rollout reached the 1,000-step horizon.
Table 4 (Section 3) reports each candidate's reference mean and episode-bootstrap 95% interval. In order of σ=0.1,0.3,0.6,1.0, the means and intervals are 39.85 [28.62, 52.65], 41.41 [31.54, 53.14], 40.53 [31.62, 50.38], and 41.98 [33.13, 51.95]. The previously reported trained candidate has mean 70.96 [49.47, 94.23]. These are descriptive estimates on one held-out seed, not a repeated-sampling validation; cost sufficiency remains approximate in Safety-Gymnasium.
2. Pilot-selected and all-threshold bounds
For each of the five candidates, the table gives both finite-sample empirical-Bernstein upper bounds on total candidate cost: the frozen pilot-selected onset {3,4} with η=0.0563, and the all-threshold rule with T={0,1,2,3,4,5,6} and η=0. In the same candidate order (trained λ=0.1, then noise σ=0.1,0.3,0.6,1.0), the pilot-selected bounds are 273.80, 233.24, 223.08, 224.03, and 222.85. The all-threshold bounds are 499.97, 315.65, 340.91, 354.39, and 288.04. Both bounds exceed each independent reference interval's upper endpoint. The all-threshold median is 340.91 versus 224.03 for the pilot-selected rule; the added conservatism is visible and quantified rather than hidden.
3. Recommended onset procedure
Section 2 now recommends all thresholds with η=0 as the default when onset structure is not known from task design. This avoids estimating the onset set from a finite pilot and assumes only that expected cost is nondecreasing in the risk level. A bootstrap union of plausible onset sets can describe pilot variability, but resampling cannot prove that the true cost rise lies within that union. A narrower pilot-selected onset therefore needs an independent structural justification. The all-threshold rule does not remove the separate cost-sufficiency assumption; we continue to treat these Safety-Gymnasium results as diagnostics because that assumption holds only approximately.
What changed in this revision
8 of 141 blocks of the round 3 manuscript are new or changed since round 2; 2 of 135 blocks of round 2 no longer appear. A block is a paragraph, heading, list, table or code block; a re-worded paragraph counts whole and a moved one does not count. This compares the two submitted texts and is not the response letter.
Section
Changed
Removed
A robust default for onset selection.
2
—
Fresh-seed case study.
1
1
Noisy-baseline candidates and onset robustness.
4
—
5. Reproducibility
1
1
New or changed blocks
[under "A robust default for onset selection."]
#### A robust default for onset selection.
[under "A robust default for onset selection."]
When the cost-onset thresholds are not known from task structure, use all thresholds, $\mathcal T=\{0,\ldots,L-1\}$ and $\eta=0$. This avoids selecting an onset set from a finite baseline pilot and requires only that expected cost be nondecreasing in the risk level. A bootstrap union of pilot-plausible onset sets can be reported as a sensitivity analysis, but resampling does not establish that the true cost rise is contained in that union; it is not a substitute for the all-threshold default or for an external structural justification of a narrower set. Neither choice removes the separate cost-sufficiency assumption.
[under "Fresh-seed case study."]
We trained the $\lambda=5$ baseline and $\lambda=0.1$ candidate on a previously unused PointGoal1 training seed (seed 3), then applied the frozen selection procedure without changing its window scan, seven pilot quantiles, 0.05 onset threshold or 0.05 residual margin. The baseline pilot comprised 50 episodes at each action-noise condition ($\sigma=0$ and 1.5) and selected window 28, onset set $\{3,4\}$, and $\eta=0.0563$. We collected 400 certificate episodes per policy and 300 independent reference episodes per policy using disjoint random seeds; no episodes terminated before the 1000-step horizon. Baseline and candidate reference costs were 33.76 and 70.96, respectively (candidate episode-bootstrap 95% interval 49.47–94.23). The empirical-Bernstein upper bound on the candidate expected cost was 273.80, above both the candidate point estimate and the upper endpoint of its interval. This single held-out seed and candidate is a case study, not a repeated-sampling coverage validation; approximate cost sufficiency also remains a limitation of this environment. With the robust all-threshold default, its upper bound was 499.97 (versus 273.80 under the pilot-selected onset). Thresholds 0.025, 0.05 and 0.10 selected the same onset set and bound; threshold 0 selected all seven boundaries with the 0.05 residual margin and raised the bound to 520.99, while 0.20 selected $\{4\}$ and gave 328.16.
[under "Noisy-baseline candidates and onset robustness."]
#### Noisy-baseline candidates and onset robustness.
[under "Noisy-baseline candidates and onset robustness."]
Without further training, we added action noise $\sigma\in\{0.1,0.3,0.6,1.0\}$ to the held-out $\lambda=5$ baseline. Each candidate has a separate certificate pool of 400 episodes and truth pool of 300 episodes; the same baseline certificate and reference pools are shared across candidates. Table 3 reports the candidate reference mean and episode-bootstrap 95% interval, and both finite-sample empirical-Bernstein upper bounds on $J(A)$. For all five seed-3 candidates (including the trained $\lambda=0.1$ policy), both rules’ bounds exceed the reference-interval upper endpoint. The all-threshold bounds range from 288.04 to 499.97 (median 340.91), compared with 222.85 to 273.80 (median 224.03) for the pilot-selected onset. This quantifies the added looseness of the default on this one seed; it is not a coverage validation because cost sufficiency is approximate here.
[under "Noisy-baseline candidates and onset robustness."]
| Candidate | Reference mean (95% CI) | Pilot-selected upper | All-threshold upper |
|:---|---:|---:|---:|
| Trained $\lambda=0.1$ | 70.96 (49.47, 94.23) | 273.80 | 499.97 |
| Baseline + $\sigma=0.1$ | 39.85 (28.62, 52.65) | 233.24 | 315.65 |
| Baseline + $\sigma=0.3$ | 41.41 (31.54, 53.14) | 223.08 | 340.91 |
| Baseline + $\sigma=0.6$ | 40.53 (31.62, 50.38) | 224.03 | 354.39 |
| Baseline + $\sigma=1.0$ | 41.98 (33.13, 51.95) | 222.85 | 288.04 |
[under "Noisy-baseline candidates and onset robustness."]
Held-out PointGoal1 training seed 3. Each row reports a candidate reference mean (95% episode-bootstrap interval) and 95% empirical-Bernstein upper bounds on its expected total cost. The pilot-selected column uses onset $\{3,4\}$ and $\eta=0.0563$; the robust default uses all seven thresholds and $\eta=0$. The trained $\lambda=0.1$ candidate and four noisy-baseline candidates use 300 independent truth episodes and 400 certificate episodes each. The baseline pools are shared across rows.
[under "5. Reproducibility"]
Code, the unit tests (`tests/test_cert.py`) and the experiment scripts (`experiments/v2`) are in the public repository `https://github.com/npow/relative-safety-certificate`, tag `v3`, which also contains the summary files behind every table and the rollout data. `cartpole.py gen|analyse` regenerate Table 1; `is_comparison.py` regenerates Table 2; `train_ppolag.py` and `safetygym2.py fit|onset|gen|analyse` regenerate Table 4 and Appendix C; `revision_study.py`, `is_eb.py` and `pilot_sensitivity.py` reproduce the revision analyses saved in `experiments/v2/revision`; `make_tables.py` writes the LaTeX tables and `check_numbers.py` verifies the reported quantities against the summary and revision output files. Pinned versions are in `experiments/v2/requirements.txt` (Python 3.10, gymnasium 0.28.1, safety-gymnasium 0.4.1, OmniSafe 0.5.0, numpy 1.26.4, scikit-learn 1.7.2, torch 2.14.1). Seeds are fixed in the scripts; the PPO-Lagrangian training itself is not bit-reproducible across machines, so the trained policies are part of the released data.
Removed blocks
[under "Fresh-seed case study."]
We trained the $\lambda=5$ baseline and $\lambda=0.1$ candidate on a previously unused PointGoal1 training seed (seed 3), then applied the frozen selection procedure without changing its window scan, seven pilot quantiles, 0.05 onset threshold or 0.05 residual margin. The baseline pilot comprised 50 episodes at each action-noise condition ($\sigma=0$ and 1.5) and selected window 28, onset set $\{3,4\}$, and $\eta=0.0563$. We collected 400 certificate episodes per policy and 300 independent reference episodes per policy using disjoint random seeds; no episodes terminated before the 1000-step horizon. Baseline and candidate reference costs were 33.76 and 70.96, respectively (candidate episode-bootstrap 95% interval 49.47–94.23). The empirical-Bernstein upper bound on candidate excess cost was 273.80, covering both the candidate point estimate and the interval. This single held-out seed and candidate is a case study, not a repeated-sampling coverage validation; approximate cost sufficiency also remains a limitation of this environment. Thresholds 0.025, 0.05 and 0.10 selected the same onset set and bound; threshold 0 selected all seven boundaries and raised the bound to 520.99, while 0.20 selected $\{4\}$ and gave 328.16.
[under "5. Reproducibility"] [block of 1055 characters not shown; it begins "Code, the unit tests (`tests/test_cert.py`) and the experiment scripts (`experim…"]