Key concerns to address
- Release the per-architecture CIFAR-10/100/ImageNet-16 sweep results and raw multi-seed outputs so that Tables 5, 6, 7 and 9 can be regenerated (Sections 6.3–6.7, 9).
- Restate Theorem 1 (Section 3.1) to include kernel and spatial dimensions, derive the assigned-cell-to-row accounting, and give the fitted ε overhead with a fit/evaluation split disjoint from the 73-architecture validation of Table 10.
- Weaken or properly prove Theorem 3's claim that FLOPs cannot be a constant-factor approximation, and either give a full proof of Proposition 5 under a precisely defined Zen score or label it a heuristic (Sections 3.2–3.3).
- Specify λ, the number of random trials, the tuning/selection separation and confidence intervals in Section 6.7, and repeat the Table 9 comparison against NEAR-only and random under ezkl-calibrated rather than estimated budgets.
- Re-run the Table 8 mock and real proof timings and the Table 3 and Table 10 calibrations on trained networks with real inputs, under one recorded ezkl version and hardware, with repeated timings (Sections 5, 6.6, 6.8).
- Discuss the ICAIIC 2026 ezkl layer-level CNN benchmarking work cited by Reviewers 2 and 3 in Section 7, and temper the 'first benchmark' claims in Sections 1 and 7 accordingly.
Message to authors
Thank you for an unusually transparent submission. The width-matched negative result in Section 5 and the multi-seed validation in Section 6.2 are valuable. We cannot accept the paper yet, for three reasons. First, the sweep data behind the proxy-correlation and search tables is unreleased. Second, the Table 8 timings come from an undocumented run. Third, Theorems 1 and 3 and Proposition 5 claim more than their proofs establish, and the Section 6.7 search lacks specified λ, random-trial counts and uncertainty. In a revision, please release the data, state the theory at the strength actually proven, and redo the budget search with proper statistics and calibrated budgets. Please also re-measure ezkl cost on trained networks under recorded settings and engage with the layer-level ezkl benchmarking prior work identified by the reviewers. With these changes the contribution could meet the bar.
Rationale
Verdict: major revision. The measurement contribution of Section 5 is a useful, carefully controlled negative result, but the paper does not meet the bar as submitted. Three sets of problems remain: the central empirical claims cannot be reproduced, the theory is weaker than stated, and the search result is not clearly established.
Decisive points: All three reviewers flag that the per-architecture sweep behind Tables 5, 6, 7 and 9 is absent from the repository (Section 6.7, Section 9). The proxy-failure and budget-search claims therefore cannot be checked. Reviewers 1 and 3 add that the Table 8 timings come from an unrecorded ezkl version, setup and hardware. Reviewer 2's critique of Section 3 is persuasive and the paper confirms it: Theorem 1 omits kernel and spatial parameters, and its residual ε is fitted empirically. Theorem 3's claim that FLOPs cannot be a constant-factor approximation does not follow from a bounded ratio. Proposition 5 is a proof sketch without a specified Zen score. Reviewer 2 also correctly notes that Section 6.7 leaves λ and the random-trial count unspecified, reports no uncertainty, and ties NEAR-only at two of three budgets. Its selected architecture also exceeds the nominal budget after ezkl calibration. Reviewer 3 cites ICAIIC 2026 layer-level ezkl benchmarking that overlaps Section 5 and is not discussed. The novelty claims in Sections 1 and 7 are unverified, not refuted.
What would change the decision: release the full sweep data and the fitted cost-model specification. Restate the theorems at the strength their proofs support. Repeat the budget search with specified λ, many random trials, confidence intervals and calibrated budgets. Re-run the ezkl timings and calibration on trained networks under recorded settings.