text: This paper introduces a simple extension to parallelize Hyperband.  Points in favor of the paper:* Addresses an important problem Points against:* Only 5-fold speedup by parallelization with 5 x 25 workers, and worse performance in the same budget than Google Vizier (even though that treats the problem as a black box) * Limited methodological contribution/novelty The paper's methodological contribution is quite limited:  it amounts to a straight-forward parallelization of successive halving (SHA).  Specifically, whenever a worker frees up, do a new run on it, at the highest rung possible while making sure to not run too many runs for too high rungs.  (I am pretty sure that is the idea, even though Algorithm 1, which is supposed to give the details, appears to have a bug in Procedure get_job  -- it would always either pick the highest rung or the lowest!) Empirically, the paper strangely does not actually evaluate a parallel version of Hyperband, but only evaluates the 5 parallel variants of SHA that Hyperband would run, each of them with all workers.  The experiments in Section 4.2 show that, using 25 workers, the best of these 5 variants obtains a 5-fold speedup over sequential Hyperband on CIFAR and an 8-fold speedup on SVHN.  I am confused: the *best* of 5 SHA variants only achieves a 5-fold speedup using 25 workers?  I.e., parallel Hyperband, which would run the 5 SHA variants in parallel, would require 125 workers but only yield a 5-fold speedup?  If I understand this correctly, I would clearly call this a negative result. Likewise, for the large-scale experiment, a single run of Vizier actually yields as good performance as the best of the 5 SHA variants,  and it is unknown beforehand which SHA variant works best -- in this example, actually Bracket 0 (which is often the best) stagnates.  Parallel Hyperband would run the 5 SHA variants in parallel,  so its performance at a budget of 10R with a total of 500 workers can be evaluated by taking the minimum of the 5 SHA variants at a budget of 2R.  This would obtain a perplexity of above 90,  which is quite a bit worse than Vizier's result of about 82.  In general, the performance of parallel Hyperband can be computed by taking the minimum of the SHA variants and multiplying the time taken by 5;  this shows that at any time in the plot (Figure 3, left) Vizier dominates parallel Hyperband.  Again, this is apparently a negative result.  (For Figure 3, right, no results for Vizier are given yet.) If I understand correctly, the experiment in Section 4.4 does not involve any run of Hyperband, but merely plots predictions of Qi et al.'s Paelo framework of how many models could be evaluated with a growing number of GPUs. Therefore, all empirical results for parallel Hyperband reported in the paper appear to be negative.  This confuses me,  especially since the authors seem to take them as positive results.  Because the original Hyperband paper argued that Bayesian optimization does not parallelize as well as random search / Hyperband,  and because Hyperband has been reported to work much better than Bayesian optimization on a single node,  I would have expected clear improvements of parallel Hyperband over parallel Bayesian optimization (=Vizier in the authors' setup).  However, this is not what I see in the results.  Am I mistaken somewhere?  If not, based on these negative results the paper does not seem to quite clear the bar for ICLR. Details, in order of appearance in the paper:- Vizier: why did the authors only use Vizier's default Bayesian optimization algorithm?  The Vizier paper by Golovin et al (2017) states that for large budgets other optimizers often perform better, and the budget in the large scale experiments is as high as 5000 function evaluations.  Also, isn't there an automatic choice built into Vizier to pick the optimizer expected to be best?  I think using a suboptimal version of Vizier would be a problem for the experimental setup. - Algorithm 1: this needs some improvement; in particular fixing the bug I mentioned above. - Section 3.1: Li et al (2017) do not analyze any algorithm theoretically.  They also do not discuss finite vs. infinite horizon.  I believe the authors meant Li et al's arXiv paper (2016) in both of these cases. - Section 3.1, point 2: this is unclear to me, even though I know Hyperband very well.  Can you please make this clearer? - "A complete theoretical treatment of asynchronous SHA is out of the scope of this paper"  -> is some theoretical treatment in scope? - Section 4.1: It seems very useful to already recommend configurations in each rung of Hyperband,  and I am surprised that the methods section does not mention this.  From the text in this experiments section, it feels a little like that was always part of Hyperband;  I didn't think it was,  so I checked the original papers and blog posts,  and both the ICLR 2017 and the arXiv 2016 paper state "In fact, the first result returned by HYPERBAND after using a budget of 5R is often competitive with results returned by other searchers after using 50R."  and Kevin Jamieson's blog post on Hyperband (https://people.eecs.berkeley.edu/~kjamieson/hyperband.html) explicitly states: "While random and the Bayesian Optimization algorithms output their first recommendation after max_iter iterations, Hyperband does not output anything until about max_iter(logeta(max_iter)+1) iterations [...]" Therefore, recommending after each rung seems to be a contribution of this paper,  and I think it would be nice to read about this in the methods section.  - Experiment 1 (SVM) used dataset size as a budget, which is what Fabolas ("Fast Bayesian optimization on large datasets") is designed for according to Klein et al (2017).  On the other hand, Experiments (2) and (3) used the number of epochs as a budget, and Fabolas is not designed for that  (one would want to use a different kernel, for epochs, e.g., like Freeze-Thaw Bayesian optimization (FTBO) by Swersky et al (2014), instead of a kernel made for dataset sizes).  Therefore, it is not surprising that Fabolas does not work as well in those cases.  The case of number of epochs as a budget would be the domain of FTBO.  I know that there is no reference implementation of FTBO,  so I am not asking for a comparison, but the comparison against Fabolas is misleading for Experiments (2) and (3).  This doesn't really change anything for the paper:  the authors could still make the case that Fabolas hasn't been designed for this case and that (to the best of my knowledge) there simply isn't an implementation of a BO algorithm that is.  Fabolas is arguably the closest thing,  so the results could still be reported, just not as an apples-to-apples comparison; probably best as "Fabolas-like, with dataset size kernel" in the figure.  The justification to not compare against Fabolas in the parallel regime is clearly valid. - A clarification question: Section 4.4 does not report on any runs of actual neural networks, does it?  And not on any runs of Hyperband, correct?  Do I understand the reasoning correctly as pointing out that standard parallelization across multiple GPUs is not great, and that thus, in combination with parallel Hyperband, runs should be done mostly on one GPU only?  How does this relate to the results in the cited paper "Accurate, Large-batch SGD: Training ImageNet in 1 Hour" (https://arxiv.org/abs/1706.02677)?  Quoting from its abstract: "Using commodity hardware, our implementation achieves ∼ 90% scaling efficiency when moving from 8 to 256 GPUs."  That seems like a very good utilization of parallel computing power? - There is no conclusion / future work.