from d2l import torch as d2l
import logging
logging.basicConfig(level=logging.INFO)
import matplotlib.pyplot as plt
from syne_tune.config_space import loguniform, randint
from syne_tune.backend.python_backend.python_backend import PythonBackend
from syne_tune.optimizer.baselines import ASHA
from syne_tune import Tuner, StoppingCriterion
from syne_tune.experiments import load_experiment19.5 Asynchronous Successive Halving
As we have seen in Section 19.3, we can accelerate HPO by distributing the evaluation of hyperparameter configurations across either multiple instances or multiples CPUs / GPUs on a single instance. However, compared to random search, it is not straightforward to run successive halving (SH) asynchronously in a distributed setting. Before we can decide which configuration to run next, we first have to collect all observations at the current rung level. This requires to synchronize workers at each rung level. For example, for the lowest rung level \(r_{\mathrm{min}}\), we first have to evaluate all \(N = \eta^K\) configurations, before we can promote the \(\frac{1}{\eta}\) of them to the next rung level.
In any distributed system, synchronization typically implies idle time for workers. First, we often observe high variations in training time across hyperparameter configurations. For example, assuming the number of filters per layer is a hyperparameter, then networks with less filters finish training faster than networks with more filters, which implies idle worker time due to stragglers. Moreover, the number of slots in a rung level is not always a multiple of the number of workers, in which case some workers may even sit idle for a full batch.
Figure Figure 19.5.1 shows the scheduling of synchronous SH with \(\eta=2\) for four different trials with two workers. We start with evaluating Trial-0 and Trial-1 for one epoch and immediately continue with the next two trials once they are finished. We first have to wait until Trial-2 finishes, which takes substantially more time than the other trials, before we can promote the best two trials, i.e., Trial-0 and Trial-3 to the next rung level. This causes idle time for Worker-1. Then, we continue with Rung 1. Also, here Trial-3 takes longer than Trial-0, which leads to an additional idling time of Worker-0. Once, we reach Rung-2, only the best trial, Trial-0, remains which occupies only one worker. To avoid that Worker-1 idles during that time, most implementations of SH continue already with the next round, and start evaluating new trials (e.g Trial-4) on the first rung.
Asynchronous successive halving (ASHA) (Li et al. 2018) adapts SH to the asynchronous parallel scenario. The main idea of ASHA is to promote configurations to the next rung level as soon as we collected at least \(\eta\) observations on the current rung level. This decision rule may lead to suboptimal promotions: configurations can be promoted to the next rung level, which in hindsight do not compare favourably against most others at the same rung level. On the other hand, we get rid of all synchronization points this way. In practice, such suboptimal initial promotions have only a modest impact on performance, not only because the ranking of hyperparameter configurations is often fairly consistent across rung levels, but also because rungs grow over time and reflect the distribution of metric values at this level better and better. If a worker is free, but no configuration can be promoted, we start a new configuration with \(r = r_{\mathrm{min}}\), i.e the first rung level.
Figure 19.5.2 shows the scheduling of the same configurations for ASHA. Once Trial-1 finishes, we collect the results of two trials (i.e Trial-0 and Trial-1) and immediately promote the better of them (Trial-0) to the next rung level. After Trial-0 finishes on rung 1, there are too few trials there in order to support a further promotion. Hence, we continue with rung 0 and evaluate Trial-3. Once Trial-3 finishes, Trial-2 is still pending. At this point we have 3 trials evaluated on rung 0 and one trial evaluated already on rung 1. Since Trial-3 performs worse than Trial-0 at rung 0, and \(\eta=2\), we cannot promote any new trial yet, and Worker-1 starts Trial-4 from scratch instead. However, once Trial-2 finishes and scores worse than Trial-3, the latter is promoted towards rung 1. Afterwards, we collected 2 evaluations on rung 1, which means we can now promote Trial-0 towards rung 2. At the same time, Worker-1 continues with evaluating new trials (i.e., Trial-5) on rung 0.
INFO:root:AWS dependencies are not imported since dependencies are missing. You can install them with
pip install 'syne-tune[aws]'
or (for everything)
pip install 'syne-tune[extra]'AWS dependencies are not imported since dependencies are missing. You can install them with
pip install 'syne-tune[aws]'
or (for everything)
pip install 'syne-tune[extra]'INFO:root:Ray Tune schedulers and searchers are not imported since dependencies are missing. You can install them with
pip install 'syne-tune[raytune]'
or (for everything)
pip install 'syne-tune[extra]'AWS dependencies are not imported since dependencies are missing. You can install them with
pip install 'syne-tune[aws]'
or (for everything)
pip install 'syne-tune[extra]'19.5.1 Objective Function
We will use Syne Tune with the same objective function as in Section 19.3.
def hpo_objective_lenet_synetune(learning_rate, batch_size, max_epochs):
from d2l import torch as d2l
from syne_tune import Reporter
model = d2l.LeNet(lr=learning_rate, num_classes=10)
trainer = d2l.HPOTrainer(max_epochs=1, num_gpus=1)
data = d2l.FashionMNIST(batch_size=batch_size)
model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)
report = Reporter()
for epoch in range(1, max_epochs + 1):
if epoch == 1:
# Initialize the state of Trainer
trainer.fit(model=model, data=data)
else:
trainer.fit_epoch()
validation_error = d2l.numpy(trainer.validation_error().cpu())
report(epoch=epoch, validation_error=float(validation_error))We will also use the same configuration space as before:
min_number_of_epochs = 2
max_number_of_epochs = 10
eta = 2
config_space = {
"learning_rate": loguniform(1e-2, 1),
"batch_size": randint(32, 256),
"max_epochs": max_number_of_epochs,
}
initial_config = {
"learning_rate": 0.1,
"batch_size": 128,
}19.5.2 Asynchronous Scheduler
First, we define the number of workers that evaluate trials concurrently. We also need to specify how long we want to run random search, by defining an upper limit on the total wall-clock time.
n_workers = 2 # Needs to be <= the number of available GPUs
max_wallclock_time = 5 * 60 # 5 minutesThe code for running ASHA is a simple variation of what we did for asynchronous random search.
mode = "min"
metric = "validation_error"
resource_attr = "epoch"
scheduler = ASHA(
config_space,
metric=metric,
mode=mode,
points_to_evaluate=[initial_config],
max_resource_attr="max_epochs",
resource_attr=resource_attr,
grace_period=min_number_of_epochs,
reduction_factor=eta,
)INFO:syne_tune.optimizer.schedulers.fifo:max_resource_level = 10, as inferred from config_spaceINFO:syne_tune.optimizer.schedulers.fifo:Master random_seed = 3326434658Here, metric and resource_attr specify the key names used with the report callback, and max_resource_attr denotes which input to the objective function corresponds to \(r_{\mathrm{max}}\). Moreover, grace_period provides \(r_{\mathrm{min}}\), and reduction_factor is \(\eta\). We can run Syne Tune as before (this will take about 12 minutes):
trial_backend = PythonBackend(
tune_function=hpo_objective_lenet_synetune,
config_space=config_space,
)
stop_criterion = StoppingCriterion(max_wallclock_time=max_wallclock_time)
tuner = Tuner(
trial_backend=trial_backend,
scheduler=scheduler,
stop_criterion=stop_criterion,
n_workers=n_workers,
print_update_interval=int(max_wallclock_time * 0.6),
)
tuner.run()INFO:syne_tune.tuner:results of trials will be saved on /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119INFO:syne_tune.backend.local_backend:Detected 4 GPUsINFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.1 --batch_size 128 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/0/checkpointsINFO:syne_tune.tuner:(trial 0) - scheduled config {'learning_rate': 0.1, 'batch_size': 128, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.02121396964605979 --batch_size 188 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/1/checkpointsINFO:syne_tune.tuner:(trial 1) - scheduled config {'learning_rate': 0.02121396964605979, 'batch_size': 188, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.3840482203452835 --batch_size 226 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/2/checkpointsINFO:syne_tune.tuner:(trial 2) - scheduled config {'learning_rate': 0.3840482203452835, 'batch_size': 226, 'max_epochs': 10}INFO:syne_tune.tuner:Trial trial_id 1 completed.INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.013331223152843625 --batch_size 212 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/3/checkpointsINFO:syne_tune.tuner:(trial 3) - scheduled config {'learning_rate': 0.013331223152843625, 'batch_size': 212, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.7643292504895935 --batch_size 50 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/4/checkpointsINFO:syne_tune.tuner:(trial 4) - scheduled config {'learning_rate': 0.7643292504895935, 'batch_size': 50, 'max_epochs': 10}INFO:syne_tune.tuner:Trial trial_id 2 completed.INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.0959169852617265 --batch_size 248 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/5/checkpointsINFO:syne_tune.tuner:(trial 5) - scheduled config {'learning_rate': 0.0959169852617265, 'batch_size': 248, 'max_epochs': 10}INFO:syne_tune.tuner:Trial trial_id 5 completed.INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.6610367929575326 --batch_size 171 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/6/checkpointsINFO:syne_tune.tuner:(trial 6) - scheduled config {'learning_rate': 0.6610367929575326, 'batch_size': 171, 'max_epochs': 10}INFO:syne_tune.tuner:Trial trial_id 4 completed.INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.04015634885327024 --batch_size 84 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/7/checkpointsINFO:syne_tune.tuner:(trial 7) - scheduled config {'learning_rate': 0.04015634885327024, 'batch_size': 84, 'max_epochs': 10}INFO:syne_tune.tuner:Trial trial_id 6 completed.INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.015141016995605775 --batch_size 99 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/8/checkpointsINFO:syne_tune.tuner:(trial 8) - scheduled config {'learning_rate': 0.015141016995605775, 'batch_size': 99, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.4985039581048586 --batch_size 182 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/9/checkpointsINFO:syne_tune.tuner:(trial 9) - scheduled config {'learning_rate': 0.4985039581048586, 'batch_size': 182, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.023269817689832193 --batch_size 112 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/10/checkpointsINFO:syne_tune.tuner:(trial 10) - scheduled config {'learning_rate': 0.023269817689832193, 'batch_size': 112, 'max_epochs': 10}INFO:syne_tune.tuner:tuning status (last metric is reported)
trial_id status iter learning_rate batch_size max_epochs epoch validation_error worker-time
0 Stopped 2 0.100000 128 10 2 0.900415 11.096493
1 Completed 10 0.021214 188 10 10 0.899418 37.178450
2 Completed 10 0.384048 226 10 10 0.278027 38.361546
3 Stopped 2 0.013331 212 10 2 0.901249 7.729127
4 Completed 10 0.764329 50 10 10 0.134100 80.459636
5 Completed 10 0.095917 248 10 10 0.457248 36.824051
6 Completed 10 0.661037 171 10 10 0.170807 37.161324
7 Stopped 2 0.040156 84 10 2 0.900794 9.689723
8 Stopped 2 0.015141 99 10 2 0.900971 9.364871
9 InProgress 2 0.498504 182 10 2 0.484833 7.792386
10 InProgress 1 0.023270 112 10 1 0.900546 4.615001
2 trials running, 9 finished (5 until the end), 185.40s wallclock-timeINFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.44097918087443216 --batch_size 114 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/11/checkpointsINFO:syne_tune.tuner:(trial 11) - scheduled config {'learning_rate': 0.44097918087443216, 'batch_size': 114, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.020495262577653203 --batch_size 142 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/12/checkpointsINFO:syne_tune.tuner:(trial 12) - scheduled config {'learning_rate': 0.020495262577653203, 'batch_size': 142, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.6712231893437652 --batch_size 227 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/13/checkpointsINFO:syne_tune.tuner:(trial 13) - scheduled config {'learning_rate': 0.6712231893437652, 'batch_size': 227, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.5652676277056541 --batch_size 150 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/14/checkpointsINFO:syne_tune.tuner:(trial 14) - scheduled config {'learning_rate': 0.5652676277056541, 'batch_size': 150, 'max_epochs': 10}INFO:syne_tune.tuner:Trial trial_id 13 completed.INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.03689802357376642 --batch_size 141 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/15/checkpointsINFO:syne_tune.tuner:(trial 15) - scheduled config {'learning_rate': 0.03689802357376642, 'batch_size': 141, 'max_epochs': 10}INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.27410621860492324 --batch_size 78 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/16/checkpointsINFO:syne_tune.tuner:(trial 16) - scheduled config {'learning_rate': 0.27410621860492324, 'batch_size': 78, 'max_epochs': 10}INFO:syne_tune.tuner:Trial trial_id 14 completed.INFO:syne_tune.backend.local_backend:running subprocess with command: /home/smola/d2l/d2l-neu/.venv-pytorch/bin/python3 /home/smola/d2l/d2l-neu/.venv-pytorch/lib/python3.11/site-packages/syne_tune/backend/python_backend/python_entrypoint.py --learning_rate 0.018186329549246198 --batch_size 212 --max_epochs 10 --tune_function_root /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/tune_function --tune_function_hash 6f999437a0f93ab6e0e8cb1b1558475a --st_checkpoint_dir /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119/17/checkpointsINFO:syne_tune.tuner:(trial 17) - scheduled config {'learning_rate': 0.018186329549246198, 'batch_size': 212, 'max_epochs': 10}INFO:syne_tune.stopping_criterion:reaching max wallclock time (300), stopping there.INFO:syne_tune.tuner:Stopping trials that may still be running.INFO:syne_tune.tuner:Tuning finished, results of trials can be found on /home/smola/syne-tune/python-entrypoint-2026-04-24-01-52-57-119--------------------
Resource summary (last result is reported):
trial_id status iter learning_rate batch_size max_epochs epoch validation_error worker-time
0 Stopped 2 0.100000 128 10 2.0 0.900415 11.096493
1 Completed 10 0.021214 188 10 10.0 0.899418 37.178450
2 Completed 10 0.384048 226 10 10.0 0.278027 38.361546
3 Stopped 2 0.013331 212 10 2.0 0.901249 7.729127
4 Completed 10 0.764329 50 10 10.0 0.134100 80.459636
5 Completed 10 0.095917 248 10 10.0 0.457248 36.824051
6 Completed 10 0.661037 171 10 10.0 0.170807 37.161324
7 Stopped 2 0.040156 84 10 2.0 0.900794 9.689723
8 Stopped 2 0.015141 99 10 2.0 0.900971 9.364871
9 Stopped 10 0.498504 182 10 10.0 0.223069 39.528309
10 Stopped 4 0.023270 112 10 4.0 0.899306 17.826207
11 Stopped 10 0.440979 114 10 10.0 0.178254 40.007729
12 Stopped 2 0.020495 142 10 2.0 0.900271 7.838986
13 Completed 10 0.671223 227 10 10.0 0.204886 32.149690
14 Completed 10 0.565268 150 10 10.0 0.165174 41.916877
15 Stopped 2 0.036898 141 10 2.0 0.899983 7.348897
16 InProgress 2 0.274106 78 10 2.0 0.294263 9.159021
17 InProgress 0 0.018186 212 10 - - -
2 trials running, 16 finished (7 until the end), 300.62s wallclock-time
validation_error: best 0.13410019874572754 for trial-id 4
--------------------Note that we are running a variant of ASHA where underperforming trials are stopped early. This is different to our implementation in Section 19.4.1, where each training job is started with a fixed max_epochs. In the latter case, a well-performing trial which reaches the full 10 epochs, first needs to train 1, then 2, then 4, then 8 epochs, each time starting from scratch. This type of pause-and-resume scheduling can be implemented efficiently by checkpointing the training state after each epoch, but we avoid this extra complexity here. After the experiment has finished, we can retrieve and plot results.
d2l.set_figsize()
e = load_experiment(tuner.name)
e.plot()
19.5.3 Visualize the Optimization Process
Once more, we visualize the learning curves of every trial (each color in the plot represents a trial). Compare this to asynchronous random search in Section 19.3. As we have seen for successive halving in Section 19.4, most of the trials are stopped at 1 or 2 epochs (\(r_{\mathrm{min}}\) or \(\eta * r_{\mathrm{min}}\)). However, trials do not stop at the same point, because they require different amount of time per epoch. If we ran standard successive halving instead of ASHA, we would need to synchronize our workers, before we can promote configurations to the next rung level.
d2l.set_figsize([6, 2.5])
results = e.results
for trial_id in results.trial_id.unique():
df = results[results["trial_id"] == trial_id]
d2l.plt.plot(
df["st_tuner_time"],
df["validation_error"],
marker="o"
)
d2l.plt.xlabel("wall-clock time")
d2l.plt.ylabel("objective function")Text(0, 0.5, 'objective function')
19.5.4 Summary
Compared to random search, successive halving is not quite as trivial to run in an asynchronous distributed setting. To avoid synchronisation points, we promote configurations as quickly as possible to the next rung level, even if this means promoting some wrong ones. In practice, this usually does not hurt much, and the gains of asynchronous versus synchronous scheduling are usually much higher than the loss of the suboptimal decision making.