Skip to content

Training recipe

All 6 methods--set optimizer=--set schedule=--set hooks=--set augment=--set recipe=37 options

The training recipe is everything around the model: how weights are updated, how the learning rate moves, what runs around the training loop and how a training sample is perturbed. None of it changes what the detector computes.

Use it

grasppanda compose graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e \
    --set optimizer=adamw -o graspnet-baseline_adamw.yaml
grasppanda compose graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e --list optimizer

grasppanda compose gsnet/gsnet_sunet14_g1b-realsense_10e \
    --set optimizer=adamw -o gsnet_adamw.yaml
grasppanda compose gsnet/gsnet_sunet14_g1b-realsense_10e --list optimizer

grasppanda compose scale_balanced_grasp/sbg_pointnet2-transformer_g1b-realsense_18e \
    --set optimizer=adamw -o scale-balanced-grasp_adamw.yaml
grasppanda compose scale_balanced_grasp/sbg_pointnet2-transformer_g1b-realsense_18e --list optimizer

grasppanda compose economicgrasp/economicgrasp_sunet14_g1b-realsense_10e \
    --set optimizer=adamw -o economicgrasp_adamw.yaml
grasppanda compose economicgrasp/economicgrasp_sunet14_g1b-realsense_10e --list optimizer

grasppanda compose finegrasp/finegrasp_sunet14_g1b-realsense_10e \
    --set optimizer=adamw -o finegrasp_adamw.yaml
grasppanda compose finegrasp/finegrasp_sunet14_g1b-realsense_10e --list optimizer

grasppanda compose hggd/hggd_r34w8_g1b-realsense_15e \
    --set optimizer=adamw -o hggd_adamw.yaml
grasppanda compose hggd/hggd_r34w8_g1b-realsense_15e --list optimizer

The first command writes a checked config; the second lists every option of the slot under that method, with what was measured. Then grasppanda probe <file> tries the config on real frames and grasppanda train <file> trains it (Swap a component). Each method page lists the same: GraspNet-baseline · GSNet · Scale-Balanced-Grasp · EconomicGrasp · FineGrasp · HGGD.

optimizer, schedule and recipe take one preset; hooks and augment take several joined by +, as in --set hooks=ema+grad-clip. --opts changes single values and wins over a preset: --opts optimizer.lr=3e-3 train.epochs=20.

Options

A probe shows that a combination trains on real frames of GraspNet-1Billion, not how good it is. Accuracy is claimed only for a method trained and evaluated here.

Option What it is Runs Trains like shipped
adamw AdamW of PyTorch with decoupled weight decay, the default of the transformer encoders. 6/6 3/6 on a par · 3 worse
worse under Scale-Balanced-Grasp, FineGrasp, HGGD
adamw-no-decay-1d AdamW with no weight decay on biases and normalisation weights, through ParamGroups. 6/6 4/6 on a par · 2 worse
worse under FineGrasp, HGGD
ademamix AdEMAMix (Pagliardini et al., arXiv 2409.03137): Adam with a second, slow momentum. 6/6 4/6 on a par · 2 worse
worse under FineGrasp, HGGD
encoder-lr-0.1 Adam at 1e-3 with the scene encoder at a tenth of the rate (it replaces the method's own optimizer, e.g. FineGrasp's Adam at 8e-3). 5/5
seed-based only		 1/5 on a par · 1 diverging · 3 worse
layer-decay AdamW with layer-wise learning-rate decay through the scene encoder (BEiT, as in timm). 5/5
seed-based only		 2/5 on a par · 1 diverging · 2 worse
worse under GSNet, EconomicGrasp; diverging under FineGrasp
prodigy Prodigy (Mishchenko and Defazio, arXiv 2306.06101): Adam that estimates its own step size. 6/6 4/6 on a par · 2 worse
worse under Scale-Balanced-Grasp, FineGrasp
schedulefree-adamw Schedule-free AdamW (Defazio et al., arXiv 2405.15682): no schedule, averaged iterates. 6/6 not compared
schedulefree-radam Schedule-free RAdam (Defazio et al., arXiv 2405.15682): no schedule and no warm-up. 6/6 not compared
schedulefree-sgd Schedule-free SGD (Defazio et al., arXiv 2405.15682): no schedule, averaged iterates. 3/6 not compared
soap SOAP (Vyas et al., arXiv 2409.11321): Adam in the eigenbasis of Shampoo's preconditioner. 5/6 4/5 on a par · 1 worse
worse under FineGrasp
stableadamw StableAdamW (Wortsman et al., NeurIPS 2023): AdamW with per-parameter update clipping. 6/6 3/6 on a par · 3 worse
worse under GraspNet-baseline, FineGrasp, HGGD
timm-adan Adan (Xie et al., arXiv 2208.06677): Adam with a Nesterov momentum estimate. 6/6 2/6 on a par · 1 diverging · 3 worse
timm-adopt ADOPT (Taniguchi et al., NeurIPS 2024): Adam that converges for any beta2. 6/6 3/6 on a par · 3 worse
worse under EconomicGrasp, FineGrasp, HGGD
timm-lamb LAMB (You et al., ICLR 2020): Adam with a per-layer trust ratio, for large batches. 6/6 5/6 on a par · 1 diverging
diverging under FineGrasp
timm-lion Lion (Chen et al., arXiv 2302.06675): sign momentum, found by program search. 6/6 0/6 on a par · 1 diverging · 5 worse
timm-madgrad MADGRAD (Defazio and Jelassi, arXiv 2101.11075): dual averaging with momentum. 6/6 3/6 on a par · 1 diverging · 2 worse
timm-mars MARS (Yuan et al., arXiv 2411.10438): variance-reduced AdamW. 6/6 4/6 on a par · 2 worse
worse under FineGrasp, HGGD
Option What it is Runs
constant No schedule: the optimizer's learning rate for the whole run. 5/6
not FineGrasp (unstable twice in the short probe)
cosine Cosine annealing of PyTorch over the whole run, stepped per epoch, without warm-up. 5/6
not FineGrasp (unstable twice in the short probe)
cosine-restarts Cosine with warm restarts (SGDR, Loshchilov and Hutter, ICLR 2017): cycles of 4 epochs, each peak half the one before. 5/6
not FineGrasp (unstable twice in the short probe)
multistep-warmup One epoch of linear warm-up, then the rate times 0.1 after epochs 6 and 9 (timm's MultiStepLRScheduler). 5/6
not FineGrasp (unstable twice in the short probe)
onecycle One-cycle schedule of PyTorch: a short warm-up to the peak rate, then a cosine to near zero. 6/6
poly Polynomial decay to zero with power 0.9 after 5 % of warm-up, the schedule of segmentation models (timm's PolyLRScheduler). 5/6
not FineGrasp (unstable twice in the short probe)
warmup-cosine Linear warm-up over the first 5 % of the run, then one cosine to 1e-6 (timm's CosineLRScheduler). 5/6
not FineGrasp (unstable twice in the short probe)
Option What it is Runs Trains like shipped
accumulate-2 Gradient accumulation: one optimizer step per two batches, an effective batch twice as large. 5/5
own in HGGD		 2/5 on a par · 3 worse
worse under GraspNet-baseline, GSNet, EconomicGrasp
agc Adaptive gradient clipping (Brock et al., arXiv 2102.06171) in place of a fixed clipping norm. 6/6 6/6 on a par
amp Mixed precision (NOT USABLE YET: the config check refuses it for every shipped detector). 0/6
not supported yet		 not compared
ema Exponential moving average of the weights with warm-up of the decay; evaluation scores it. 6/6 6/6 on a par
freeze-encoder Freeze the scene encoder for the first two epochs, then train it with the rest. 4/5
seed-based only
not FineGrasp (unstable twice in the short probe)		 4/4 on a par
grad-clip Clip the global L2 norm of the gradient at 1.0 before every optimizer step. 6/6 5/6 on a par · 1 worse
worse under FineGrasp
swa Stochastic weight averaging: the mean of the weights after each of the last epochs. 4/6 4/4 on a par
Option What it is Runs Trains like shipped
dropout Remove 20 % of the points of every second sample, before the cloud is sampled. 5/6 5/5 on a par
jitter Clipped Gaussian noise of 1 mm on the input points, after the other point transforms. 5/6
not HGGD (not supported yet)		 5/5 on a par
shift Translate the scene by up to 10 cm sideways and -5 to 10 cm in depth; object poses follow. 5/6
not HGGD (not supported yet)		 3/5 on a par · 2 worse
worse under EconomicGrasp, FineGrasp
Option What it is Runs
adamw-onecycle AdamW with a one-cycle schedule, the recipe Pointcept trains PTv3 with, at a rate for grasp batches. 6/6
finetune-encoder Fine-tuning recipe for a pretrained scene encoder: layer-wise decay, warm-up and cosine, one frozen epoch, EMA. 4/5
seed-based only
not FineGrasp (unstable twice in the short probe)
ptv3-adamw-onecycle Pointcept's PTv3 recipe with its tenth of the rate for the transformer blocks. 5/5
seed-based only
What the columns say, and who is left out
  • Runs: out of the methods the option is a swap under, those under which its probe trained on real frames without error. A method that ships the option is named own and not counted; a method still undecided is counted and named. Measured on 20 training iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, one run per cell; nothing was measured on GraspClutter6D or on Kinect frames.
  • Trains like shipped: out of the methods under which the swap ran, those under which every comparable loss term ended within the method's tolerance of the component it ships, with nothing rising; a method left out is named with why. Measured on 250 iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, at a constant learning rate (FineGrasp: its own warm-up), beside the method as shipped trained the same way twice.
  • Schedules have no level 2: level 2 holds the schedule of the method as shipped fixed (a constant rate; FineGrasp: its own warm-up), so a schedule swap is not compared.
  • How we measure defines every word of these columns.

Who is left out, and why:

  • encoder-lr-0.1, runs: seed-based only: it is written for the scene encoder of the seed-based skeleton (model.backbone), which HGGD does not have; compose refuses it.
  • encoder-lr-0.1, trains like shipped: worse under GraspNet-baseline, GSNet, EconomicGrasp; diverging under FineGrasp.
  • layer-decay, runs: seed-based only: it is written for the scene encoder of the seed-based skeleton (model.backbone), which HGGD does not have; compose refuses it.
  • schedulefree-adamw, trains like shipped: not compared under GraspNet-baseline, GSNet, Scale-Balanced-Grasp, EconomicGrasp, FineGrasp, HGGD: it trained on a schedule of its own, not on that of the method as shipped.
  • schedulefree-radam, trains like shipped: not compared under GraspNet-baseline, GSNet, Scale-Balanced-Grasp, EconomicGrasp, FineGrasp, HGGD: it trained on a schedule of its own, not on that of the method as shipped.
  • schedulefree-sgd, runs: not GSNet, EconomicGrasp, FineGrasp (unstable twice in the short probe).
  • schedulefree-sgd, trains like shipped: not compared under GraspNet-baseline, Scale-Balanced-Grasp, HGGD: it trained on a schedule of its own, not on that of the method as shipped; not compared under GSNet, EconomicGrasp, FineGrasp: the swap did not run at level 1.
  • soap, runs: not HGGD (not verified: one CUDA illegal memory access on a shared GPU).
  • timm-adan, trains like shipped: worse under GSNet, EconomicGrasp, HGGD; diverging under FineGrasp.
  • timm-lion, trains like shipped: worse under GraspNet-baseline, GSNet, EconomicGrasp, FineGrasp, HGGD; diverging under Scale-Balanced-Grasp.
  • timm-madgrad, trains like shipped: diverging under GraspNet-baseline; worse under Scale-Balanced-Grasp, FineGrasp.
  • freeze-encoder, runs: seed-based only: it is written for the scene encoder of the seed-based skeleton (model.backbone), which HGGD does not have; compose refuses it.
  • freeze-encoder, trains like shipped: not compared under FineGrasp: the swap did not run at level 1.
  • swa, runs: not GraspNet-baseline, Scale-Balanced-Grasp (not verified: the probe is shorter than the epoch at which averaging starts).
  • dropout, runs: not FineGrasp (fails: the transform order of the composed config is wrong (DepthNormals must run first)).
  • finetune-encoder, runs: seed-based only: it is written for the scene encoder of the seed-based skeleton (model.backbone), which HGGD does not have; compose refuses it.
  • ptv3-adamw-onecycle, runs: seed-based only: it needs --set backbone=ptv3, a slot HGGD does not have.
Sources and extras

The work each option follows; extra: the optional install it needs (every extra).

Source Options
torch adamw, cosine, onecycle
The weight-decay rule of PyTorch Image Models adamw-no-decay-1d
Kim, pytorch_optimizer (kozistr/pytorch_optimizer), via the library · extra optim ademamix, prodigy, soap, stableadamw
Pointcept's factor for the transformer blocks of PTv3 encoder-lr-0.1
Bao et al., BEiT (arXiv 2106.08254): layer-wise decay as in timm.optim layer-decay
Defazio et al., The Road Less Scheduled (arXiv 2405.15682), via schedulefree · extra optim schedulefree-adamw, schedulefree-radam, schedulefree-sgd
Wightman, PyTorch Image Models (timm.optim), via timm · extra optim timm-adan, timm-adopt, timm-lamb, timm-lion, timm-madgrad, timm-mars
Wightman, PyTorch Image Models (timm.scheduler), via timm · extra optim cosine-restarts, multistep-warmup, poly, warmup-cosine
Brock et al. (arXiv 2102.06171), via timm.utils.adaptive_clip_grad · extra optim agc
Tarvainen and Valpola, NeurIPS 2017 (arXiv 1703.01780), via torch.optim.swa_utils ema
Howard and Ruder, ACL 2018 (arXiv 1801.06146): gradual unfreezing freeze-encoder
Izmailov et al., UAI 2018 (arXiv 1803.05407), via torch.optim.swa_utils swa
Qi et al., PointNet++ (arXiv 1706.02413); adapted from Pointcept/Pointcept@1342eda, MIT dropout, jitter
Pointcept (Wu et al.), RandomShift; adapted from Pointcept/Pointcept@1342eda, MIT shift
Pointcept's PTv3 recipe adamw-onecycle, ptv3-adamw-onecycle
Bao et al., BEiT (arXiv 2106.08254): layer-wise decay as in timm.optim · extra optim finetune-encoder

Compare under every method: optimizers · schedules · hooks · augmentations · whole recipes

Choosing

  • The method's own recipe stays where no preset replaces a part; the composed file lists what each preset says must be re-tuned.
  • A pretrained scene encoder: --set recipe=finetune-encoder, or its parts optimizer=encoder-lr-0.1, optimizer=layer-decay and hooks=freeze-encoder. They name the scene encoder, so they apply to the seed-based methods only.
  • Learning-rate-free and schedule-free optimizers: prodigy wants lr: 1.0; each schedulefree-* preset sets the optimizer, no schedule and its hook together.
  • One weight average per run: ema or swa; evaluation.weights: ema scores the average.
  • Point augmentation moves points, which a method that reads images cannot follow; dropout removes points and moves none.

Details

Where the recipe lives in a config
Config key Registry What it decides When it runs
optimizer OPTIMIZERS how gradients become weight updates once per train.accumulate batches
scheduler SCHEDULERS the learning rate over the run after every optimizer step or every epoch (train.scheduler_unit)
hooks HOOKS anything else around the loop: weight averages, freezing, clipping variants at before_run, before_epoch, before_step, after_iter, after_epoch, after_run
data.train.dataset.transforms TRANSFORMS augmentation of one training sample in the data-loader workers, per sample

Plain options of the loop need no component: train.amp, train.accumulate, train.grad_clip with train.grad_clip_mode, train.ema, train.sync_bn. Three method-neutral files under configs/_base_/schedules/ can be inherited without compose: adamw_onecycle_10e.yaml, adamw_warmup-cosine_10e.yaml, schedulefree-adamw_10e.yaml.

Input and output
  • Optimizer. Built from the optimizer node; the engine supplies params (or what the detector's configure_optimizers returns) and, where asked, model. The bridges return the library's own optimizer, so checkpoints hold its state unchanged.
  • Scheduler. Built per optimizer; the trainer steps it per optimizer step (train.scheduler_unit: iteration) or per epoch. The engine fills in the length of the run where a schedule needs it. scheduler: null is a constant rate.
  • Hook. A subclass of grasppanda.engine.hooks.Hook; every callback receives the training state. Hooks run in ascending priority (evaluation is 70, checkpointing 80). A weight average installs itself as state.ema, which evaluation.weights: ema scores.
  • Transform. Maps one PointSample to one PointSample. It may remove or reorder points only through sample.gather(index), which carries the labels; move the scene only with the object poses; and not move points at all when the sample carries images, as HGGD's does (the config check refuses that list). Scaling the scene is not allowed: grasp widths, depths and the gripper are in metres.
  • The config check tests every key against the constructor of the named class without importing the optional libraries; free-form arguments of a library go into options. A wrong optimizer name is found when the optimizer is built, with the list of valid ones.
Optimizers and parameter groups 
optimizer: {type: AdamW, lr: 1.0e-3, weight_decay: 0.05}                    # torch.optim, by class
optimizer: {type: TimmOptimizer, name: lamb, lr: 2.0e-3, weight_decay: 0.02}
optimizer: {type: PytorchOptimizer, name: prodigy, lr: 1.0}
optimizer: {type: ScheduleFree, name: adamw, lr: 2.5e-3, warmup_steps: 200}
  • lr and weight_decay are arguments of the bridge; everything else (betas, eps, momentum) goes into options. A lookahead_ prefix wraps any timm optimizer.
  • A schedule-free optimizer needs scheduler: null and ScheduleFreeHook; its presets set all three.

ParamGroups wraps any optimizer with a rate per module and layer-wise decay:

optimizer:
  type: ParamGroups
  optimizer: {type: AdamW, lr: 1.0e-3, weight_decay: 0.05}
  lr_multipliers: {backbone: 0.1}      # dotted prefix of the parameter name, or "re:<regex>"
  layer_decay: 0.75                    # null: off
  layer_decay_module: backbone
  layer_decay_depth: 2
  no_decay_1d: true                    # no weight decay on biases and normalisation weights

The rate of a parameter is lr × the first matching multiplier × layer_decay ** (L - i) for layer i of the L layers of layer_decay_module, counted in registration order (a U-Net's decoder counts as later layers). To train a pretrained encoder, load its weights with runtime.load_from=<checkpoint> and runtime.allow_weight_mismatch=true, then use the encoder presets.

Schedules, hooks and augmentation 
scheduler: {type: OneCycleLR, max_lr: 1.0e-3, pct_start: 0.05, div_factor: 10.0, cycle_momentum: false}
scheduler: {type: TimmSchedule, name: cosine, warmup_steps: 0.05, warmup_lr_init: 1.0e-6, lr_min: 1.0e-6}
hooks:
  - {type: EMAHook, decay: 0.999, warmup: true, interval: 1, buffers: copy}
  - {type: FreezeHook, modules: [backbone], epochs: 2, eval_mode: true}
evaluation: {weights: ema}
  • TimmSchedule.warmup_steps below 1 is a fraction of the run, from 1 a step count; warmup_lr_init and lr_min are absolute rates.
  • EMAHook, SWAHook and train.ema exclude each other. EMAHook warms its decay up and copies buffers by default (batch-norm statistics and HGGD's learned anchors stay those of the trained model).
  • FreezeHook sets requires_grad per epoch, so nothing is rebuilt when modules are released. AdaptiveGradClipHook and train.grad_clip exclude each other.
  • --set augment=dropout+shift+jitter writes RandomDropout before the sampling step and RandomShift and RandomJitter after the base list. Units are metres in the camera frame; RandomJitter perturbs only the network input.
Components and their sources

pip install -e ".[optim]" installs timm, pytorch_optimizer and schedulefree (all Apache-2.0); the rest needs nothing beyond the core install.

Component Kind What it is Code
TimmOptimizer, TimmSchedule optimizer, schedule any optimizer or schedule of timm by name bridge to timm.optim, timm.scheduler
PytorchOptimizer optimizer any optimizer of pytorch_optimizer by name bridge to pytorch_optimizer.load_optimizer
ScheduleFree, ScheduleFreeHook optimizer, hook schedule-free AdamW, RAdam or SGD (arXiv 2405.15682) and the switch between its training point and its average bridge to schedulefree
ParamGroups optimizer rate per module, layer-wise decay (BEiT, arXiv 2106.08254), no decay on 1-D parameters written here
classes of torch.optim and torch.optim.lr_scheduler optimizer, schedule AdamW, SGD, OneCycleLR, CosineAnnealingLR, … PyTorch
WarmupCosineLR schedule linear warm-up then cosine, in closed form (FineGrasp's schedule) written here
EMAHook, SWAHook hook weight averages (arXiv 1703.01780, arXiv 1803.05407) torch.optim.swa_utils.AveragedModel
FreezeHook hook freeze sub-modules for the first epochs (gradual unfreezing, arXiv 1801.06146) written here
AdaptiveGradClipHook hook adaptive gradient clipping (arXiv 2102.06171) bridge to timm.utils.adaptive_clip_grad
RandomJitter, RandomDropout, RandomShift transform noise on the points, removal of points with their labels, a shift of the scene and the object poses adapted from Pointcept @ 1342eda (MIT)
Pitfalls met while building this
  • OneCycleLR and momentum. Its default cycle_momentum: true rewrites betas or momentum every step; the presets set false. Its peak is max_lr: changing optimizer.lr changes nothing.
  • Schedule-free and batch norm. The evaluated weights are an average, but the batch-norm statistics were collected at the training point; this toolbox does not re-estimate them.
  • Short probes and warm-ups. With warmup_steps: 200 a 20-iteration probe runs inside the warm-up, so its losses fall slowly; that says nothing about a full run.
  • Per-epoch units and warm-up. With train.scheduler_unit: epoch a warm-up from 0 spends the first epoch at rate 0; multistep-warmup starts at 1e-4.
  • Lists replace. hooks and transforms of a child config replace the inherited list; compose writes the base's entries out again, a hand-written child must repeat them (HGGD needs its EpochSignalHook).
  • SWAHook.start_epoch must be below train.epochs; the run refuses to start otherwise. A short probe ends before averaging starts, so such a probe says nothing about fit.
  • Mixed precision is refused for the ported methods. When it was tried, the label assigner of the seed-based skeleton wrote half-precision values into a single-precision tensor, and HGGD's first loss was not finite. hooks=amp sets train.amp: true, which the config check now refuses unless the detector declares amp_ready = True; none of the shipped detectors does. The preset is the switch for a detector of your own.
  • A wrong optimizer name passes the config check and fails when the optimizer is built, with the list of valid names.
Not wrapped
Candidate Why not
Normalisation and activation as config keys no shared layer builder exists in the stages; a hook that swapped modules after construction would not run at test time.
RandomScale, ElasticDistortion grasp widths, depths and the gripper are in metres, and labels attach to rigid object poses.
Chromatic transforms the scene encoders read no colour; HGGD's image pipeline is not a point transform.
Mix3D scene mixing works in the collate function, and two scenes' labels would have to be merged.
Sharpness-aware minimisation (SAM) its step() needs a second forward and backward pass.
ReduceLROnPlateau steps on a monitored metric, which the trainer does not hand to schedules; refused by name.
Batch-norm re-estimation after SWA update_bn iterates plain tensors; batches here are Batch objects.