Skip to content

How we measure

Every option was composed under each method it fits; those accepted were probed on real frames. The few without a record of their own (two adapters compose inserts, two point-painting wrappers, post=target-mask and loss=distribution-focal, measured inside decoupled-grid-bins), the refusals and the options that need your consent are named per option.

A probe shows that a combination trains on real frames of GraspNet-1Billion, not how good it is. Accuracy is claimed only for a method trained and evaluated here.

Slot by slot

The 244 options a user can --set, slot by slot:

Slot Options Run under every method tried Train like shipped under every method compared
Image encoder 38 36 of 37 23 of 36
Scene encoder 47 37 of 47 9 of 47
Seed proposal 24 22 of 24 1 of 24
View 21 19 of 21 3 of 21
Assigner 4 0 of 4 1 of 3
Grouper 28 25 of 28 6 of 28
Head bundle 10 8 of 10 3 of 10
Loss 20 17 of 20 10 of 19
Post-processing 15 14 of 14 — (trains nothing)
Optimizer 17 15 of 17 0 of 14
Schedule 7 1 of 7 — (held fixed)
Hooks 7 4 of 7 4 of 6
Augmentation 3 0 of 3 2 of 3
Recipe 3 2 of 3 — (own schedules)
Which slot applies to which method
Slot GraspNet-baseline GSNet Scale-Balanced-Grasp EconomicGrasp FineGrasp HGGD
Image encoder — — — — — ✓
Scene encoder ✓ ✓ ✓ ✓ ✓ —
Seed proposal ✓ ✓ ✓ ✓ ✓ —
View ✓ ✓ ✓ ✓ ✓ —
Assigner ✓ ✓ ✓ ✓ ✓ —
Grouper ✓ ✓ ✓ ✓ ✓ —
Head bundle ✓ ✓ ✓ ✓ ✓ —
Loss ✓ ✓ ✓ ✓ ✓ —
Post-processing ✓ ✓ ✓ ✓ ✓ ✓
Optimizer ✓ ✓ ✓ ✓ ✓ ✓
Schedule ✓ ✓ ✓ ✓ ✓ ✓
Hooks ✓ ✓ ✓ ✓ ✓ ✓
Augmentation ✓ ✓ ✓ ✓ ✓ ✓
Recipe ✓ ✓ ✓ ✓ ✓ ✓

The four levels

A cell, one option under one method, is judged at four levels; each needs the one before it.

  • composes: the config check accepts the composed config.
  • runs: it trains on real frames without an error.
  • learns: its losses fall.
  • predicts: it ranks grasps: every frame has one, and the scores differ.
Exact definitions, and why a level can stay undecided
  • composes: the contract rules and the config check accept the composed config: grasppanda compose writes it (any verdict but refused). A refusal by policy is undecided (needs consent), and so is a request that is not a combination.
  • runs: the probe ran clean on real frames: the verdict runs or runs-empty. A runs-unstable probe counts as not running only when its repeat (repeat in the record: the same config, seed and schedule) was unstable too; without a repeat the level is undecided (unstable, not repeated); when the repeat ran clean the cell runs, and learns and predicts are read from the repeat. fails does not run.
  • learns: judge 2 only. Every loss term was active (reported, finite, not exactly 0) in at least 50 % of the iterations and fell by at least 5 % (fall = (first - mean of the last quarter) / |first| over its active iterations), and no term is flagged flat, rising or constant; a term that is 0 at the last iteration (zero_at_end) is judged on its active iterations. Every unit with trainable parameters received a gradient (no_gradient is empty) and every part changed (unchanged is empty).

  • predicts: judge 2 only. Every predicted frame has at least one grasp, and on every frame the smallest and the largest score differ (score_quantiles): a model whose scores collapsed to one value ranks nothing.

  • not checked: the record was judged by the first judge, which recorded neither the trend of the loss terms nor the scores of the predicted grasps.

  • unstable, not repeated: the probe was runs-unstable and has not been repeated: an unstable run counts as not running only when a second probe of the same config is unstable too.
  • needs consent: refused by a policy that waits for the user's consent (trust_remote_code needs accept_remote_code: true): a rule about this machine, not about the combination.
  • not a combination: the request names a component that lacks an option only a preset supplies: it is refused before any contract is compared, and is no combination.
  • not probed: the grid has the cell, but no record yet.
  • not reached by the method as shipped: the method as shipped, probed the same way, does not reach this level either (for example FineGrasp at the constant rate, whose fusion of crop scales gives every seed one feature in a short run, so every grasp gets one score): under that method the level says nothing about the option at this budget.

Probe verdicts

  • runs: built, trained on real frames with finite losses, and predicted grasps.
  • runs-empty: as runs, but no probed frame has a grasp.
  • runs-unstable: a loss or a weight became non-finite, or a part got no seed or no gradient.
  • fails: an exception after the config check had passed.
  • refused: the config check said no, with the fix; nothing was built.
  • wired: a synthetic scene only (no dataset): shows the stages connect, nothing else.
Exact definitions
  • runs: On real frames: the detector was built, every requested training iteration ran with finite losses and at least one real seed, every top-level part of the model that holds trainable parameters received a gradient in at least one iteration, the weights stayed finite, and prediction returned one valid grasp group for every probed frame, at least one of them with a grasp.
  • runs-empty: As runs, except that no probed frame has a grasp: prediction returned one valid grasp group for every probed frame and every group is empty. Every stage ran; a model trained for a few iterations often keeps no seed. A run that is also unstable is runs-unstable.
  • runs-unstable: On real frames: the detector was built, trained and predicted, but a loss or a weight became non-finite (a non-finite loss ends the training there), or an iteration had no real seed (no scene point passed the proposal, so only the proposal was trained), or no iteration supervised a seed, or a top-level part with trainable parameters received no gradient in any iteration, or fewer iterations ran or fewer frames were predicted than were asked for.
  • fails: An exception was raised after the config check had passed. The report names the phase (build, data, optimizer, loss, backward, step, hook, predict), the component it came from and the message.
  • refused: The config check refused the config (the same check as grasppanda print-config --check --for train). Nothing was built; the report carries the messages.
  • wired: No real frame was read. On a synthetic table-top scene with random labels the detector was built, back-propagated a loss through every stage and predicted. This shows that the stages connect; it says nothing about real data.

Trains like shipped

Level 2: a swap that ran trains 250 iterations beside the shipped component, and their losses are compared term by term:

  • on a par: every comparable loss ends within the method's tolerance of its own component's.
  • worse: a comparable loss ends above that tolerance, or a part got no gradient.
  • not comparable: the option redefines every loss term.
  • diverging: a loss rises at the end, a value became non-finite, or no grasp was predicted.
  • did not train: the run did not finish (record value fails).

Level 2 needs runs, not learns: a term flat after 20 iterations can fall by iteration 250.

Exact definitions, and what level 2 leaves out
  • fails: the run did not finish: an exception, a crash, a timeout, or no real frame read. A run that died of GPU memory is not recorded (it says nothing about the config) and is run again.
  • no reference: the method as shipped has no finished run under the same settings: nothing to compare with.
  • diverging: a loss term is rising over the last half of the run (see rising), a loss or a weight became non-finite, or no grasp was predicted on any of the 4 frames while the method as shipped predicts some.
  • worse: not diverging, and a comparable loss term ends above the tolerance of the method times the shipped method's (mean of the last 25 iterations), or a part of the model that holds trainable parameters received no gradient in any iteration.
  • not comparable: not diverging, nothing worse, but the cell changes the definition of every term the method as shipped has: there is no term to compare.
  • on a par: every comparable loss term ends within the tolerance of the method, at least one term is comparable, nothing rises, every value is finite and a frame has a grasp. A term far below the shipped method's is reported (lower), not judged.

Not compared, on every page:

  • Post-processing: post-processing trains nothing: it changes predictions, not the training.
  • Schedules: level 2 holds the schedule of the method as shipped fixed (a constant rate; FineGrasp: its own warm-up), so a schedule swap is not compared.
  • A cell on a schedule of its own (a recipe preset with a scheduler, a schedule-free optimizer): it trained on a schedule of its own, not on that of the method as shipped.
  • A swap that did not run: the swap did not run at level 1; it stays named.

Why a request is refused

  • incompatible by design (structural): a genuine incompatibility.
  • not supported yet (not-implemented): the combination makes sense, but the code that joins the two parts does not exist yet.
  • needs a setting compose adds (fixed-by-compose): an adapter or an option lifts it, and grasppanda compose inserts or sets it.
  • needs a preset's settings (not-a-combination): the components fit, but this config does not state them fully.
  • needs your consent (policy): refused on purpose until the config says otherwise (code a model hub would run on this machine).

What it was measured on

  • Does it run? 20 training iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, one run per cell; nothing was measured on GraspClutter6D or on Kinect frames.
  • Does it train like the shipped component? 250 iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, at a constant learning rate (FineGrasp: its own warm-up), beside the method as shipped trained the same way twice.