View¶
Seed-based · 5 methods
--set view=21 optionsThe view stage chooses the approach direction of every seed: two of the three degrees of freedom of the grasp orientation. The grouper crops the scene in the frame it chooses, so it decides what the head gets to see.
Use it¶
grasppanda compose graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e \
--set view=lattice-view -o graspnet-baseline_lattice-view.yaml
grasppanda compose graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e --list view
grasppanda compose gsnet/gsnet_sunet14_g1b-realsense_10e \
--set view=lattice-view -o gsnet_lattice-view.yaml
grasppanda compose gsnet/gsnet_sunet14_g1b-realsense_10e --list view
grasppanda compose scale_balanced_grasp/sbg_pointnet2-transformer_g1b-realsense_18e \
--set view=lattice-view -o scale-balanced-grasp_lattice-view.yaml
grasppanda compose scale_balanced_grasp/sbg_pointnet2-transformer_g1b-realsense_18e --list view
grasppanda compose economicgrasp/economicgrasp_sunet14_g1b-realsense_10e \
--set view=lattice-view -o economicgrasp_lattice-view.yaml
grasppanda compose economicgrasp/economicgrasp_sunet14_g1b-realsense_10e --list view
grasppanda compose finegrasp/finegrasp_sunet14_g1b-realsense_10e \
--set view=lattice-view -o finegrasp_lattice-view.yaml
grasppanda compose finegrasp/finegrasp_sunet14_g1b-realsense_10e --list view
The first command writes a checked config; the second lists every option of the slot under that method, with what was measured. Then grasppanda probe <file> tries the config on real frames and grasppanda train <file> trains it (Swap a component). Each method page lists the same: GraspNet-baseline · GSNet · Scale-Balanced-Grasp · EconomicGrasp · FineGrasp.
Options¶
A probe shows that a combination trains on real frames of GraspNet-1Billion, not how good it is. Accuracy is claimed only for a method trained and evaluated here.
What the columns say, and who is left out
- Runs: out of the methods the option is a swap under, those under which its probe trained on real frames without error. A method that ships the option is named own and not counted; a method still undecided is counted and named. Measured on 20 training iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, one run per cell; nothing was measured on GraspClutter6D or on Kinect frames.
- Trains like shipped: out of the methods under which the swap ran, those under which every comparable loss term ended within the method's tolerance of the component it ships, with nothing rising; a method left out is named with why. Measured on 250 iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, at a constant learning rate (FineGrasp: its own warm-up), beside the method as shipped trained the same way twice.
- How we measure defines every word of these columns.
Who is left out, and why:
lattice-angular-cost, trains like shipped: worse under GSNet, Scale-Balanced-Grasp, EconomicGrasp, FineGrasp.lattice-best-view, trains like shipped: worse under GSNet, Scale-Balanced-Grasp, EconomicGrasp, FineGrasp.objectness-view, runs: own in GraspNet-baseline; not EconomicGrasp, FineGrasp (incompatible by design).objectness-view-graspable, runs: own in Scale-Balanced-Grasp; not EconomicGrasp, FineGrasp (incompatible by design).direction-regression, trains like shipped: worse under GSNet, EconomicGrasp, FineGrasp; diverging under Scale-Balanced-Grasp.direction-vmf, trains like shipped: worse under GSNet, Scale-Balanced-Grasp, EconomicGrasp, FineGrasp.rotation-6d, trains like shipped: worse under GSNet, EconomicGrasp, FineGrasp; diverging under Scale-Balanced-Grasp.rotation-9d, trains like shipped: worse under GSNet, EconomicGrasp, FineGrasp; diverging under Scale-Balanced-Grasp.rotation-quaternion, trains like shipped: worse under GSNet, Scale-Balanced-Grasp, EconomicGrasp, FineGrasp.two-vector, trains like shipped: worse under GraspNet-baseline, GSNet, EconomicGrasp, FineGrasp.two-vector-baseline, trains like shipped: worse under GSNet, Scale-Balanced-Grasp, EconomicGrasp, FineGrasp.
Choosing¶
- The labels know 300 directions. Every option snaps its choice to one of the 300 label views for training, so a finer lattice does not give finer labels.
- Fewer classes:
lattice-fibonacci-120,icosphere-162orhealpix-192; only approaches away from the camera:lattice-hemisphere-60. - A classification instead of 300 regressions:
lattice-best-view;lattice-angular-costmakes near misses cheap. - No lattice at all:
direction-regression, with a confidencedirection-vmf; a whole grasp frame:two-vectororrotation-6d. - The head keeps the in-plane angle. A stage that regresses a full rotation contributes its approach axis; the rest of the frame is supervised and logged, not decoded.
Details¶
Which option for which need
| You want | Take | Why |
|---|---|---|
| the view scoring of GSNet, EconomicGrasp and FineGrasp | template-view |
a score per label view; the training view is sampled from the scores |
| the view stage of GraspNet-baseline or Scale-Balanced-Grasp | objectness-view, objectness-view-graspable |
also predicts objectness, which their heads read |
| the same scoring as a base for the options below | lattice-view |
computes what template-view computes, on a lattice of any size |
| a smaller output layer | lattice-fibonacci-120, icosphere-162, healpix-192 |
fewer classes; each view is supervised by its nearest label view |
| only approaches away from the camera | lattice-hemisphere-60 |
the 60-of-120 template of RGB Matters |
| a classification of the best view | lattice-best-view, lattice-angular-cost |
cross-entropy on the most graspable view; the second adds the angular class loss of RGB Matters |
| a direction between lattice points at inference | lattice-soft-argmax |
softmax-weighted mean of the best view directions |
| training crops closer to what inference chooses | lattice-topk |
samples the training view among the 10 best views |
| no lattice in the output | direction-regression, direction-vmf |
3 numbers per seed and a cosine loss; or a von Mises-Fisher likelihood with a concentration |
| a full grasp frame | two-vector, two-vector-baseline, rotation-6d, rotation-9d, rotation-quaternion |
Gram-Schmidt of two vectors, or the rotation mappings of RoMa |
Input and output
approach, seeds = view(seeds), and view.loss(approach, matched) for its own loss.
| receives | SeedSet: M seeds of the batch, coordinates (M, 3) in metres in the camera frame and features (M, C); C is injected (in_channels) |
| returns | ApproachSet, one row per seed, and the SeedSet (a stage may add a residual to the features, a score, or mark seeds invalid) |
direction |
(M, 3) unit vectors; read by the decoder of every head: rotation = approach_to_rotation(direction, in-plane angle) |
rotation |
(M, 3, 3), column 0 the direction; the crop frame of the grouper at inference |
view_index |
(M,) index into view.template, the 300 views the labels are indexed by; the assigner looks the labels up there in training |
| declares | label_formats (dense or compact labels it can read), needs_matched, seed_scores (scores it adds), applies_seed_weights |
Three facts shape every option:
- The labels know 300 directions. GraspNet-1Billion and GraspClutter6D labels are indexed by a 300-view Fibonacci lattice. A stage may predict on another lattice or none, but it snaps its choice to one of those views for the label lookup.
- Train and inference crops differ by the snap. In training the crop frame is the label frame of the snapped view; at inference it is the direction the stage returned.
- The head owns the in-plane angle. Every head decodes
approach_to_rotation(direction, angle bin).
What each kind of stage needs:
| Stage | From the labels | From the assigner | From the head |
|---|---|---|---|
lattice stages (Lattice, Icosphere, Healpix) |
the graspness of all 300 label views per seed (dense or compact) | the label lookup at view_index |
nothing |
DirectionViewStage |
the same, reduced to the most graspable view | the same | nothing |
TwoVectorViewStage, RomaRotationViewStage |
the same, plus the grid or angle bin of the snapped view | also crop_rotation, the label frame of the snapped view |
the same num_angle (checked) |
No new stage has released weights. A checkpoint of a ported method loaded into a model with another view stage leaves only that stage untrained.
The settings that matter
A lattice stage:
model:
view:
type: LatticeViewStage # or IcosphereViewStage (subdivisions) / HealpixViewStage (nside)
num_views: 300
hemisphere: false # true: only approaches away from the camera
target: score # score | best_view
selection: argmax # inference: argmax | soft_argmax
train_selection: multinomial # training: multinomial | topk | argmax
top_k: null # used by soft_argmax and topk
temperature: 1.0 # softmax temperature of soft_argmax
angular_weight: 0.0 # best_view only: weight of the angular class loss
- Size. An icosphere has
10 · 4**subdivisions + 2views, HEALPix12 · nside**2. Beyond 300 views neighbouring views share one label column. target: scoreregresses the normalised graspness of every view (GSNet's).target: best_viewclassifies the most graspable view; use it withtrain_selection: argmaxand aCrossEntropyloss.selection: soft_argmaxchanges inference only. A temperature of about 0.05 fits regressed scores in [0, 1], 1.0 fits logits; keeptop_ksmall (8).
A regression stage:
model:
view:
type: TwoVectorViewStage
keep: approach # approach (6-D order) | baseline (Contact-GraspNet order)
direction_weight: 1.0 # loss/view: 1 - cos to the most graspable label view
frame_weight: 1.0 # loss/view_frame: distance to the label frame, half-turn aware
num_angle: 12 # must equal the head's
loss/view_frametakes the smaller of the distances to the label frame and to its half-turn copy: a parallel-jaw grasp turned by 180° about its approach is the same grasp.representationofRomaRotationViewStage:6dand9dare continuous;quaternionandrotvecare there for comparison (rotvechas no preset and no probe record).residual: trueadds the hidden features of the stage to the seed features, as GSNet's stage does.
Sources and licences
| Component | Idea from | Code | Extra |
|---|---|---|---|
TemplateViewStage, ObjectnessViewStage |
GSNet (ICCV 2021), GraspNet-1Billion (CVPR 2020) | written for GraspPanda | none |
LatticeViewStage |
the same scoring on a free lattice; the hemisphere template of RGB Matters (arXiv 2103.02184, ICRA 2021) | written for GraspPanda; the angular class loss adapted from GouMinghao/rgb_matters@2e176728f6 (MIT, see THIRD_PARTY_NOTICES.md) |
none |
IcosphereViewStage |
subdivided icosahedron | bridge to trimesh (MIT) | trimesh |
HealpixViewStage |
HEALPix (Górski et al., ApJ 2005) | bridge to healpix (BSD-3-Clause) | healpix |
DirectionViewStage |
the approach network of Generalizing-Grasp (arXiv 2404.01727, CVPR 2024); the vMF likelihood of vMF-Contact (arXiv 2411.03591, ICRA 2025) | written for GraspPanda (both repositories have no licence file) | none |
TwoVectorViewStage |
the 6-D rotation representation (arXiv 1812.07035, CVPR 2019); the order of Contact-GraspNet (arXiv 2103.14127) | written for GraspPanda (the idea only) | none |
RomaRotationViewStage |
6-D, 9-D + SVD (arXiv 2006.14616), quaternion, rotation vector | bridge to RoMa (BSD-3-Clause) | roma |
Pitfalls met while building this
-
A finer lattice does not give finer labels. Angles for 200 000 random directions, computed with the lattices of
grasppanda.ops:Lattice Views Random direction to nearest view, mean / max View to its nearest label view, mean / max Label columns read Fibonacci (the labels) 300 4.5° / 9.0° 0° 300 Fibonacci 120 7.1° / 14.1° 4.6° / 7.7° 120 icosphere 162 6.1° / 10.8° 4.5° / 7.7° 162 icosphere 642 3.0° / 5.4° 4.5° / 8.1° 300 HEALPix 192 5.6° / 11.5° 4.5° / 7.9° 190 HEALPix 768 2.8° / 5.6° 4.5° / 8.4° 300 The third column is the price of a coarse lattice; the fourth is the crop mismatch of any lattice that is not the label lattice.
-
TemplateViewStage(num_views=N)with N ≠ 300 is not another lattice size. Its template is also the label lattice, so the assigner refuses the labels. UseLatticeViewStage(num_views=N). - Soft-arg-max needs a temperature that fits the outputs. With regressed scores and a temperature of 1 the weights are nearly uniform; with every view voting, opposite views can cancel (the stage then falls back to the best view).
- A single best view is a brittle target. On a symmetric object two distant views can
tie and the target flips between them;
target: scoreandangular_weightsuffer less. - Flip augmentation yields mirrored label frames. The frame loss skips those seeds; their direction loss still counts.
- The vMF loss is a log-density and goes negative when the concentration is high.
- healpy is GPL-2.0, so the bridge uses the BSD-licensed
healpixpackage.
Not wrapped
| Candidate | Why not |
|---|---|
| Contact-GraspNet code (NVlabs/contact_graspnet) | its licence limits use to non-commercial research. Its parameterisation is two-vector-baseline; its contact labels need an asset the toolbox does not have. |
| vMF-Contact, Generalizing-Grasp and S4G code | no licence files. The closed-form likelihood, the cosine regression and the 6-D representation are written here; their other terms are not. |
| DexGraspNet 2.0 (diffusion on seed features) | no licence file; a generative head is a head, not a view stage. |
| OrbitGrasp, Edge Grasp Network | detectors of their own with their own simulated labels. |
| PyTorch3D rotation conversions | a compiled package for functions RoMa has in pure PyTorch. |
| Several grasps per seed at inference | ApproachSet has one row per seed; it needs a change of the skeleton. |
| A head that takes the regressed in-plane rotation | belongs to the head slot; until it exists the frame of the rotation stages is auxiliary. |