Grouper¶
--set grouper=28 optionsThe grouper looks at the neighbourhood of every seed in the frame of its approach direction and turns it into one feature. The head predicts angle, depth, width and score from that feature alone, so the grouper decides what the head can know about the local shape.
Use it¶
grasppanda compose graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e \
--set grouper=roi-grid -o graspnet-baseline_roi-grid.yaml
grasppanda compose graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e --list grouper
grasppanda compose gsnet/gsnet_sunet14_g1b-realsense_10e \
--set grouper=roi-grid -o gsnet_roi-grid.yaml
grasppanda compose gsnet/gsnet_sunet14_g1b-realsense_10e --list grouper
grasppanda compose scale_balanced_grasp/sbg_pointnet2-transformer_g1b-realsense_18e \
--set grouper=roi-grid -o scale-balanced-grasp_roi-grid.yaml
grasppanda compose scale_balanced_grasp/sbg_pointnet2-transformer_g1b-realsense_18e --list grouper
grasppanda compose economicgrasp/economicgrasp_sunet14_g1b-realsense_10e \
--set grouper=roi-grid -o economicgrasp_roi-grid.yaml
grasppanda compose economicgrasp/economicgrasp_sunet14_g1b-realsense_10e --list grouper
grasppanda compose finegrasp/finegrasp_sunet14_g1b-realsense_10e \
--set grouper=roi-grid -o finegrasp_roi-grid.yaml
grasppanda compose finegrasp/finegrasp_sunet14_g1b-realsense_10e --list grouper
The first command writes a checked config; the second lists every option of the slot under that method, with what was measured. Then grasppanda probe <file> tries the config on real frames and grasppanda train <file> trains it (Swap a component). Each method page lists the same: GraspNet-baseline · GSNet · Scale-Balanced-Grasp · EconomicGrasp · FineGrasp.
Options¶
A probe shows that a combination trains on real frames of GraspNet-1Billion, not how good it is. Accuracy is claimed only for a method trained and evaluated here.
What the columns say, and who is left out
- Runs: out of the methods the option is a swap under, those under which its probe trained on real frames without error. A method that ships the option is named own and not counted; a method still undecided is counted and named. Measured on 20 training iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, one run per cell; nothing was measured on GraspClutter6D or on Kinect frames.
- Trains like shipped: out of the methods under which the swap ran, those under which every comparable loss term ended within the method's tolerance of the component it ships, with nothing rising; a method left out is named with why. Measured on 250 iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, at a constant learning rate (FineGrasp: its own warm-up), beside the method as shipped trained the same way twice.
- How we measure defines every word of these columns.
Who is left out, and why:
depth-pool, runs: not GraspNet-baseline, Scale-Balanced-Grasp (incompatible by design).multi-scale, trains like shipped: worse under GraspNet-baseline; diverging under Scale-Balanced-Grasp.voxel-set-abstraction, runs: not GraspNet-baseline, Scale-Balanced-Grasp (not supported yet).
Sources and extras
The work each option follows; extra: the optional install it needs (every extra).
| Source | Options |
|---|---|
| Parts from PointNet++ (Qi et al., NeurIPS 2017), DGCNN (Wang et al., TOG 2019), PointRCNN (Shi et al., CVPR 2019), RandLA-Net (Hu et al., CVPR 2020), Point Transformer (Zhao et al., ICCV 2021) and PointNeXt (Qian et al., NeurIPS 2022), written here on grasppanda.ops | ball, box, cylinder-attentive, cylinder-edgeconv, cylinder-invres, cylinder-vector-attention, knn |
| GSNet (Wang et al., ICCV 2021) | CylinderGroup |
| EconomicGrasp (Wu et al., ECCV 2024) | CylinderInteractionGroup |
| GraspPanda (axis adapter; no published model) | depth-pool, depth-embed |
| FineGrasp (arXiv 2507.05978) | multi-scale |
| Multi-scale grouping of PointNet++ (Qi et al., NeurIPS 2017) on the seed crops of the grasp methods, written here | multi-scale-linear |
PyTorch Geometric (Fey and Lenssen, 2019), MIT: PointNetConv, PointTransformerConv, EdgeConv, GATv2Conv, TransformerConv; via torch_geometric · extra torch_geometric |
pyg-edge-conv, pyg-gatv2-conv, pyg-point-transformer-conv, pyg-pointnet-conv, pyg-transformer-conv |
| Part-A² (Shi et al., TPAMI 2020), RoI-aware point cloud pooling; adapted from open-mmlab/OpenPCDet@233f849, Apache-2.0 | roi-aware |
| PV-RCNN (Shi et al., CVPR 2020), RoI-grid pooling; adapted from open-mmlab/OpenPCDet@233f849, Apache-2.0 | roi-grid |
| PV-RCNN (Shi et al., CVPR 2020), voxel set abstraction; adapted from open-mmlab/OpenPCDet@233f849, Apache-2.0 | voxel-set-abstraction |
| The depth-sliced crop of GraspNet-baseline (Fang et al., CVPR 2020) with the shapes and poolings of LocalGroup, written here on grasppanda.ops | depth-ball, depth-box, depth-cylinder-attentive, depth-knn |
| GraspNet-baseline (Fang et al., CVPR 2020) | depth-cylinder-multi-scale, DepthCylinderGroup |
| Scale-Balanced-Grasp (Ma and Huang, CoRL 2022) crop with a projection of the seed feature added here | depth-cylinder-projection |
Choosing¶
- Match the head. A grouper returns one feature per seed or one per depth bin (the two
tabs above), and the head must read the same; compose inserts
depth-embedordepth-poolwhere they differ, and says so. - The shape of the crop. A cylinder along the approach axis is what the ported methods
use;
ballhas no axis,boxis a cylinder with a square section,knnfollows the seed density. - Keep where a feature was found.
roi-gridandroi-awarepool the cells of a gripper-aligned box instead of taking one maximum over the crop. - Several radii:
multi-scale-linearfuses four crops with one linear layer,multi-scalewith attention. - Sizes belong to the seed density. A crop among the seeds sees the seeds of the proposal; after changing one, check the other.
Details¶
The five steps of a crop
Every published crop is the same five steps, and almost every option replaces one of them.
| Step | What the ported methods do | What the options add |
|---|---|---|
| query: which points, in which shape | a cylinder along the approach axis | ball, k nearest, oriented box, grid of balls, grid of cells |
| geometry of a neighbour | offset in the crop frame | the same with its length |
| encoder per neighbour | shared MLP (plus self-attention in EconomicGrasp) | DGCNN edge feature, PointNeXt inverted residual, layers of PyTorch Geometric |
| pooling | maximum | mean, attentive pooling, vector attention, all with a padding mask |
| fusion, residual | attention over radii (FineGrasp), linear + gate (Scale-Balanced-Grasp) | linear, max or mean over radii; a projected seed residual |
Input and output
class Grouper(nn.Module):
support: ClassVar[Literal["seeds", "scene"]] # which points are grouped
out_axes: ClassVar[tuple[str, ...]] # ("seed", "channel") or ("seed", "depth", "channel")
out_channels: int # width of the returned feature
uses_seed_features: bool = False # forward takes the seed features too
def forward(self, support, centres, rotation[, seed_feats]) -> GroupFeatures: ...
support |
points (N, 3) in metres, camera frame, with features (N, C). With support = "seeds" the seeds of the batch; with support = "scene" the whole scene cloud, with zero feature channels under a sub-sampling encoder. |
centres |
(M, 3) crop centre per seed: the seed at inference, the centre the assigner chose in training (so a crop may be empty). |
rotation |
(M, 3, 3) crop frame; column 0 is the approach axis. In training it may be a reflection: do not assume a right-handed frame. |
| returns | GroupFeatures: (M, out_channels) or (M, D, out_channels), one row per seed. |
What grasppanda compose checks before anything is built:
out_axesequals the head'sin_axes. An axis is never added or squeezed implicitly;depth-embedanddepth-pooldo it explicitly.- One depth entry per depth bin of the head: the length of
hmax, ornum_depth, equalshead.num_depth. - The width of the seed features a grouper adds to its crops equals what the encoder returns (below).
The queries return the first nsample hits in index order and repeat the first hit in
empty slots; a maximum does not notice the copies, a mean or an attention would, so the
groupers mask them.
The settings that matter
LocalGroup: shape, encoder and pooling are options (all 48 combinations are legal and
unit-tested).
model:
grouper:
type: LocalGroup
shape: cylinder # ball | knn | box | cylinder
radius: 0.05 # ball / cylinder radius, half side of the box (m)
hmin: -0.02 # box and cylinder: ends along the approach axis (m)
hmax: 0.04
nsample: 16 # neighbours per crop (k of knn)
encoder: mlp # mlp | edge | invres
pool: max # max | mean | attentive | vector_attention
mlp_channels: [256, 256] # the last width goes to the head
edge tells the MLP how a neighbour differs from the seed; invres adds capacity after the
pooling. max is robust to padding; vector_attention is the only pooling that reads the
seed's own feature.
DepthLocalGroup is the same over the scene, gathered once to the farthest reach; bin
d pools the members with hmin < x < hmax[d], so near bins hold fewer members.
seed_residual: projected works with any encoder width.
RoI poolings (roi-grid, roi-aware) keep where in the box a feature was found.
roi-aware leaves empty cells at zero, so with 1024 seeds most of its 64 cells are empty:
try grid_size: 2 or 3. voxel-set-abstraction reads the scene and returns one feature
per seed; scene_features: true needs a dense encoder.
PyGConvGroup hands the crop to a layer of PyTorch Geometric as a bipartite graph: the
members at their offsets in the crop frame, the seed at the origin, nothing padded.
conv_args passes further arguments, for example --opts 'model.grouper.conv_args={heads: 8}'.
Another layer needs three lines in PYG_CONVS (grasppanda/models/groupers/pyg.py).
Several radii: MultiRadiusGroup runs one copy of branch per radius and fuses them
(linear, max, mean); branch is any seeds-support grouper with a radius.
MultiScaleGroup is the same with FineGrasp's attention over the scales.
The depth adapters: depth-embed computes a per-seed crop once and tells the depth bins
apart by a learned embedding; depth-pool takes the maximum over the bins of a scene crop.
They make a swap possible; no paper proposes them.
No grouper has released weights. To start the other stages from a checkpoint of the method,
use runtime.load_from=<file> with runtime.allow_weight_mismatch=true.
The width of the seed features
Scale-Balanced-Grasp's crop adds sigmoid(W f) · f of the seed feature f to its
256-channel crop, so the encoder must return 256 channels. DepthCylinderGroup states the
width it needs, and grasppanda compose compares it with the width the encoder declares.
When they differ, compose puts DepthCylinderProjectionGroup in its place (the same crop
with a linear layer in front of the residual) and says so; --no-adapter leaves the pair
refused with the ways out. The composed model is then a variant of Scale-Balanced-Grasp,
not the paper's.
The projection is a class of its own on purpose: the config hash of a shipped config covers
the constructor defaults of its components, so a new option on a ported class would change
the hash of every config that uses it. MultiRadiusGroup beside MultiScaleGroup follows
the same rule.
Code and licences
The LocalGroup and DepthLocalGroup parts are written here on grasppanda.ops from the
papers (a few lines each; the official RandLA-Net code is CC BY-NC-SA and was not used).
RoIGridGroup, RoIAwareGroup and VoxelSetAbstractionGroup adapt the pooling rules of
OpenPCDet@233f849
(Apache-2.0) onto grasppanda.ops. PyGConvGroup is a bridge to torch_geometric (MIT;
extra torch_geometric). CylinderGroup, CylinderInteractionGroup, MultiScaleGroup and
DepthCylinderGroup are the crops of the ported methods. No grouper has weights.
Pitfalls met while building this
- Padding and non-max pooling. Empty slots repeat the first hit; with a mean or an
attention those copies would outvote the real members. Use the mask
crop_membersreturns in a pooling of your own. - Empty crops in training. The crop centre need not be a seed, so a seed-support crop may be empty. Its first slot then counts as the only member, so no pooling reduces an empty set.
- Do not add an option to a component that shipped configs use. It changes their config hash, and every measured cell built from them stops matching what compose writes. New behaviour goes into a new class or a subclass.
- A bridge passes the config check without its library.
PyGConvGroupwithouttorch_geometricstops when the model is built, with the install line. knnover the scene is short-sighted. With 20 000 points the 64 nearest lie within about a centimetre, so the far depth bins ofdepth-knnsee the same members as the near ones. Raisensampleor prefer a radius.- Memory. A scene-support crop holds
seeds × nsample × widthactivations per layer (and per depth bin).
Not wrapped
| Candidate | Why not |
|---|---|
| DOGraspNet's offset crop (code) | a non-commercial licence; the idea is encoder: edge, written here. |
| CUDA extensions of OpenPCDet and MMDetection3D | compiled, not installed here, and they rotate about one axis only; the rules were adapted onto grasppanda.ops. |
| Voxel RoI pooling (Voxel R-CNN), the multi-stride sources of PV-RCNN | they read intermediate levels of a sparse encoder; the grouper gets the final features only. |
| Closing-region crops (GPD, PointNetGPD, REGNet) | they need the complete grasp pose, but the grouper runs before the head knows it; they belong to a re-scoring stage the skeleton does not have. |
| Region patches of the image-to-region family (RegionNormalizedGrasp) | no licence file, and the seeds of this slot are points, not image patches. |
| KPConv, PosPool, point-pair features, Set-Transformer pooling | not reached yet; each would be one more encoder or pool value of LocalGroup. |