Skip to content

Scene encoders

Seed-based · 5 methods--set backbone=47 options

The scene encoder is the first learned stage of a seed-based method: it turns the point cloud of one depth view into one feature vector per point. Its reach and resolution decide what a seed knows about its surroundings before any crop is taken.

Use it

grasppanda compose graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e \
    --set backbone=ptv3 -o graspnet-baseline_ptv3.yaml
grasppanda compose graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e --list backbone

grasppanda compose gsnet/gsnet_sunet14_g1b-realsense_10e \
    --set backbone=ptv3 -o gsnet_ptv3.yaml
grasppanda compose gsnet/gsnet_sunet14_g1b-realsense_10e --list backbone

grasppanda compose scale_balanced_grasp/sbg_pointnet2-transformer_g1b-realsense_18e \
    --set backbone=ptv3 -o scale-balanced-grasp_ptv3.yaml
grasppanda compose scale_balanced_grasp/sbg_pointnet2-transformer_g1b-realsense_18e --list backbone

grasppanda compose economicgrasp/economicgrasp_sunet14_g1b-realsense_10e \
    --set backbone=ptv3 -o economicgrasp_ptv3.yaml
grasppanda compose economicgrasp/economicgrasp_sunet14_g1b-realsense_10e --list backbone

grasppanda compose finegrasp/finegrasp_sunet14_g1b-realsense_10e \
    --set backbone=ptv3 -o finegrasp_ptv3.yaml
grasppanda compose finegrasp/finegrasp_sunet14_g1b-realsense_10e --list backbone

The first command writes a checked config; the second lists every option of the slot under that method, with what was measured. Then grasppanda probe <file> tries the config on real frames and grasppanda train <file> trains it (Swap a component). Each method page lists the same: GraspNet-baseline · GSNet · Scale-Balanced-Grasp · EconomicGrasp · FineGrasp.

Options

The encoders are split by family over three pages; this table says which page lists which.

A probe shows that a combination trains on real frames of GraspNet-1Billion, not how good it is. Accuracy is claimed only for a method trained and evaluated here.

Page Options Encoders
Scene encoders: point-based networks 23 assanet, dela, dgcnn, dgcnn-static, pointmetabase, pointmetabase-xl, pointmlp, pointnet2, pointnet2-dense, pointnet2-msg, pointnet2-msg-dense, pointnet2-transformer, pointnext, pointnext-b, pointnext-l, pointnext-xl, pointvector, pointvector-xl, pyg-edgeconv, pyg-point-transformer, pyg-pointgnn, pyg-pointnet, randlanet
Scene encoders: sparse-voxel networks 8 minkunet18a, minkunet18a-openscene, minkunet34c, minkunet34c-pointcontrast, oacnns, sparse-unet14, spunet, spunet-gaussiancross
Scene encoders: transformers and sequence models 16 litept, litept-structured3d, octformer, point-mamba, point-transformer-v1, point-transformer-v2, point-vit, ptv3, ptv3-encoder, ptv3-ppt, ptv3-scannet200, ptv3-sonata, ptv3-sonata-small, ptv3-utonia, uni3d, volt

The adapters and point painting: options with no probe of their own.

Option What it is Runs
feature-propagation FeaturePropagation: the adapter that turns a sub-sampling encoder into a dense one. no record
furthest-point-subsample FurthestPointSubsample: the adapter that turns a dense encoder into a sub-sampling one. no record
image-painted Point painting: image features painted onto the input points of a sparse U-Net (ImagePainted around SparseUNet14, the encoder of GSNet, EconomicGrasp and FineGrasp). no record
image-painted-subsampled ImagePaintedSubsampled: the PointNet++ of GraspNet-baseline with painted input points. no record
What the columns say, and who is left out
  • Runs: out of the methods the option is a swap under, those under which its probe trained on real frames without error. A method that ships the option is named own and not counted; a method still undecided is counted and named. Measured on 20 training iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, one run per cell; nothing was measured on GraspClutter6D or on Kinect frames.
  • How we measure defines every word of these columns.

Who is left out, and why:

  • feature-propagation, runs: an adapter: compose inserts it where the next stage needs it, so it is measured inside the cells it was inserted in; inserted in 65 cells (43 runs, 18 runs-empty, 4 refused).
  • furthest-point-subsample, runs: an adapter: compose inserts it where the next stage needs it, so it is measured inside the cells it was inserted in; inserted in 90 cells (54 runs, 36 runs-empty).
  • image-painted, runs: point painting: a wrapper written into a config by hand (compose does not write it); no probe record.
  • image-painted-subsampled, runs: point painting: a wrapper written into a config by hand (compose does not write it); no probe record.
Sources and licences

Code base: the upstream repository, commit and licence that THIRD_PARTY_NOTICES.md lists for the class's file (code adapted from it, or a module tree that follows it so that its weights load); written here: no third-party code. Extra: the optional install the class needs (every extra). Output: where the features live and how many channels the preset returns; compose sets a free out_channels to the width the method's other stages need. Released weights carry licences of their own.

Class Options Work Code base Extra Output
FeaturePropagation feature-propagation PointNet++ feature propagation (Qi et al., NeurIPS 2017) written here — dense · 256
FurthestPointSubsample furthest-point-subsample PointNet++ furthest point sampling (Qi et al., NeurIPS 2017) written here — subsampled, 1024 points · the width of the encoder it wraps
ImagePainted image-painted PointPainting (Vora et al., CVPR 2020), PointAugmenting (Wang et al., CVPR 2021) written here — dense · 512
ImagePaintedSubsampled image-painted-subsampled PointPainting (Vora et al., CVPR 2020), PointAugmenting (Wang et al., CVPR 2021) written here — subsampled, 1024 points · 256

Compare under every method

Choosing

  • Change as little as possible: an encoder of the kind the method has needs no adapter. Otherwise compose inserts FeaturePropagation or FurthestPointSubsample and says so.
  • The width: compose sets out_channels to the width of the encoder it replaces where the encoder has that option, so the other stages keep theirs.
  • Room-scale settings: most of these networks were published for rooms or streets. The presets restate radii and grids for a table top by analogy; they are starting points, not tuned values.
  • Released weights are an initialisation trained on other data, often under their own licence; each family page lists the files that load and what does not.
  • Optional libraries: a config that names a missing library passes the config check and stops when the model is built, with the install line.

Details

Input and output 
class Backbone(nn.Module):                # grasppanda/models/base.py
    output: "dense" | "subsampled"        # where the returned features live
    in_channels: int
    out_channels: int
    def forward(self, points: PointSet, feats: Tensor) -> PointFeatures: ...
points coords (N, 3) in metres in the camera frame, all clouds of the batch concatenated with batch offsets; a table-top view has 15 000 to 20 000 points about 5 mm apart.
feats (N, in_channels): constant ones, coordinates and normals, or nothing, as the method feeds; compose sets in_channels to match.
output dense one feature per input point, bound to the same point set.
output subsampled features on a sub-sample, the same number for every cloud of the batch; those points become the seeds.
out_channels the width of the features, injected into the proposal, the view stage and the grouper.

Three contracts tie an encoder to its neighbours; grasppanda compose resolves each or refuses with the fix:

  1. Where the features live. A proposal requires dense or subsampled features; for the other kind compose wraps the encoder in an adapter and says so.
  2. The point features. Compose sets the encoder's in_channels to what the method feeds.
  3. The width. An encoder declares the width it returns, a grouper the width it needs, and the config check compares them without building anything. Under Scale-Balanced-Grasp a fixed width gets the projected grouper (Grouper, "The width of the seed features").

No contract checks the unit of length: the header of every preset says what its radii and grids were derived from.

Probing under Scale-Balanced-Grasp

A probe of Scale-Balanced-Grasp needs the reference's width histogram, which cannot be built from the eight frames of a probe; without it the probe ends as wired (a synthetic scene). Import the file once (configs/scale_balanced_grasp/README.md), then:

grasppanda probe <config> --opts data.train.dataset.scale_statistic.source=legacy_import \
    data.train.dataset.scale_statistic.reference_sha256=<SHA-256 of the reference's file>

Scene encoders checked against the code their authors published (PTv3 and its variants, LitePT, PointNet2) are on Encoders checked against their authors' code.