Image encoders¶
--set image_encoder=38 optionsAn image encoder turns the colour image, or colour and depth, into feature maps. HGGD predicts its grasp heat-map from them; the seed-based methods can read them only through point painting, below.
Use it¶
grasppanda compose hggd/hggd_r34w8_g1b-realsense_15e \
--set image_encoder=resnet34 -o hggd_resnet34.yaml
grasppanda compose hggd/hggd_r34w8_g1b-realsense_15e --list image_encoder
HGGD is the one method with this slot. The first command writes a checked config; the second lists every option of the slot under that method, with what was measured. Then grasppanda probe <file> tries the config on real frames and grasppanda train <file> trains it (Swap a component). Each method page lists the same: HGGD.
One option of a preset changes on the same line, for example
--opts model.image_encoder.depth=depth_stem model.image_encoder.frozen=true.
Options¶
A probe shows that a combination trains on real frames of GraspNet-1Billion, not how good it is. Accuracy is claimed only for a method trained and evaluated here.
What the columns say, and who is left out
- Runs: out of the methods the option is a swap under, those under which its probe trained on real frames without error. A method that ships the option is named own and not counted; a method still undecided is counted and named. Measured on 20 training iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, one run per cell; nothing was measured on GraspClutter6D or on Kinect frames.
- Trains like shipped: out of the methods under which the swap ran, those under which every comparable loss term ended within the method's tolerance of the component it ships, with nothing rising; a method left out is named with why. Measured on 250 iterations on 8 RealSense frames of GraspNet-1Billion (one scene), seed 0, at a constant learning rate (FineGrasp: its own warm-up), beside the method as shipped trained the same way twice.
- How we measure defines every word of these columns.
Choosing¶
- Depth or not.
depth: rgb_onlyuses pretrained features as released;fourth_channelsuits convolutional pyramids you fine-tune;depth_stemsuits a frozen encoder. - Freeze foundation models (DINOv2 and v3, SigLIP 2, RADIO) on small data:
frozen: true. - Any stride fits. A pyramid decoder takes maps of any strides and widths; only
hggd-residualkeeps HGGD's own decoder, so its checkpoints load. - Weights download on first use and print their licence. A preset never switches
non-commercial weights on, and an encoder that runs code from its hub repository needs
accept_remote_code: true. - The seed-based methods read an image through point painting, which you write into a config by hand.
Point painting¶
no record image-painted, image-painted-subsampled · point painting: a wrapper written into a config by hand (compose does not write it); no probe record.
ImagePainted (around a dense scene encoder) and ImagePaintedSubsampled (around a
sub-sampling one) project every point into the colour image and give the 3-D network the image
features at its pixel, as extra input channels (fusion: input) or as a residual on its output
(fusion: output). A painted config also changes the dataset node of every split, so compose
does not write it: start from the presets image-painted and image-painted-subsampled and the
fragment below.
The complete fragment, for GraspNet-baseline and GSNet
A file next to the shipped configs, for example configs/experimental/my_painted.yaml.
GraspNet-baseline has a sub-sampling encoder, so it takes ImagePaintedSubsampled; GSNet
has a dense one and takes ImagePainted.
_base_:
- ../graspnet_baseline/graspnet-baseline_pointnet2_g1b-realsense_18e.yaml
model:
backbone:
_delete_: true # replace the base config's encoder node
type: ImagePaintedSubsampled # around a sub-sampling encoder
fusion: input # input | output
paint_channels: 32
image_scale: 0.5 # encode 640 x 360 instead of the 1280 x 720 sensor image
image_encoder: # any image-encoder node with depth: rgb_only
type: TimmImageEncoder
name: resnet18.a1_in1k
pretrained: true
out_indices: [1, 2] # strides 4 and 8
depth: rgb_only
backbone:
type: PointNet2 # the encoder of GraspNet-baseline
in_channels: 32 # 0 point features + 32 painted channels
npoints: [2048, 1024, 512, 256]
radii: [0.04, 0.1, 0.2, 0.3]
nsamples: [64, 32, 16, 16]
fps_skip_near_origin: true
fps_tie_break: pointnet2_cuda
data:
train:
dataset:
images: [rgb]
transforms: # the base list without RandomFlip and RandomRotation
- {type: RandomSample, num_points: 20000}
- {type: FilterObjects, min_visible_points: 50, exclude_object_ids: [18]}
- {type: SampleGraspLabels, point_fraction: 0.25, min_points: 300, collision: zero_scores, visible_within: 0.01, grids: deferred}
val: {dataset: {images: [rgb]}}
test: {dataset: {images: [rgb]}}
_base_:
- ../gsnet/gsnet_sunet14_g1b-realsense_10e.yaml
model:
backbone:
_delete_: true # replace the base config's encoder node
type: ImagePainted # around a dense encoder
fusion: input # input | output
paint_channels: 32
image_scale: 0.5 # encode 640 x 360 instead of the 1280 x 720 sensor image
image_encoder: # any image-encoder node with depth: rgb_only
type: TimmImageEncoder
name: resnet18.a1_in1k
pretrained: true
out_indices: [1, 2] # strides 4 and 8
depth: rgb_only
backbone:
type: SparseUNet14 # the encoder of GSNet
in_channels: 35 # 3 constant ones + 32 painted channels
out_channels: 512
voxel_size: 0.005
data:
train:
dataset:
images: [rgb]
transforms: # the base list without RandomFlip and RandomRotation
- {type: RandomSample, num_points: 15000}
- {type: FilterObjects, min_visible_points: 50}
- {type: SampleGraspLabels, point_fraction: 0.25, min_points: 300, collision: zero_scores, grids: deferred}
- {type: ConstantFeatures, channels: 3}
val: {dataset: {images: [rgb]}}
test: {dataset: {images: [rgb]}}
Under every seed-based detector type, the wrapper of the kind its proposal needs builds and
asks for the colour image (tests/unit/test_all_methods_skeleton.py). For another
seed-based method, inherit its config and wrap its own encoder:
ImagePaintedSubsampled around the sub-sampling encoder of Scale-Balanced-Grasp,
ImagePainted around the dense ones of EconomicGrasp and FineGrasp. With fusion: input
the wrapped encoder's in_channels is what the detector feeds plus paint_channels; with
fusion: output it stays as it was. A painted FineGrasp config needs
images: [depth, rgb], because FineGrasp already reads the depth image for its normals.
What the data pipeline delivers, and what it does not
images: [rgb]puts the colour image intoobservation.rgb; the camera matrix is always there and points are in the camera frame, so the pixel of a point is its projection. A scene encoder that declaresuses_observation = Trueis called with the observation.- No painting under point augmentation. A dataset that serves images refuses
RandomFlipandRandomRotation, because a moved cloud no longer projects onto its image. Carrying the per-point pixel index into the batch would lift this; it changes the data contract of every method and is not done. - No probe record. The grid composes its cells, and compose does not write painted configs, so no verdict is shown here. One painted config per seed-based method was probed by hand while the wrappers were built; those runs are not records and cannot be reproduced from the repository.
Details¶
Input and output
| input | image: (B, in_channels, H, W), float, upright |
input_layout |
rgb: colour in [0, 1], normalised by the encoder for its weights; rgbd: the same followed by depth in metres; drgb: HGGD's own order, the reference encoder only |
| output | a list of (B, out_channels[i], h_i, w_i) maps, finest first |
out_strides, out_channels |
stride and width of every map; the bridges measure them with one forward pass when built |
HGGD hands over a transposed [depth, R, G, B] image. For any encoder but its own, HGGD turns
it upright, re-orders the channels to the encoder's layout and turns the maps back; a pyramid
decoder projects every map to model.encoder.decoder_channels, merges them and resamples to
the stride of HGGD's proposals. The reference encoder keeps the reference decoder, so the
model is the one of the shipped config and converted checkpoints load.
image_encoder is a typed slot: a node is checked against the constructor of its class
before anything is built.
RGB-D: what each depth mode does
No library ships weights pretrained on RGB-D, and timm's in_chans=4 only repeats the colour
filters. So depth is an explicit option of every bridge:
depth |
What the encoder sees | Pretrained weights |
|---|---|---|
rgb_only (default of the presets) |
colour, 3 channels | used as released; depth still reaches HGGD's region stage as points |
fourth_channel |
colour + depth; the first convolution (or patch embedding) gets a fourth input | colour filters as released; the depth filters start at zero (or at the mean colour filter) and always train |
depth_stem |
colour + a residual computed from depth by a small stem of its own | untouched; with frozen: true only the stem and the decoder learn |
The settings that matter
| Setting | How to choose |
|---|---|
depth |
rgb_only to use pretrained features as they are; fourth_channel for convolutional pyramids you fine-tune; depth_stem when the encoder is frozen |
frozen |
true for foundation ViTs and small data; batch-norm statistics stay fixed too |
out_indices (timm, transformers) |
which levels come out; for ViTs [3, 6, 9, 12] gives the decoder four depths to merge |
input_size |
for models of one fixed size (MaxViT, Hiera, EVA-02, timm's DINOv2 and SigLIP): the image is resampled to it |
size_multiple |
for models that take any multiple of their patch or window: the image is resampled up to it |
model_args |
free-form arguments of the library (drop_path_rate, window_size, trust_remote_code) |
model.encoder.decoder_channels |
64 by default; wider helps a wide encoder and costs memory in the region stage |
model.encoder.detach_features |
true (the reference recipe): the region loss does not train the image network |
The bridges expose far more than the presets: TimmImageEncoder(name=<any timm model with
features_only>), HFImageEncoder(name=<hub id or local directory>),
TorchvisionImageEncoder(name=<resnet*, regnet*, or a model with .features>).
Weights and their licences
timm and transformers download a preset's weights on first use (HF_HOME chooses the
cache) and print one line with the name and the licence the library states; a local file is
checkpoint:. Non-commercial weights are supported, never switched on by a preset: such a
preset sets pretrained: false.
| Option | Weights and their licence |
|---|---|
hggd-residual |
none released for the encoder alone; the authors' HGGD checkpoints convert with tools/convert_checkpoint.py and load into this model. |
resnet18 |
resnet18.a1_in1k, Apache-2.0 (timm). |
resnet34 |
resnet34.a1_in1k, Apache-2.0 (timm). |
resnet34-rgbd |
resnet34.a1_in1k, Apache-2.0 (timm). |
resnet50 |
resnet50.a1_in1k, Apache-2.0 (timm). |
convnextv2-tiny |
convnextv2_tiny.fcmae_ft_in22k_in1k, CC BY-NC 4.0: NON-COMMERCIAL weights, off in the preset (pretrained: false; --opts model.image_encoder.pretrained=true fetches them). |
convnextv2-atto |
convnextv2_atto.fcmae_ft_in1k, CC BY-NC 4.0: NON-COMMERCIAL weights, off in the preset (pretrained: false; --opts model.image_encoder.pretrained=true fetches them). |
efficientnetv2-s |
tf_efficientnetv2_s.in21k_ft_in1k, Apache-2.0 (timm). |
mobilenetv4-conv-small |
mobilenetv4_conv_small.e2400_r224_in1k, Apache-2.0 (timm). |
mobilenetv4-hybrid-medium |
mobilenetv4_hybrid_medium.e500_r224_in1k, Apache-2.0 (timm). |
repvit-m1 |
repvit_m1.dist_in1k, Apache-2.0 (timm). |
fastvit-sa12 |
fastvit_sa12.apple_in1k, Apple's FastViT licence (timm: fastvit-license; read it before any use beyond research). |
fastvit-t8 |
fastvit_t8.apple_in1k, Apple's FastViT licence (timm: fastvit-license). |
efficientvit-b1 |
efficientvit_b1.r224_in1k, Apache-2.0 (timm). |
swinv2-tiny |
swinv2_tiny_window8_256.ms_in1k, MIT (timm). |
maxvit-tiny |
maxvit_tiny_tf_224.in1k, Apache-2.0 (timm). |
hiera-small-abswin |
hiera_small_abswin_256.sbb2_e200_in12k_ft_in1k, Apache-2.0 (timm). |
sam2-hiera-tiny |
sam2_hiera_tiny.fb_r896_2pt1, Apache-2.0 (timm). |
dinov2-small-reg |
vit_small_patch14_reg4_dinov2.lvd142m, Apache-2.0 (timm). |
dinov2-small-reg-depth-stem |
vit_small_patch14_reg4_dinov2.lvd142m, Apache-2.0 (timm). |
dinov3-vits16 |
vit_small_patch16_dinov3.lvd1689m, DINOv3 License (timm: dinov3-license; Meta's own hub repositories are gated, timm's mirror is not). |
dinov3-convnext-tiny |
convnext_tiny.dinov3_lvd1689m, DINOv3 License (timm: dinov3-license). |
siglip2-base |
vit_base_patch16_siglip_256.v2_webli, Apache-2.0 (timm). |
eva02-tiny |
eva02_tiny_patch14_224.mim_in22k, MIT (timm). |
pe-spatial-tiny |
vit_pe_spatial_tiny_patch16_512.fb, Apache-2.0 (timm). |
hf-dinov2-small |
facebook/dinov2-small, Apache-2.0 (hub card). |
hf-dinov2-small-reg |
facebook/dinov2-with-registers-small, Apache-2.0 (hub card). |
hf-hiera-tiny |
facebook/hiera-tiny-224-hf, CC BY-NC 4.0: NON-COMMERCIAL weights, off in the preset (pretrained: false; --opts model.image_encoder.pretrained=true fetches them). |
hf-depth-anything-v2-small |
depth-anything/Depth-Anything-V2-Small-hf, Apache-2.0 (hub card). The Base and Large checkpoints are CC BY-NC 4.0. |
hf-siglip2-base |
google/siglip2-base-patch16-256, Apache-2.0 (hub card). |
hf-convnextv2-tiny |
facebook/convnextv2-tiny-22k-224, Apache-2.0 according to its hub card (timm lists the same family as CC BY-NC 4.0; check before commercial use). |
hf-swinv2-tiny |
microsoft/swinv2-tiny-patch4-window8-256, Apache-2.0 (hub card). |
hf-radio-v3-base |
nvidia/C-RADIOv3-B (runs hub code: refused until --opts model.image_encoder.accept_remote_code=true), NVIDIA Open Model License (hub card: nvidia-open-model-license; read it before any use beyond research). |
tv-resnet34 |
ResNet34_Weights.IMAGENET1K_V1 from download.pytorch.org (torchvision, BSD-3-Clause code; the weights carry no separate licence statement). |
tv-convnext-tiny |
ConvNeXt_Tiny_Weights.IMAGENET1K_V1 from download.pytorch.org (torchvision). |
tv-efficientnetv2-s |
EfficientNet_V2_S_Weights.IMAGENET1K_V1 from download.pytorch.org (torchvision). |
tv-mobilenetv3-large |
MobileNet_V3_Large_Weights.IMAGENET1K_V2 from download.pytorch.org (torchvision). |
tv-regnet-y-800mf |
RegNet_Y_800MF_Weights.IMAGENET1K_V2 from download.pytorch.org (torchvision). |
Pitfalls met while building this
- HGGD's image is transposed. A pretrained network fed it sees a mirrored, rotated picture, so HGGD turns the image upright for every encoder but its own.
- Fixed-size transformers (timm's DINOv2, EVA-02, SigLIP, MaxViT, Hiera) assert the image
size they were built for;
input_sizeandsize_multipleresample, and the presets carry values that ran. - Swin V2 in timm needs every stage to be a multiple of its window; the transformers version pads by itself.
- Hiera's MAE checkpoints have positions for 224 × 224 only;
hf-hiera-tinysqueezes the image to that square,hiera-small-abswinandsam2-hiera-tinytake it at its size. - Channels-last maps of timm's Swin cannot be told apart by shape; the bridge reads timm's declared output format.
- Depth Anything V2 is a depth-estimation model on the hub:
api: depth_backbonetakes its DINOv2 encoder out. - Meta's own DINOv3 repositories are gated; timm's mirrors are not and are what the presets use.
- A nested encoder node gets its defaults filled in by the config loader, so a wrapper
cannot inject
in_channelsinto the encoder it wraps; the painted wrappers read it from the wrapped node instead.
Not wrapped
| Candidate | Why not |
|---|---|
| MambaVision | its hub code needs mamba_ssm, a compiled CUDA extension; its weights are non-commercial. |
| Meta's gated DINOv3 repositories | HFImageEncoder loads them once you have access; the presets use timm's mirrors. |
| timm's MAE Hiera | cannot be built for another image size; use hiera-small-abswin. |
| RGB-D fusion encoders (CMX, CMNeXt, TokenFusion, DFormer) | two streams fused at every stage: not one library model behind forward(image); the contract input_layout: rgbd is ready for one. |
| torchvision's transformers | TorchvisionImageEncoder cuts chains of stages only; timm has the same architectures. |
| Image features at every level of the 3-D decoder (as DITR does) | needs the wrapped encoder to expose its levels; input and output fusion are offered. |