The End Of Knowledge - Vault 5/28 - CVPR - 2017 - PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

graph LR classDef main fill:#f9d4d4, stroke-width:4px, font-size:16px, font-weight:bold classDef pointnet fill:#d4f9d4, stroke-width:3px, font-size:14px classDef pointcloud fill:#d4d4f9, stroke-width:3px, font-size:14px classDef challenges fill:#f9f9d4, stroke-width:3px, font-size:14px classDef architecture fill:#f9d4f9, stroke-width:3px, font-size:14px classDef results fill:#d4f9f9, stroke-width:3px, font-size:14px A[PointNet: Deep Learning
on Point Sets
for 3D Classification
and Segmentation] --> B[PointNet: 3D point cloud
classification and segmentation. 1] A --> C[Point cloud: raw 3D,
convertible format. 2] C --> D[Existing features: handcrafted,
task-specific. PointNet learns. 3] A --> E[Challenges: unordered set input,
permutation invariance. 4] E --> F[Symmetric functions: same value
for any ordering. 5] A --> G[PointNet vanilla: identical point transforms,
symmetric aggregation, post-transform. 6] G --> H[Input alignment: learned transformation
to canonical space. 7] G --> I[Embedding space alignment: feature
transformer network. 8] G --> J[Regularization: constrain transformation matrix
close to orthogonal. 9] B --> K[Classification: input & feature transforms,
embeddings, pooling, scores. 10] B --> L[Segmentation: local embeddings +
global for point-wise classification. 11] B --> M[ModelNet40: PointNet matches or
beats 3D CNNs. 12] B --> N[ShapeNet: surpasses prior art,
partial & complete inputs. 13] B --> O[Semantic: clearly segments scenes
into walls, chairs, tables. 14] B --> P[Robustness: handles missing points,
outliers, perturbations. 15-16] B --> Q[Critical points: capture object
contours, skeletons, key structures. 17,19] Q --> R[Upper bound shape: explains
PointNets corruption robustness. 18] B --> S[Unified approach: same architecture
for multiple tasks. 20] B --> T[Validated: theory & experiments
on representation, approximation, robustness. 21] A --> U[Pioneering work: released code
& data, new direction. 22] A --> V[3D importance: perception, interaction
needs drive 3D learning. 23-24] C --> W[Existing learning: converts to
other formats. PointNet learns directly. 25] B --> X[Training: 1024 sampled points/shape,
handles variable sizes. 26] B --> Y[Network: MLPs, ReLU, batch norm,
pooling, orthogonal constraint. 27] B --> Z[Joint training: end-to-end
input & feature transforms. 28] B --> AA[Meaningful latent space: critical regions,
invariant to corruption. 29] B --> AB[Performance & efficiency: simple operations,
avoids convolutions, effective, robust. 30] class A main class B,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,X,Y,Z,AA,AB pointnet class C,D,W pointcloud class E,F challenges class U,V architecture

Resume:

1.- PointNet: Deep learning architecture for 3D point cloud classification and segmentation.

2.- Point cloud: 3D representation closest to raw sensor data, easily convertible to/from other 3D formats.

3.- Existing point cloud features: Handcrafted for specific tasks. PointNet enables end-to-end learning on point clouds.

4.- Challenges: Designing neural networks for unordered set input, invariant to n! permutations.

5.- Symmetric functions: Function value same for any argument ordering. Can construct symmetric neural networks.

6.- PointNet vanilla: Transform points identically, aggregate by symmetric function, post-transform. Approximates any continuous symmetric function.

7.- Input alignment: Align to canonical space via learned transformation matrix. Similar to spatial transformer networks.

8.- Embedding space alignment: Align intermediate point embeddings using a feature transformer network.

9.- Regularization: Constrain transformation matrix close to orthogonal to avoid bad local minima.

10.- Classification architecture: Input & feature transformers, point embeddings, max pooling, category scores.

11.- Segmentation extension: Concatenate local point embeddings with global feature for point-wise classification.

12.- ModelNet40 results: PointNet achieves better or on par classification vs 3D CNNs.

13.- ShapeNet part segmentation: Surpasses previous state-of-the-art on partial and complete inputs.

14.- Semantic segmentation: Clearly segments 3D scenes into walls, chairs, tables, etc.

15.- Robustness to missing points: Only 2% accuracy drop with 50% points removed. More robust than 3D CNN.

16.- Robustness to outliers and perturbations: PointNet handles corrupted data better than 3D CNN.

17.- Critical point set: Subset of input points that contribute to global feature. Captures object contours/skeletons.

18.- Upper bound shape: Region in space where points yield same global feature. Explains PointNet's corruption robustness.

19.- Feature learning generalization: Critical points capture key structures for unseen object categories.

20.- Unified 3D recognition approach: Same architecture for classification, part & semantic segmentation.

21.- Theoretical and experimental validation: Symmetric function representation, approximation capacity, robustness properties demonstrated.

22.- Beginning of point cloud deep learning: An exciting new direction with released code & data.

23.- 3D data importance: Emerging applications require 3D perception and interaction. Drives need for 3D deep learning.

24.- Converting 3D formats: Point clouds in canonical form, easily convertible to/from meshes, voxels, etc.

25.- Existing deep learning on point clouds: Most methods convert to other formats. PointNet learns directly on points.

26.- Training data size: 1024 points sampled per shape. Handles variable input sizes.

27.- Network details: Multilayer perceptrons, ReLU, batch norm, max pooling. Transformation matrix close to orthogonal.

28.- Joint end-to-end training: Input & feature transformers trained together with rest of network.

29.- Meaningful latent space: Global feature represents critical regions invariant to data corruption.

30.- Performance & efficiency: Simple point-wise operations avoid costly convolutions. Highly effective and robust.

Knowledge Vault built byDavid Vivancos 2024