The End Of Knowledge - Vault 5/32 - CVPR - 2018 - Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies

graph LR classDef model fill:#f9d4d4, font-weight:bold, font-size:14px classDef capture fill:#d4f9d4, font-weight:bold, font-size:14px classDef technical fill:#d4d4f9, font-weight:bold, font-size:14px classDef results fill:#f9f9d4, font-weight:bold, font-size:14px A[Total Capture: A
3D Deformation Model
for Tracking Faces,
Hands, and Bodies] --> B[3D model tracks faces,
hands, bodies 1] B --> C[Frank: stitched parts,
single skeleton hierarchy 2] B --> D[Adam: total capture,
simpler than Frank 3] A --> E[Body motion transmits
social communication 4] A --> F[Systems analyze parts,
not full spectrum 5] B --> G[Body: SMPL with
shape, pose, translation 6] B --> H[Face: PCA blend of
shape, expression 7] B --> I[Hand: mesh deformed
by bone scaling 8] A --> J[3D keypoints fit
models via triangulation 9] A --> K[ICP matches points
to mesh models 10] A --> L[Objective: keypoint, ICP,
seam, prior costs 11] L --> M[Seam constraints avoid
part discontinuities 12] L --> N[Priors prevent overfitting
noise, inconsistencies 13] D --> O[Adam dataset: 70 subjects,
short motions 14] D --> P[PCA on fitted surfaces
yields shape 15] A --> Q[Total capture: unified
parameters, no seams 16] A --> R[Overlap with silhouettes
quantifies performance 17] A --> S[Challenging sequences
demonstrate realism 18] A --> T[Markerless hands outperform
body, face 19] A --> U[Learning detectors may
beat marker-based 20] class B,C,D,G,H,I model class E,F,Q,T capture class J,K,L,M,N technical class O,P,R,S,U results

Resume:

1.- Total capture: 3D deformation model for tracking faces, hands, and bodies simultaneously.

2.- Frank model: Created by stitching together individual part models into a single skeleton hierarchy.

3.- Adam model: Derived from Frank, enables total body motion capture with a simpler parameterization.

4.- Social communication: Transmitted by organized motion of the whole body, including facial expressions, hand gestures, and body posture.

5.- Existing systems: Focus on particular scales or parts, making it difficult to concurrently analyze full spectrum of social signaling.

6.- Body model: SMPL model with minor modifications, defined by shape, pose, and translation parameters.

7.- Face model: Generative PCA model built from FaceWarehouse dataset, combining shape and expression subspaces.

8.- Hand model: Artist-rigged mesh deformed via linear blend skinning, with scaling parameters for each bone.

9.- 3D keypoint detection: Used to fit the models, obtained by triangulation of 2D keypoint detections across multiple views.

10.- Iterative Closest Point (ICP): Used to match uncorresponded 3D points from multiview stereo reconstruction to the mesh models.

11.- Objective function: Includes anatomical keypoint, ICP, seam constraint, and prior costs to fit the models to data.

12.- Seam constraints: Encourage vertices around seam parts to be close, avoiding discontinuities between part models.

13.- Prior cost: Set over model parameters to avoid overfitting to noise in 3D point clouds and inconsistencies in joint locations.

14.- Capturing Adam dataset: 70 subjects performing short range of motion, reconstructed with Frank model to build Adam.

15.- Shape deformation space: PCA analysis on fitted surfaces warped to canonical pose captures variations across entire body.

16.- Total motion capture: Optimizes a cost function similar to Frank's, but with a unified set of parameters and no seam constraints.

17.- Quantitative evaluation: Measures overlap with ground truth silhouettes, comparing SMPL, Frank, and Adam models.

18.- Qualitative results: Demonstrates the method on various challenging sequences, showing more realism than SMPL alone.

19.- Markerless hand capture: Shows better localization quality than body and face capture in the results.

20.- Potential to outperform marker-based methods: Learning-based keypoint detectors can provide measurements for occluded parts.

Knowledge Vault built byDavid Vivancos 2024