Visual Vibrometry: Estimating Material Properties From Small Motion in Video

Abe Davis, Katherine L. Bouman, Justin G. Chen, Michael Rubinstein, Frédo Durand, William T. Freeman

**Concept Graph & Resume using Claude 3 Opus | Chat GPT4o | Llama 3:**

graph LR
classDef vibration fill:#f9d4d4, font-weight:bold, font-size:14px
classDef computer_vision fill:#d4f9d4, font-weight:bold, font-size:14px
classDef analysis fill:#d4d4f9, font-weight:bold, font-size:14px
classDef experiments fill:#f9f9d4, font-weight:bold, font-size:14px
classDef applications fill:#f9d4f9, font-weight:bold, font-size:14px
A[Visual Vibrometry: Estimating

Material Properties From

Small Motion in

Video] --> B[Objects vibrate at resonant

frequencies and modes 1] A --> C[Computer vision recovers subtle

vibrations from videos 2] C --> D[Relates motion frequencies to

resonant frequencies 3] D --> E[Estimates material properties

like stiffness 4] B --> F[Small-scale deformation around

objects rest state 5] F --> G[Simple to analyze via

linear modal analysis 6] A --> H[Non-destructive testing assesses

physical properties without damage 7] H --> I[Video enables trading temporal

for spatial resolution 8] C --> J[Ubiquity of video allows

data-driven approaches 9] J --> K[Applications in material

recognition 10] A --> L[Analysis uses global power

spectrum of motion 11] L --> M[Spikes in spectrum indicate

vibration modes/shapes 12] M --> N[Mode shapes relate motion

at different points 13] B --> O[Resonant frequencies are global,

viewpoint-invariant properties 14] A --> P[Modal analysis: frequency depends

on geometry and material 15] P --> Q[Spectrum ambiguity resolved by

known geometry information 16] A --> R[Local pixel motion spectra

calculated and visualized 17] R --> S[Spectra averaged for global

power spectrum 18] A --> T[Experiment 1: Clamped rod

with known geometry 19] T --> U[Rods fundamental frequency

encodes material properties 20] U --> V[Estimated elasticity matches

ground truth measurements 21] A --> W[Experiment 2: Hanging fabric

with similar geometry 22] W --> X[Small fabric vibrations relate

to physical properties 23] X --> Y[Power spectra from imperceptible

ambient fabric motion 24] Y --> Z[Similar geometry enables

meaningful spectra comparison 25] Z --> AA[Regression predicts fabric

material properties 26] AA --> AB[Accurate predictions of weight

and stiffness trends 27] AB --> AC[Features invariant to camera

and viewpoint 28] C --> AD[Cameras promising for vibrational

analysis vs equipment 29] AD --> AE[Enables new computer

vision applications 30] class A,B,F,O vibration class C,I,J,AD,AE computer_vision class D,E,G,L,M,N,P,Q,R,S analysis class T,U,V,W,X,Y,Z,AA,AB,AC experiments class H,K applications

Material Properties From

Small Motion in

Video] --> B[Objects vibrate at resonant

frequencies and modes 1] A --> C[Computer vision recovers subtle

vibrations from videos 2] C --> D[Relates motion frequencies to

resonant frequencies 3] D --> E[Estimates material properties

like stiffness 4] B --> F[Small-scale deformation around

objects rest state 5] F --> G[Simple to analyze via

linear modal analysis 6] A --> H[Non-destructive testing assesses

physical properties without damage 7] H --> I[Video enables trading temporal

for spatial resolution 8] C --> J[Ubiquity of video allows

data-driven approaches 9] J --> K[Applications in material

recognition 10] A --> L[Analysis uses global power

spectrum of motion 11] L --> M[Spikes in spectrum indicate

vibration modes/shapes 12] M --> N[Mode shapes relate motion

at different points 13] B --> O[Resonant frequencies are global,

viewpoint-invariant properties 14] A --> P[Modal analysis: frequency depends

on geometry and material 15] P --> Q[Spectrum ambiguity resolved by

known geometry information 16] A --> R[Local pixel motion spectra

calculated and visualized 17] R --> S[Spectra averaged for global

power spectrum 18] A --> T[Experiment 1: Clamped rod

with known geometry 19] T --> U[Rods fundamental frequency

encodes material properties 20] U --> V[Estimated elasticity matches

ground truth measurements 21] A --> W[Experiment 2: Hanging fabric

with similar geometry 22] W --> X[Small fabric vibrations relate

to physical properties 23] X --> Y[Power spectra from imperceptible

ambient fabric motion 24] Y --> Z[Similar geometry enables

meaningful spectra comparison 25] Z --> AA[Regression predicts fabric

material properties 26] AA --> AB[Accurate predictions of weight

and stiffness trends 27] AB --> AC[Features invariant to camera

and viewpoint 28] C --> AD[Cameras promising for vibrational

analysis vs equipment 29] AD --> AE[Enables new computer

vision applications 30] class A,B,F,O vibration class C,I,J,AD,AE computer_vision class D,E,G,L,M,N,P,Q,R,S analysis class T,U,V,W,X,Y,Z,AA,AB,AC experiments class H,K applications

**Resume: **

**1.-** Objects vibrate at preferred resonant frequencies and modes, often imperceptibly.

**2.-** Recent computer vision work can recover subtle vibrations from regular videos.

**3.-** Relating temporal motion frequencies to an object's resonant frequencies enables estimating its vibration modes.

**4.-** Vibration modes + object geometry information allows estimating material properties like stiffness.

**5.-** Motion considered is very small-scale deformation around an object's rest state.

**6.-** Small deformation scale makes vibrations difficult to see but simple to analyze via linear modal analysis.

**7.-** Drawing from non-destructive testing field, which assesses physical properties of structures without damage.

**8.-** Typical vibrational analysis uses contact sensors or laser vibrometers. Video enables trading temporal resolution for spatial resolution.

**9.-** Ubiquity of video allows data-driven approaches impractical with specialized vibrational analysis equipment.

**10.-** Estimating material properties from video has applications in material recognition.

**11.-** Analysis uses a global power spectrum summarizing temporal frequencies of motion in the video.

**12.-** Objects vibrate at preferred resonant frequencies, visible as spikes in the motion spectrum. Each spike is a vibration mode/shape.

**13.-** Vibration mode shapes relate motion at different object points - opposite moving points have opposite phase.

**14.-** Resonant frequencies are global object properties, largely invariant to viewpoint changes.

**15.-** Modal analysis: resonant frequency pattern depends on object geometry, scaled by material properties.

**16.-** Motion spectrum gives ambiguous combination of structural and material properties, resolved by known geometry information.

**17.-** Local pixel motion spectra calculated, phase and amplitude visualized to see mode shapes.

**18.-** Spectra are powered and averaged to get global constructive power spectrum used to estimate material properties.

**19.-** Experiment 1: Estimating material properties of clamped rod with known geometry from high-speed video.

**20.-** Rod's fundamental frequency encodes material properties. Known geometry allows isolating material properties' effect.

**21.-** Rod motion extracted, power spectrum calculated, fundamental frequency spike identified using expected resonance ratios and mode shapes.

**22.-** Estimated rod elasticity closely matches ground truth measurements for various materials and lengths.

**23.-** Experiment 2: Estimating material properties of hanging fabric with unknown but similar geometry from regular video.

**24.-** Small fabric vibrations governed by simpler relationships than large wind-blown deformations, relate to physical properties.

**25.-** Power spectra calculated from imperceptible ambient fabric motion recorded with regular camera.

**26.-** Similar fabric geometry enables meaningful spectra comparison, with trends apparent across samples.

**27.-** Regression model predicts fabric material properties from power spectrum features.

**28.-** Accurate predictions of area weight and stiffness trends from both sound-excited and passive fabric motion.

**29.-** Features are largely invariant to camera and viewpoint, outperform prior work.

**30.-** Cameras offer promise for vibrational analysis vs specialized equipment, enables new computer vision applications.

Knowledge Vault built byDavid Vivancos 2024