The End Of Knowledge - Vault 5/16 - CVPR - 2016 - Structural-RNN: Deep Learning on Spatio-Temporal Graphs

graph LR classDef main fill:#f9f9f9, stroke:#333, stroke-width:1px, font-weight:bold, font-size:14px classDef spatial_temporal fill:#d4f9d4, stroke:#333, stroke-width:1px, font-weight:bold, font-size:14px classDef structural_rnn fill:#d4d4f9, stroke:#333, stroke-width:1px, font-weight:bold, font-size:14px classDef factor_graphs fill:#f9d4d4, stroke:#333, stroke-width:1px, font-weight:bold, font-size:14px classDef applications fill:#f9f9d4, stroke:#333, stroke-width:1px, font-weight:bold, font-size:14px A[Structural-RNN: Deep Learning
on Spatio-Temporal Graphs] --> B[CNNs, RNNs successful in
spatial, temporal understanding 1] A --> C[Objects have correlated states,
interactions across space, time 2] A --> D[Prior knowledge improves
spatio-temporal reasoning 3] A --> E[Structural-RNN injects spatio-temporal
structures into neural networks 4] E --> F[Most previous approaches
problem-specific or limited 5] E --> G[Transforms user-defined graph
into recurrent neural networks 6] G --> H[Benefits: structure, deep learning,
inference, training, flexibility 7] E --> I[Graph nodes represent components,
edges represent interactions 8] I --> J[Factor graphs used as
intermediate representation 9] J --> K[Semantic node groupings
allow sharing functions 10] J --> L[Factor nodes parameterized
by RNNs 11] L --> M[Node RNNs combine context
to predict labels 12] L --> N[Edge RNNs model evolving
interactions over time 12] E --> O[RNNs wired into bipartite
graph structure 13] E --> P[Generic approach, applies to
any spatio-temporal graph 14] E --> Q[Demonstrated on diverse problems:
human activity, motion, driving 15] Q --> R[Human motion graph of
interacting body parts 16] R --> S[Trained on motion capture
to predict next frame 17] R --> T[Generates realistic motion vs.
ERD, LSTM baselines 18] R --> U[Learned RNN cells encode
semantic motion concepts 19] U --> V[Cells correspond to left, right
leg motion 20] Q --> W[Allows manipulating structure of
learned neural networks 21] W --> X[Transferred leg RNNs generate
novel motion combinations 22] Q --> Y[Not possible with unstructured
giant neural network 23] Q --> Z[Impressive results on activity
recognition, driving anticipation 24] E --> AA[Principled way to transform
graphs into structured RNNs 25] E --> AB[Factor graphs as intermediate
representation 26] E --> AC[Scalable due to ability
to share factors 27] E --> AD[Can learn features from scratch
or use hand-designed 28] E --> AE[Source code made
publicly available 29] E --> AF[Allows injecting priors, demonstrates
benefits on diverse problems 30] class A main class B,C,D,I,P,Q,R,S spatial_temporal class E,F,G,H,O,AA,AB,AC,AD,AE,AF structural_rnn class J,K,L,M,N factor_graphs class T,U,V,W,X,Y,Z applications

Resume:

1.- CNNs and RNNs have been successfully applied to spatial and temporal understanding, but don't capture rich spatio-temporal structures in the real world.

2.- Objects in a scene have correlated states and interactions that propagate across space and time, which humans exploit but algorithms often don't.

3.- Prior knowledge about spatio-temporal interactions can be incorporated into the design of learning algorithms to improve reasoning about what will happen next.

4.- Structural-RNN provides a principled way to inject high-level spatio-temporal structures into neural networks, combining the benefits of structured models and deep learning.

5.- Most previous structured deep learning approaches are problem-specific or don't address applications with both rich spatial and temporal interactions.

6.- Structural-RNN transforms a user-defined spatio-temporal interaction graph capturing algorithmic priors into a rich structure of recurrent neural networks.

7.- Benefits include combining structure with deep learning, simple feed-forward inference, end-to-end training, and flexibility to modify the spatio-temporal graph.

8.- The spatio-temporal graph's nodes represent problem components, edges represent interactions. Features are carried on nodes and edges at each time step.

9.- Factor graphs are used as an intermediate representation. Node factors are defined for each node, edge factors for spatial and temporal edges.

10.- Semantic groupings of nodes allow sharing factor functions, improving scalability. Factor nodes of the same type share functions as the graph unrolls.

11.- Factor nodes are parameterized by RNNs - node factors become node RNNs, edge factors become spatial and temporal edge RNNs.

12.- Node RNNs combine contextual information to predict labels. Spatial and temporal edge RNNs model evolving interactions over time.

13.- RNNs are wired into a bipartite graph structure, with edge RNNs modeling individual interactions and node RNNs combining them to make predictions.

14.- The approach is generic and can be applied to any spatio-temporal graph. Training uses edge features as RNN inputs to predict labels.

15.- Structural-RNN is demonstrated on diverse spatio-temporal problems with different data modalities - human activity, human motion, and driving maneuver anticipation.

16.- Human motion has a graph structure of interacting body parts generating complex motions. Joint angles are node features.

17.- A motion capture dataset was used to train the model to predict the next frame given the current one.

18.- Structural-RNN generates more natural and realistic looking predicted motion compared to baselines like ERD and LSTM.

19.- Analysis revealed semantic concepts encoded in the learned RNN memory cells, such as right arm cells firing when moving the hand near the face.

20.- Other semantic cells were found corresponding to left and right leg motion, activating when the respective leg moved forward.

21.- The high-level priors in the spatio-temporal graph allow manipulating the structure of the learned neural networks in interesting ways.

22.- Leg RNNs from a slow motion model were transferred into a fast motion model, generating novel combinations of motion patterns.

23.- Such high-level manipulations are not possible with a single unstructured giant neural network.

24.- Impressive results were also obtained on the other applications of human activity recognition and driving maneuver anticipation.

25.- The Structural-RNN approach provides a generic, principled way to transform spatio-temporal graphs into structured recurrent neural networks.

26.- Factor graphs serve as an intermediate representation in the transformation from the interaction graph to the RNN structure.

27.- The approach is scalable due to the ability to share factors, reducing the number of learnable parameters.

28.- Models can be trained end-to-end to learn features from scratch, or can incorporate hand-designed input features.

29.- Source code for the Structural-RNN approach has been made publicly available online.

30.- Structural-RNN allows injecting high-level spatio-temporal structures and priors into deep networks and demonstrates benefits on several diverse problems.

Knowledge Vault built byDavid Vivancos 2024