The End Of Knowledge - Vault 6/33 - CVPR - 2018 - Language to Action: towards Interactive Task Learning with Physical Agents

graph LR classDef core fill:#f9d4d4, font-weight:bold, font-size:14px classDef language fill:#d4f9d4, font-weight:bold, font-size:14px classDef learning fill:#d4d4f9, font-weight:bold, font-size:14px classDef causality fill:#f9f9d4, font-weight:bold, font-size:14px classDef challenges fill:#f9d4f9, font-weight:bold, font-size:14px Main[Language to Action:
towards Interactive Task
Learning with Physical
Agents] Main --> A[Interactive task learning:
human-robot communication 1] A --> B[Common ground enables
effective communication 2] A --> C[Grounding language to
physical environment 3] Main --> D[Language understanding] D --> E[Verb semantics: action
frames specification 4] E --> F[Implicit and explicit
arguments in verbs 5] D --> G[Grounding language to
robotic actions 9] D --> H[Grounded verb semantics:
resulting states representation 10] D --> I[Social pragmatic theory
of language acquisition 11] Main --> J[Learning approaches] J --> K[Incremental learning through
human-robot interaction 12] K --> L[Hypothesis spaces for
verb meaning generalization 13] J --> M[Reinforcement learning for
interaction optimization 14] J --> N[Bootstrapping from web
images for learning 17] J --> O[Dynamic change representation
using video data 18] J --> P[Incremental and interactive
lifelong learning algorithms 21] Main --> Q[Causality and effects] Q --> R[Causality modeling guides
perception and grounding 6] Q --> S[Crowdsourcing descriptions of
action effects 7] S --> T[18 dimensions of
object change identified 8] Q --> U[Naive physics: basic
cause-effect understanding 15] Q --> V[Action effect prediction:
causes and potential effects 16] Q --> W[Causal reasoning for
decision-making and planning 23] W --> X[Physical vs. social
cause-effect knowledge distinction 24] Main --> Y[Challenges and future directions] Y --> Z[Multidisciplinary collaboration needed 19] Y --> AA[Rich, interpretable robot
representations for understanding 20] Y --> AB[Incorporating prior knowledge
to bootstrap learning 22] Y --> AC[Combining neural and
symbolic representations 25] Y --> AD[Social signals in
human-robot teaching scenarios 26] Y --> AE[Language-independent internal representations 27] Y --> AF[Extended dialogue with
common sense knowledge 28] Y --> AG[Dexterity challenges in
robotic manipulation tasks 29] Y --> AH[Knowledge sharing between
robots and adaptation 30] class A,B,C core class D,E,F,G,H,I language class J,K,L,M,N,O,P learning class Q,R,S,T,U,V,W,X causality class Y,Z,AA,AB,AC,AD,AE,AF,AG,AH challenges

Resume:

1.- Interactive task learning: Teaching robots new tasks through natural interaction and communication with humans.

2.- Common ground: Shared knowledge and beliefs that enable effective communication between humans and robots.

3.- Grounding language to perception: Connecting linguistic expressions to objects and events in the physical environment.

4.- Verb semantics: Capturing the meaning of action verbs using frames that specify key ingredients of the action.

5.- Implicit and explicit arguments: Verb arguments that may or may not be explicitly stated in language but are important for understanding.

6.- Causality modeling: Representing how the world changes as a result of actions to guide perception and grounding.

7.- Crowdsourcing action effects: Using human input to collect descriptions of how objects change after specific actions.

8.- Dimensions of change: Identifying 18 dimensions along which objects can change as a result of actions.

9.- Grounding language to action: Translating high-level language commands into sequences of primitive robotic actions.

10.- Grounded verb semantics: Representing verbs in terms of resulting states rather than sequences of primitive actions.

11.- Social pragmatic theory: Children acquire language as a byproduct of social interaction, using basic cognitive skills.

12.- Incremental learning approach: Robots continually acquire and refine verb semantics through interaction with humans and the environment.

13.- Hypothesis spaces: Representing possible verb meanings and generalizing from specific experiences to more abstract concepts.

14.- Reinforcement learning for interaction: Learning when to ask questions to resolve ambiguities in a way that maximizes long-term reward.

15.- Naive physics: Basic understanding of cause-effect relationships between actions and perceived states of the world.

16.- Action effect prediction: Given an action, identifying potential effects, or given an effect, identifying potential causes.

17.- Bootstrapping from web images: Using web-retrieved images to supplement annotated examples for learning action effects.

18.- Dynamic change representation: Using video data to capture the temporal aspects of action effects.

19.- Multidisciplinary collaboration: The need for experts from various fields to work together on language communication with robots.

20.- Rich and interpretable representations: Internal robot representations that can bring humans and robots to a joint understanding.

21.- Incremental and interactive algorithms: Learning methods that support lifelong learning from interactions with humans and the environment.

22.- Incorporating prior knowledge: Providing robots with strong initial knowledge to bootstrap learning.

23.- Causal reasoning: The importance of understanding cause-effect relationships for decision-making and action planning.

24.- Physical vs. social cause-effect knowledge: Distinguishing between knowledge about physical actions and knowledge guiding social interactions.

25.- Combining neural networks and symbolic representations: Using both approaches to leverage their respective strengths.

26.- Social signals in learning: Leveraging body language, eye gaze, and joint attention in human-robot teaching scenarios.

27.- Language-independent representations: Developing internal representations that can work across different human languages.

28.- Extended natural dialogue: The potential role of common sense knowledge in enabling longer, more coherent conversations.

29.- Dexterity challenges: The difficulty of achieving human-level dexterity in robotic manipulation tasks.

30.- Knowledge sharing between robots: The potential for robots to share learned models and adapt them to new situations.

Knowledge Vault built byDavid Vivancos 2024