Knowledge Vault 3/12 - GTEC BCI & Neurotechnology Spring School 2024 - Day 1
Neuromotor interfaces for human computer interaction at Meta
Alexandre Gramfort, Meta Reality Labs (USA)
<Resume Image >

Concept Graph & Resume using Claude 3 Opus | Chat GPT4 | Llama 3:

graph LR classDef meta fill:#f9d4d4, font-weight:bold, font-size:14px; classDef emg fill:#d4f9d4, font-weight:bold, font-size:14px; classDef models fill:#d4d4f9, font-weight:bold, font-size:14px; classDef results fill:#f9f9d4, font-weight:bold, font-size:14px; classDef future fill:#f9d4f9, font-weight:bold, font-size:14px; A[Alexandre Gramfort] --> B[Meta's EMG for HCI.
1] A --> C[EMG: neurons to computer. 2] C --> D[PNS: motor neurons to muscles. 3] B --> E[EMG enables deviceless input. 4] B --> F[EMG differs between users. 5] F --> G[One user's data doesn't help
others. 6] B --> H[Large user data enables generic
models. 7] B --> I[Precise electrode placement crucial. 8] A --> J[Preprint evaluated unseen users. 9] J --> K[More users, larger models reduced
errors. 10] B --> L[EMG most viable for hi-bandwidth
HCI. 11] B --> M[Hardware design crucial with ML. 12] B --> N[16-channel bipolar, 2 kHz sampling
used. 13] B --> O[New user data helps beyond generic
models. 14] O --> P[Small initial set, more gain from
personalization. 15] A --> Q[Meta to release EMG datasets. 16] A --> R[Motor units: neuron and muscle fibers. 17] R --> S[Motor unit firing gives measurable
EMG. 18] B --> T[EMG faster than physical movements. 19] T --> U[EMG could enable higher human-computer
throughput. 20] B --> V[Muscle variability makes EMG decoding
hard. 21] V --> W[Large diverse training data needed for
robustness. 22] J --> X[Paper focused on user-independent
models. 23] B --> Y[User changes easier than user
differences. 24] A --> Z[Open EMG standards would aid research. 25] B --> AA[Muscle changes hard without similar
training data. 26] B --> AB[EMG can estimate precise force. 27] B --> AC[Motion artifacts challenging, but
EMG dominates. 28] A --> AD[Impossible movements a future
research direction. 29] A --> AE[Large datasets and ML enable EMG
HCI. 30] class A,B,F,G,H,I,L,M,N,O,P,Q,V,W,Y,Z,AA,AB,AC,AD,AE meta; class C,D,R,S,T,U emg; class J,K,X results; class E future;


1.-Alexandre Gramfort from Meta Reality Labs presented work on electromyography (EMG) for human-computer interfaces, with results detailed in a bioRxiv preprint.

2.-EMG allows direct access from neurons and muscles to a computer, potentially providing higher bandwidth output compared to typical input devices.

3.-The peripheral nervous system connects motor neurons from the motor cortex to muscles in the body, which can be sensed with EMG.

4.-Examples showed EMG enabling virtual button presses, joystick-like 4D movements, fast typing on any surface, and handwriting recognition, all without holding devices.

5.-EMG signals differ substantially between users due to anatomical differences. More training data from a user improves EMG decoding for that individual.

6.-However, data from one user does not help decoding EMG signals from a new unseen user, due to individual anatomical differences.

7.-Deep learning models trained on EMG data from thousands of users (up to 5000) enable building user-independent generic EMG interfaces.

8.-Precise electrode placement over relevant muscles is important. Shoulder EMG cannot decode finger movements. Electrodes must be near the EMG signal source.

9.-The preprint evaluated EMG models on new unseen users for classification of discrete gestures, continuous wrist movements, and handwriting of letters.

10.-Results showed that training on more users (up to 5000) and using larger deep learning models substantially reduced errors on new users.

11.-EMG seems to be the most viable current approach for high-bandwidth, minimally invasive human-computer interfaces based on sensing the nervous system.

12.-Hardware design, such as optimizing electrode placement and count for strong signal-to-noise ratio, is crucial alongside machine learning for EMG interfaces.

13.-The raw EMG data used a 16-channel bipolar montage and was sampled at 2 kHz. Electrode placement variability is much smaller than user variability.

14.-Personalization by updating an EMG model with some data from a new user still helps beyond the best user-independent generic models.

15.-The gain from personalization is larger when the initial training set is small. With thousands of users, good user-independent models are feasible.

16.-Meta is working to publicly release some EMG datasets in the future to facilitate research, such as data on two-handed surface typing.

17.-Background on the nervous system and motor units was provided. Each motor neuron innervates a group of muscle fibers called a motor unit.

18.-When a motor neuron fires, all muscle fibers in its motor unit contract together, providing a strong enough signal to measure with EMG.

19.-EMG signals reflect the neural output of the motor cortex and are much faster than the resulting physical movements of muscles and joints.

20.-Accessing this faster, higher-bandwidth EMG signal and bypassing physical movement could enable higher information throughput from human to computer than typical interfaces.

21.-However, muscles exhibit substantial variability between people in terms of size, strength, anatomy, and neural innervation, making EMG decoding challenging.

22.-Having EMG training data from a large and diverse set of individuals is crucial for building robustness to this individual variability.

23.-The results in the paper focused specifically on user-independent models with no calibration or updating to each new individual user.

24.-Detecting non-stationarities over time within a user, like due to fatigue or electrode impedance changes from sweat, is easier than handling user variability.

25.-Open-source standards would facilitate data sharing and research on EMG. Developers have started discussions on standardizing an EMG file format.

26.-EMG signal changes from muscle weakness or injury would be challenging unless similar examples are included in the training data.

27.-Precise force estimation is possible with EMG, as the signal amplitude correlates with the strength of muscle recruitment and contraction.

28.-Motion artifacts affecting the electrodes are a challenge, as the core EMG signal is electrical. But most information still comes from EMG itself.

29.-Futuristic ideas like interfaces for new impossible-for-humans movements would require advances beyond the current work, but are interesting long-term research directions.

30.-In summary, high-bandwidth EMG-based human-computer interfaces are becoming increasingly feasible, driven by access to large diverse datasets and modern machine learning approaches.

Knowledge Vault built byDavid Vivancos 2024