Meta Ray-Ban video lets attacker read passwords, 77.8% hit rate

The paper's starting point is that smart glasses such as Meta Ray-Ban look like ordinary eyewear, so people are unlikely to notice they are being filmed — and because the attack reads fingers from the front, it bypasses the habit of guarding against someone looking over your shoulder. Earlier video-analysis work had been limited to simple conditions such as tablets with large keys or 4-digit PINs. This study claims a broader scope: natural typing on a phone, with no prior knowledge of the device or keyboard app.

The pipeline tracks fingertips at sub-pixel accuracy using an image-recognition model (SAM 3) and a hand-tracking model (MediaPipe Hands), then uses a 3D neural network to pinpoint the moment a key is pressed, discarding backspace corrections. It works out the on-screen keyboard area from the shift and layout-switch keys that must be pressed to type uppercase letters and digits, then combines per-key press probabilities from the video with statistics from about 2.2 billion leaked passwords to narrow the candidates.

The limits the researchers report point to where the risk is lower: accuracy drops for young people who type fast with very small finger movements, for people who type hesitantly, and for those who press with both thumbs almost simultaneously.