Discover a novel computer vision-based attack that automatically discloses touch screen inputs, including passcodes, from a distance of three meters using spying cameras like Google Glass. Learn how the attack tracks fingertip movements, analyzes shadow formations, and applies advanced computer vision techniques to recognize over 90% of tapped passcodes, even when the screen is not visible to the naked eye. Explore the methodology behind this stealthy attack, including optical flow, deformable part-based model object detection, k-means clustering, and planar homography. Understand the potential impact of this security threat in public spaces and conferences. Examine the proposed countermeasure: a context-aware Privacy Enhancing Keyboard (PEK) for Android systems that randomizes the keyboard layout for sensitive information input.
Overview
Syllabus
Intro
Introduction Blind recognition of touched keys Evaluation Countermeasures Conclusion
Use computer vision techniques to recognize possible touched keys and use a language model to correct the prediction. Poor success rate for passwords.
Taking Videos Use sneaky cameras including Google Glass, web cameras, smartphone cameras, even smartwatch! - Factors: camera angle, distance, lighting Adjust the camera angle at a distance to record the device and touching fingertip movement.
Preprocessing Keep the area of moving hand on screen. - Use Deformable Part-based Model (DPM) - an object tracker - to track the area of interest for a moving target
Detecting Touching Frames Derive a pattern of the touching finger movement Finger moves downward, stops and then upward. Track feature points on the hand by optical flow. Al fingers keep the same gesture during touching Use the frame in which velocity of most tracked points changes the direction as touching frame.
Deriving Homography Matrix Derive touch screen corners, intersection of the four edges of the touch screen. Canny edge detector to detect edges Hough line transform to get the lines. Use these four pairs of corner points to derive the homography matrix.
Locating Touching Fingertip Use the DPM object detector to locate the touching fingertip in touching frames. Derive a large box bounding the touching fingertip.
Estimating Touched Area Deriving the fingertip contour Use k means clustering to cluster pixels in a small bounding box to get the fingertip contour. Two groups bright fingertip and dark screen. Deriving the accurate touched area Fit a line over central points of the contour and get
Recognizing Touched Keys Which pixels are the touched points in this estimated tiny touched area? If the touched point is found, map the estimated touched point to the reference image of the software keyboard.
Step 7. Recognizing Touched Keys (Cont'd) Apply k-means clustering to estimated touched area. -5, because of illumination and shadowing Use a point in the upper part of the darkest cluster as the touched point.
Taught by
Black Hat
