Document Type

Article

Publication Date

9-9-2024

Journal / Book Title

Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies

Abstract

Shoulder surfing attacks (SSAs) are a type of observation attack designed to illicitly gather sensitive data from “over the shoulder” of victims. This attack can be directed at mobile devices, desktop screens, Personal Identification Number (PIN) pads at an Automated Teller Machine (ATM), or written text. Existing solutions are generally focused on authentication techniques (e.g., logins) and are limited to specific attack scenarios (e.g., mobile devices or PIN Pads). We present ShouldAR, a mobile and usable system to detect SSAs using multimodal eye gaze information (i.e., from both the potential attacker and victim). ShouldAR uses an augmented reality headset as a platform to incorporate user eye gaze tracking, rear-facing image collection and eye gaze analysis, and user notification of potential attacks. In a 24-participant study, we show that the prototype is capable of detecting 87.28% of SSAs against both physical and digital targets, a two-fold improvement on the baseline solution using a rear-facing mirror, a widely used solution to the SSA problem. The ShouldAR approach provides an AR-based, active SSA defense that applies to both digital and physical information entry in sensitive environments.

Comments

This work is licensed under a Creative Commons Attribution 4.0 International License.

DOI

10.1145/3678573

Journal ISSN / Book ISBN

85203641776 (Scopus)

Published Citation

Matthew Corbett, Brendan David-John, Jiacheng Shang, and Bo Ji. 2024. ShouldAR: Detecting Shoulder Surfing Attacks Using Multimodal Eye Tracking and Augmented Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8, 3, Article 97 (September 2024), 23 pages. https://doi.org/10.1145/3678573

Download
Link to Publisher

Share

COinS