EAGER: Automatic Reconstruction of Typed Input from Compromising Reflections

Principal Investigator: Jan-Michael Frahm

Funding Agency: National Science Foundation

Agency Number: IIS-1148895

Abstract

This project explores computer vision techniques aimed at exploiting compromising reflections associated with data input in mobile electronic devices such as smart phones. The ubiquity of these personal communication devices and their growing roles in data manipulation tasks, make unintended visual emanations an exploitable liability to data security. Nevertheless, there is still a gap in understanding of both the limitations of these techniques as well as the availability of effective mitigation mechanisms. It is the goal of this work to contribute to filling this conceptual gap.

The study builds upon recent state of the art techniques for automatic reconstruction of typed input from compromising reflections, comprising of robust keystroke event detection and classification mechanisms coupled to natural language processing modules. Such paradigm is both effective and amenable to low cost implementation in commodity devices. Based on these new developments, threat scenarios are no longer restricted to controlled scenarios using specialized equipment, but rather consist of highly flexible and possibly impromptu attacks. The project develops advanced cross-platform data input transcription prototypes used within a threat validation framework. This framework provides a characterization of both threat scenario operational limitations (e.g., imaging resolution, scene illumination, computational requirements) as well as the performance characteristics (e.g., robustness, accuracy) of the different vulnerability exploitation mechanisms. Moreover, the results of the analysis of diverse threat scenarios are being used to identify and develop appropriate mitigation mechanisms when possible.