Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

5-2021

Abstract

The paper introduces a novel, holistic approach for robust Screen-Camera Communication (SCC), where video content on a screen is visually encoded in a human-imperceptible fashion and decoded by a camera capturing images of such screen content. We first show that state-of-the-art SCC techniques have two key limitations for in-the-wild deployment: (a) the decoding accuracy drops rapidly under even modest screen extraction errors from the captured images, and (b) they generate perceptible flickers on common refresh rate screens even with minimal modulation of pixel intensity. To overcome these challenges, we introduce DeepLight, a system that incorporates machine learning (ML) models in the decoding pipeline to achieve humanly-imperceptible, moderately high SCC rates under diverse real-world conditions. DeepLight's key innovation is the design of a Deep Neural Network (DNN) based decoder that collectively decodes all the bits spatially encoded in a display frame, without attempting to precisely isolate the pixels associated with each encoded bit. In addition, DeepLight supports imperceptible encoding by selectively modulating the intensity of only the Blue channel, and provides reasonably accurate screen extraction (IoU values ≥ 83%) by using state-of-the-art object detection DNN pipelines. We show that a fully functional DeepLight system is able to robustly achieve high decoding accuracy (frame error rate < 0.2) and moderately-high data goodput (≥0.95 Kbps) using a human-held smartphone camera, even over larger screen-camera distances (~ 2m).

Keywords

Screen-camera communication, Visible light communication, Imperceptible, Deep neural networks, Perception, Flicker-free

Discipline

Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

IPSN '21: Proceedings of the 20th International Conference on Information Processing in Sensor Networks: Virtual, Nashville, TN, May 18-21

First Page

238

Last Page

253

ISBN

9781450380980

Identifier

10.1145/3412382.3458269

Publisher

ACM

City or Country

New York

Embargo Period

5-26-2021

Copyright Owner and License

Publisher

Additional URL

https://doi.org/10.1145/3412382.3458269

Download

Share

COinS