Publication Type

PhD Dissertation

Version

publishedVersion

Publication Date

12-2019

Abstract

The widespread availability of sensors on personal devices (e.g., smartphones, smartwatches) and other cheap, commoditized IoT devices in the environment has opened up the opportunity for developing applications that capture and enhance various lifestyle-driven daily activities of individuals. Moreover, there is a growing trend of leveraging ubiquitous computing technologies to improve physical health and wellbeing. Several of the lifestyle monitoring applications rely primarily on the capability of recognizing contextually relevant human movements, actions and gestures. As such, gesture recognition techniques, and gesture-based analytics have emerged as a fundamental component for realizing personalized lifestyle applications.

This thesis explores how such wealth of data sensed from ubiquitously available devices can be utilized for inferring fine-grained gestures. Subsequently, it explores how gestures can be used to profile user behavior during daily activities and outlines mechanisms to tackle various real-world challenges. With two daily activities (shopping and exercising) as examples, it then demonstrates that unobtrusive, accurate and robust monitoring of various aspects of these activities is indeed possible with minimal overhead. Such monitoring can then, in future, enable useful applications (e.g., smart reminder in a retail store or digital personal coach in a gym).

First, this thesis presents the IRIS platform, which explores how appropriate mining of sensors available in personal devices such as a smartphone and a smartwatch can be used to infer micro-gestural activities, and how such activities help reveal latent behavioral attributes of individual consumers inside a retail store. It first investigates how inertial sensor data (e.g., accelerometer, gyroscope) from a smartphone can be used to appropriately decompose an entire store visit into a series of modular and hierarchical individual interactions, modeled as a sequence of in-aisle interactions, interspersed with non-aisle movement. Further, by combining such sensor data from a wrist-worn smartwatch and by deriving discriminative features, the IRIS platform demonstrates that different facets of a shopper’s interaction with individual items (e.g., picking an item, putting an item in trolley), as well as attributes of the overall shopping episode or the store, can be inferred.

This thesis next investigates the possibility of using a wearable-free sensing modality for fine-grained and unobtrusive monitoring of multiple aspects of individuals’ gym exercises. It describes the W8-Scope approach that requires no on-body instrumentation and leverages only simple accelerometer and magnetometer sensors (on a cheap IoT device) attached to the weight stack of an exercise machine to infer various exercise gestures, and thereby identify related novel attributes such as the amount of weight lifted, the correctness of exercise execution and identify the user who is performing the exercise. It then also experimentally demonstrates the feasibility of evolving W8-Scope’s machine learning-based classifiers to accommodate the medium-time scale (e.g., across weeks or months) changes in an individual’s exercise behavior (an issue that has received insufficient attention to date).

Finally, this thesis explores the possibility of accurately inferring complex activities and gestures performed concurrently by multiple individuals in an indoor gym environment. It introduces a system that utilizes a hybrid architecture, combining sensor data from ‘earables’ with non-personal IoT sensors attached to gym equipment, for individual-specific fine-grained monitoring of weight-based exercises in a gym. Using real-world studies conducted with multiple concurrent gym-goers, this thesis validates that accurate association of “user-equipment” pairings is indeed possible, for a majority of common exercises, in spite of the significant signal dampening on the earable. Moreover, it demonstrates how features from the earable and IoT sensors can be combined to significantly increase the accuracy and robustness of exercise recognition. In future, the real-time exercise analytics capabilities developed in this thesis can be used to enable targeted and individualized real-time feedback on user dynamics and increase user engagement.

Keywords

Mobile Systems, Gesture Recognition, Wearables, IoT, Lifestyle Monitoring, Pervasive Sensing

Degree Awarded

PhD in Information Systems

Discipline

Software Engineering

Supervisor(s)

MISRA, Archan

Publisher

Singapore Management University

City or Country

Singapore

Copyright Owner and License

Author

Download

Share

COinS