Publications:


Journal & Conference:

Exploiting Scene and Body Contexts in Controlling Continuous Vision Body Cameras

Shiwei Fang, Ketan Mayer-Patel, Shahriar Nirjon

Ad Hoc Networks Journal, Elsevier, Volumn 113, Mar. 2021.

PDF
Abstract
Ever-increasing performance at decreasing price has fueled camera deployments in a wide variety of real-world applications—making the case stronger for battery-powered, continuous-vision camera systems. However, given the state-of-the-art battery technology and embedded systems, most battery-powered mobile devices still do not support continuous vision. In order to reduce energy and storage requirements, there have been proposals to offload energy-demanding computations to the cloud Naderiparizi et al. (2016), to discard uninteresting video frames Naderiparizi et al. (2017), and to use additional sensors to detect and predict when to turn on the camera Bahl et al. (2012) . However, these proposals either require a fat communication bandwidth or have to sacrifice capturing of important events.
In this paper, we present — ZenCam, which is an always-on body camera that exploits readily available information in the encoded video stream from the on-chip firmware to classify the dynamics of the scene. This scene-context is further combined with simple inertial measurement unit (IMU)-based activity level-context of the wearer to optimally control the camera configuration at run-time to keep the device under the desired energy budget. We describe the design and implementation of ZenCam and thoroughly evaluate its performance in real-world scenarios. Our evaluation shows a 29.8%–35% reduction in energy consumption and 48.1-49.5% reduction in storage usage when compared to a standard baseline setting of 1920x1080 at 30fps while maintaining a competitive or better video quality at the minimal computational overhead.

EyeFi: Fast Human Identification Through Vision and WiFi-based Trajectory Matching

Shiwei Fang, Md Tamzeed Islam, Sirajum Munir, Shahriar Nirjon

International Conference on Distributed Computing in Sensor Systems (DCOSS). IEEE, May 2020.

PDF
Abstract
Human sensing, motion trajectory estimation, and identification are central to a wide range of applications in many domains such as retail stores, surveillance, public safety, public address, smart homes and cities, and access control. Existing solutions either require facial recognition or installation and maintenance of multiple units, or they lack long-term re-identification capability. In this paper, we propose a novel system -- called EyeFi -- that combines WiFi and camera on a standalone device to overcome these limitations. EyeFi integrates a WiFi chipset to an overhead camera and fuses motion trajectories obtained from both vision and RF modalities to identify individuals. In order to do that, EyeFi uses a student-teacher model to train a neural network to estimate the Angle of Arrival (AoA) of WiFi packets from the CSI values. Based on extensive evaluation using real-world data, we observe that EyeFi improves WiFi CSI based AoA estimation accuracy by more than 30% and offers 3,800 times computational speed over the state-of-the-art solution. In a real-world environment, EyeFi's accuracy of person identification averages 75% when the number of people varies from 2 to 10.
Project Page

SuperRF: Enhanced 3D RF Representation Using Stationary Low-Cost mmWave Radar

Shiwei Fang, Shahriar Nirjon

International Conference on Embedded Wireless Systems and Networks (EWSN). ACM, Feb. 2020.

PDF
Abstract
This paper introduces SuperRF – which takes radio frequency (RF) signals from an off-the-shelf, low-cost, 77GHz mmWave radar and produces an enhanced 3D RF representation of a scene. SuperRF is useful in scenarios where camera and other types of sensors do not work, or not allowed due to privacy concerns, or their performance is impacted due to bad lighting conditions and occlusions, or an alternate RF sensing system like synthetic aperture radar (SAR) is too large, inconvenient, and costly. Applications of SuperRF includes navigation and planning of autonomous and semi-autonomous systems, human-robot interactions and social robotics, and elderly and/or patient monitoring in-home healthcare scenarios. We use low-cost, off-the-shelf parts to capture RF signals and to train SuperRF. The novelty of SuperRF lies in its use of deep learning algorithm, followed by a compressed sensing-based iterative algorithm that further enhances the output, to generate a fine-grained 3D representation of an RF scene from its sparse RF representation – which a mmWave radar of the same class cannot achieve without instrumenting the system with large sized multiple antennas or physically moving the antenna over a longer period in time. We demonstrate the feasibility and effectiveness through an in-depth evaluation.
Project Page

ZenCam: Context-Driven Control of Autonomous Body Cameras

Shiwei Fang, Ketan Mayer-Patel, Shahriar Nirjon

International Conference on Distributed Computing in Sensor Systems (DCOSS). IEEE, May 2019. (Best Paper Award)

PDF
Abstract
In this paper, we present - ZenCam, which is an always-on body camera that exploits readily available information in the encoded video stream from the on-chip firmware to classify the dynamics of the scene. This scene-context is further combined with simple inertial measurement unit (IMU)-based activity level-context of the wearer to optimally control the camera configuration at run-time to keep the device under the desired energy budget. We describe the design and implementation of ZenCam and thoroughly evaluate its performance in real-world scenarios. Our evaluation shows a 29.8-35% reduction in energy consumption and 48.1-49.5% reduction in storage usage when compared to a standard baseline setting of 1920×1080 at 30fps while maintaining a competitive or better video quality at the minimal computational overhead.

Low Swing TSV Signaling using Novel Level Shifters with Single Supply Voltage

Shiwei Fang, Emre Salman

IEEE International Symposium on Circuits and Systems (ISCAS), May 2015.

PDF
Abstract
Low swing TSV signaling is proposed for three-dimensional (3D) integrated circuits (ICs) to reduce dynamic power consumption. Novel level shifters are designed to lower the voltage swing before the TSV and to pull the voltage swing back to full rail at the far end of the TSV. Proposed level shifters operate with a single supply voltage, thereby reducing the overall cost. Critical TSV capacitance beyond which the proposed scheme saves dynamic power is determined. Up to 42% reduction in overall power is demonstrated with a voltage swing of 0.5 V, where the supply voltage is 1 V.

Workshop, Demo, and Poster:

Dataset: Person Tracking and Identification using Cameras and Wi-Fi Channel State Information (CSI) from Smartphones

Shiwei Fang, Sirajum Munir, Shahriar Nirjon

3rd Workshop on Data Acquisition To Analysis (DATA). ACM, Nov. 2020.

PDF
Abstract
Human sensing, motion trajectory estimation, and identification are crucial to applications such as customer analysis, public safety, smart homes and cities, and access control. In the wake of the global COVID-19 pandemic, the ability to perform contact tracing effectively is vital to limit the spread of infectious diseases. Although vision-based solutions such as facial recognition can potentially scale to millions of people for identification, the privacy implications and laws to banning such a technology limit its applicability in the real world. Other techniques may require installations and maintenance of multiple units, and/or lack long-term re-identification capability. We present a dataset to fuse WiFi Channel State Information (CSI) and camera-based location information for person identification and tracking. While previous works focused on collecting WiFi CSI from stationary transmitters and receivers (laptop, desktop, or router), our WiFi CSI data are generated from a smartphone that is carried while someone is moving. In addition, we collect camera-generated real-world coordinate for each WiFi packet that can serve as ground truth location. The dataset is collected in different environments and with various numbers of persons in the scene at several days to capture real-world variations.

Demo Abstract: Fusing WiFi and Camera for Fast Motion Tracking and Person Identification

Shiwei Fang, Sirajum Munir, Shahriar Nirjon

ACM Conference on Embedded Networked Sensor Systems (SenSys), Nov. 2020.

PDF
Abstract
Human sensing, motion tracking, and identification are at the center of numerous applications such as customer analysis, public safety, smart cities, and surveillance. To enable such capabilities, existing solutions mostly rely on vision-based approaches, e.g., facial recognition that is perceived to be too privacy invasive. Other camera-based approaches using body appearances lack long-term re-identification capability. WiFi-based approaches require the installation and maintenance of multiple units. We propose a novel system - called EyeFi - that overcomes these limitations on a standalone device by fusing camera and WiFi data. We use a three-antenna WiFi chipset to measure WiFi Channel State Information (CSI) to estimate the Angle of Arrival (AoA) using a neural network trained with a novel student-teacher model. Then, we perform cross modal (WiFi, camera) trajectory matching to identify individuals using the MAC address of the incoming WiFi packets. We demonstrate our work using real-world data and showcase improvements over traditional optimization-based methods in terms of accuracy and speed.

Non-Line-of-Sight Around the Corner Human Presence Detection Using Commodity WiFi Devices

Shiwei Fang, Ron Alterovitz, Shahriar Nirjon

Workshop on Device-Free Human Sensing (DFHS). ACM, Nov. 2019.

PDF
Abstract
As robots penetrate into real-world environments, practical human-robot co-existence issues such as the requirement for safe human-robot interaction are becoming increasingly important. In almost every vision-capable mobile robot, the field of view of the robot is occluded by the presence of obstacles such as indoor walls, furniture, and humans. Such occlusions force the robots to be stationary or to move slowly so that they can avoid collisions and violations of entry into the personal spaces of humans. We see this as a barrier to robots being able to optimally plan motions with reasonable speeds. In order to solve this problem, we propose to augment the sensing capability of a robot by using a commodity WiFi receiver. Using our proposed method, a robot can observe the changes in the properties of received signals, and thus be able to infer whether a human is present behind the wall or obstacles, which enhances its ability to plan and navigate efficiently and intelligently.

Demo Abstract: AI-Enhanced 3D RF Representation Using Low-Cost mmWave Radar

Shiwei Fang, Shahriar Nirjon

ACM Conference on Embedded Networked Sensor Systems (SenSys), Nov. 2018.

PDF
Abstract
This paper introduces a system that takes radio frequency (RF) signals from an off-the-shelf, low-cost, 77 GHz mm Wave radar and produces an enhanced 3D RF representation of a scene. Such a system can be used in scenarios where camera and other types of sensors do not work, or their performance is impacted due to bad lighting conditions and occlusions, or an alternate RF sensing system like synthetic aperture radar (SAR) is too large, inconvenient, and costly. The enhanced RF representation can be used in numerous applications such as robot navigation, human-computer interaction, and patient monitoring. We use off-the-shelf parts to capture RF signals and collect our own data set for training and testing of the approach. The novelty of the system lies in its use of AI to generate a fine-grained 3D representation of an RF scene from its sparse RF representation which a mmWave radar of the same class cannot achieve.

Distributed Adaptive Model Predictive Control of a Cluster of Autonomous and Context-Sensitive Body Cameras

Shiwei Fang, Ketan Mayer-Patel, Shahriar Nirjon

Workshop on Wearable Systems and Applications (WearSys). ACM, June 2017.

PDF
Abstract
Increasing deployment of body cameras by the law enforcement agencies makes us rethink the relation between the camera and the public. In contrast to current implementations of a body camera that use a power-hungry default configuration and can only be turned on and off by an officer, we propose an idea that the camera should be autonomous and active all the time. By leveraging the information from an on-board inertial measurement unit (IMU), these autonomous cameras should dynamically adjust their configuration in order to keep the device under the desired energy budget. To enable such a system, we propose a distributed adaptive model predictive controller for a system of body cameras, which allows the collaboration between multiple cameras which is currently not available in existing implementations.