Internet of Battlefield Things (IoBTs) are well positioned to take advantage of recent technology trends that have led to the development of low-power neural accelerators and low-cost high-performance sensors. However, a key challenge that needs to be dealt with is that despite all the advancements, edge devices remain resource-constrained, thus prohibiting complex deep neural networks from deploying and deriving actionable insights from various sensors. Furthermore, deploying sophisticated sensors in a distributed manner to improve decision-making also poses an extra challenge of coordinating and exchanging data between the nodes and server. We propose an architecture that abstracts away these thorny deployment considerations from an end-user (such as a commander or warfighter). Our architecture can automatically compile and deploy the inference model into a set of distributed nodes and server while taking into consideration of the resource availability, variation, and uncertainties.
Exploiting Scene and Body Contexts in Controlling Continuous Vision Body Cameras
Shiwei Fang, Ketan Mayer-Patel, Shahriar Nirjon
Ad Hoc Networks Journal, Elsevier, Volumn 113, Mar. 2021.
Ever-increasing performance at decreasing price has fueled camera deployments in a wide variety of real-world applications—making the case stronger for battery-powered, continuous-vision camera systems. However, given the state-of-the-art battery technology and embedded systems, most battery-powered mobile devices still do not support continuous vision. In order to reduce energy and storage requirements, there have been proposals to offload energy-demanding computations to the cloud Naderiparizi et al. (2016), to discard uninteresting video frames Naderiparizi et al. (2017), and to use additional sensors to detect and predict when to turn on the camera Bahl et al. (2012) . However, these proposals either require a fat communication bandwidth or have to sacrifice capturing of important events.
In this paper, we present — ZenCam, which is an always-on body camera that exploits readily available information in the encoded video stream from the on-chip firmware to classify the dynamics of the scene. This scene-context is further combined with simple inertial measurement unit (IMU)-based activity level-context of the wearer to optimally control the camera configuration at run-time to keep the device under the desired energy budget. We describe the design and implementation of ZenCam and thoroughly evaluate its performance in real-world scenarios. Our evaluation shows a 29.8%–35% reduction in energy consumption and 48.1-49.5% reduction in storage usage when compared to a standard baseline setting of 1920x1080 at 30fps while maintaining a competitive or better video quality at the minimal computational overhead.
EyeFi: Fast Human Identification Through Vision and WiFi-based Trajectory Matching
Human sensing, motion trajectory estimation, and identification are central to a wide range of applications in many domains such as retail stores, surveillance, public safety, public address, smart homes and cities, and access control. Existing solutions either require facial recognition or installation and maintenance of multiple units, or they lack long-term re-identification capability. In this paper, we propose a novel system -- called EyeFi -- that combines WiFi and camera on a standalone device to overcome these limitations. EyeFi integrates a WiFi chipset to an overhead camera and fuses motion trajectories obtained from both vision and RF modalities to identify individuals. In order to do that, EyeFi uses a student-teacher model to train a neural network to estimate the Angle of Arrival (AoA) of WiFi packets from the CSI values. Based on extensive evaluation using real-world data, we observe that EyeFi improves WiFi CSI based AoA estimation accuracy by more than 30% and offers 3,800 times computational speed over the state-of-the-art solution. In a real-world environment, EyeFi's accuracy of person identification averages 75% when the number of people varies from 2 to 10.
This paper introduces SuperRF – which takes radio frequency (RF) signals from an off-the-shelf, low-cost, 77GHz mmWave radar and produces an enhanced 3D RF representation of a scene. SuperRF is useful in scenarios where camera and other types of sensors do not work, or not allowed due to privacy concerns, or their performance is impacted due to bad lighting conditions and occlusions, or an alternate RF sensing system like synthetic aperture radar (SAR) is too large, inconvenient, and costly. Applications of SuperRF includes navigation and planning of autonomous and semi-autonomous systems, human-robot interactions and social robotics, and elderly and/or patient monitoring in-home healthcare scenarios. We use low-cost, off-the-shelf parts to capture RF signals and to train SuperRF. The novelty of SuperRF lies in its use of deep learning algorithm, followed by a compressed sensing-based iterative algorithm that further enhances the output, to generate a fine-grained 3D representation of an RF scene from its sparse RF representation – which a mmWave radar of the same class cannot achieve without instrumenting the system with large sized multiple antennas or physically moving the antenna over a longer period in time. We demonstrate the feasibility and effectiveness through an in-depth evaluation.
In this paper, we present - ZenCam, which is an always-on body camera that exploits readily available information in the encoded video stream from the on-chip firmware to classify the dynamics of the scene. This scene-context is further combined with simple inertial measurement unit (IMU)-based activity level-context of the wearer to optimally control the camera configuration at run-time to keep the device under the desired energy budget. We describe the design and implementation of ZenCam and thoroughly evaluate its performance in real-world scenarios. Our evaluation shows a 29.8-35% reduction in energy consumption and 48.1-49.5% reduction in storage usage when compared to a standard baseline setting of 1920×1080 at 30fps while maintaining a competitive or better video quality at the minimal computational overhead.
Low Swing TSV Signaling using Novel Level Shifters with Single Supply Voltage
Shiwei Fang, Emre Salman
IEEE International Symposium on Circuits and Systems (ISCAS), May 2015.
Low swing TSV signaling is proposed for three-dimensional (3D) integrated circuits (ICs) to reduce dynamic power consumption. Novel level shifters are designed to lower the voltage swing before the TSV and to pull the voltage swing back to full rail at the far end of the TSV. Proposed level shifters operate with a single supply voltage, thereby reducing the overall cost. Critical TSV capacitance beyond which the proposed scheme saves dynamic power is determined. Up to 42% reduction in overall power is demonstrated with a voltage swing of 0.5 V, where the supply voltage is 1 V.
With the rise of hailing services, people are increasingly relying on shared mobility (e.g., Uber, Lyft) drivers to pick up for transportation. However, such drivers and riders have difficulties finding each other in urban areas as GPS signals get blocked by skyscrapers, in crowded environments (e.g., in stadiums, airports, and bars), at night, and in bad weather. It wastes their time, creates a bad user experience, and causes more CO2 emissions due to idle driving. In this work, we explore the potential of Wi-Fi to help drivers to determine the street side of the riders. Our proposed system is called CarFi that uses Wi-Fi CSI from two antennas placed inside a moving vehicle and a data-driven technique to determine the street side of the rider. By collecting real-world data in realistic and challenging settings by blocking the signal with other people and other parked cars, we see that CarFi is 95.44% accurate in rider-side determination in both line of sight (LoS) and non-line of sight (nLoS) conditions, and can be run on an embedded GPU in real-time.
Design and Deployment of a Multi-Modal Multi-Node Sensor Data Collection Platform
Sensing and data collection platforms are the crucial components of high-quality datasets that can fuel advancements in research. However, such platforms usually are ad-hoc designs and are limited in sensor modalities. In this paper, we discuss our experience designing and deploying a multi-modal multi-node sensor data collection platform that can be utilized for various data collection tasks. The main goal of this platform is to create a modality-rich data collection platform suitable for Internet of Things (IoT) applications with easy reproducibility and deployment, which can accelerate data collection and downstream research tasks.
Dataset: Person Tracking and Identification using Cameras and Wi-Fi Channel State Information (CSI) from Smartphones
Shiwei Fang, Sirajum Munir, Shahriar Nirjon
The 3rd International Workshop on Data: Acquisition To Analysis (DATA ’20) (SenSys + BuildSys). ACM, Nov. 2020.
Human sensing, motion trajectory estimation, and identification are crucial to applications such as customer analysis, public safety, smart homes and cities, and access control. In the wake of the global COVID-19 pandemic, the ability to perform contact tracing effectively is vital to limit the spread of infectious diseases. Although vision-based solutions such as facial recognition can potentially scale to millions of people for identification, the privacy implications and laws to banning such a technology limit its applicability in the real world. Other techniques may require installations and maintenance of multiple units, and/or lack long-term re-identification capability. We present a dataset to fuse WiFi Channel State Information (CSI) and camera-based location information for person identification and tracking. While previous works focused on collecting WiFi CSI from stationary transmitters and receivers (laptop, desktop, or router), our WiFi CSI data are generated from a smartphone that is carried while someone is moving. In addition, we collect camera-generated real-world coordinate for each WiFi packet that can serve as ground truth location. The dataset is collected in different environments and with various numbers of persons in the scene at several days to capture real-world variations.
Demo Abstract: Fusing WiFi and Camera for Fast Motion Tracking and Person Identification
Shiwei Fang, Sirajum Munir, Shahriar Nirjon
ACM Conference on Embedded Networked Sensor Systems (SenSys), Nov. 2020.
Human sensing, motion tracking, and identification are at the center of numerous applications such as customer analysis, public safety, smart cities, and surveillance. To enable such capabilities, existing solutions mostly rely on vision-based approaches, e.g., facial recognition that is perceived to be too privacy invasive. Other camera-based approaches using body appearances lack long-term re-identification capability. WiFi-based approaches require the installation and maintenance of multiple units. We propose a novel system - called EyeFi - that overcomes these limitations on a standalone device by fusing camera and WiFi data. We use a three-antenna WiFi chipset to measure WiFi Channel State Information (CSI) to estimate the Angle of Arrival (AoA) using a neural network trained with a novel student-teacher model. Then, we perform cross modal (WiFi, camera) trajectory matching to identify individuals using the MAC address of the incoming WiFi packets. We demonstrate our work using real-world data and showcase improvements over traditional optimization-based methods in terms of accuracy and speed.
Non-Line-of-Sight Around the Corner Human Presence Detection Using Commodity WiFi
Devices
Shiwei Fang, Ron Alterovitz, Shahriar Nirjon
Workshop on Device-Free Human Sensing (DFHS). ACM, Nov. 2019.
As robots penetrate into real-world environments, practical human-robot co-existence issues such as the requirement for safe human-robot interaction are becoming increasingly important. In almost every vision-capable mobile robot, the field of view of the robot is occluded by the presence of obstacles such as indoor walls, furniture, and humans. Such occlusions force the robots to be stationary or to move slowly so that they can avoid collisions and violations of entry into the personal spaces of humans. We see this as a barrier to robots being able to optimally plan motions with reasonable speeds. In order to solve this problem, we propose to augment the sensing capability of a robot by using a commodity WiFi receiver. Using our proposed method, a robot can observe the changes in the properties of received signals, and thus be able to infer whether a human is present behind the wall or obstacles, which enhances its ability to plan and navigate efficiently and intelligently.
Demo Abstract: AI-Enhanced 3D RF Representation Using Low-Cost mmWave Radar
Shiwei Fang, Shahriar Nirjon
ACM Conference on Embedded Networked Sensor Systems (SenSys), Nov. 2018.
This paper introduces a system that takes radio frequency (RF) signals from an off-the-shelf, low-cost, 77 GHz mm Wave radar and produces an enhanced 3D RF representation of a scene. Such a system can be used in scenarios where camera and other types of sensors do not work, or their performance is impacted due to bad lighting conditions and occlusions, or an alternate RF sensing system like synthetic aperture radar (SAR) is too large, inconvenient, and costly. The enhanced RF representation can be used in numerous applications such as robot navigation, human-computer interaction, and patient monitoring. We use off-the-shelf parts to capture RF signals and collect our own data set for training and testing of the approach. The novelty of the system lies in its use of AI to generate a fine-grained 3D representation of an RF scene from its sparse RF representation which a mmWave radar of the same class cannot achieve.
Distributed Adaptive Model Predictive Control of a Cluster of Autonomous and
Context-Sensitive Body Cameras
Shiwei Fang, Ketan Mayer-Patel, Shahriar Nirjon
Workshop on Wearable Systems and Applications (WearSys). ACM, June 2017.
Increasing deployment of body cameras by the law enforcement agencies makes us rethink the relation between the camera and the public. In contrast to current implementations of a body camera that use a power-hungry default configuration and can only be turned on and off by an officer, we propose an idea that the camera should be autonomous and active all the time. By leveraging the information from an on-board inertial measurement unit (IMU), these autonomous cameras should dynamically adjust their configuration in order to keep the device under the desired energy budget. To enable such a system, we propose a distributed adaptive model predictive controller for a system of body cameras, which allows the collaboration between multiple cameras which is currently not available in existing implementations.