Low-Latency Edge AI
The design and development of low latency AI systems for resource constrained environment
AI accelerators at the edge are scarce and shared across many tenants, yet applications still need predictable, low-latency inference. My research builds analytical performance models and serving systems that let edge clusters share accelerators efficiently while meeting per-tenant latency, accuracy, and energy targets.
Performance Models for Shared Accelerators
Sharing an accelerator across tenants requires predicting how they interfere with each other. Ibis (Liang et al., 2023) develops analytic queueing models that capture how DNN inference workloads behave under contention on shared edge accelerators such as GPUs and edge TPUs. It uses these models to drive cluster resource management that limits interference between co-located tenants.
This queueing-based approach also extends to collaborative CPU-accelerator execution for memory-constrained edge TPUs (SwapLess (Ng et al., 2026)), which won the Best Paper Award at IEEE DCOSS-IoT 2026 and anchors the model-sharing systems in Foundation Model Systems.
Hardware-Accelerated Communication
Serving latency isn’t only a compute problem. The network path between model-serving components affects serving latency as much as compute does. Our study on understanding the benefits of hardware-accelerated communication (Hanafy et al., 2023) shows that offloaded inference requests pass through multi-stage pipelines that span multiple compute nodes and proxies within an edge facility, where RDMA and GPUDirect RDMA can cut the latency these hops add.
Adaptive Inference
Dělen (Liang et al., 2023) tackles multi-tenant sharing through multi-exit DNNs, which can return a faster, less accurate result from an intermediate layer instead of always running the full model. Dělen gives each application a simple API to declare a runtime bound on the latency, accuracy, or energy of its inference requests, and it picks the exit point and resource allocation that satisfies that bound.
Meeting these bounds depends on knowing what a given device can actually achieve. Our study of energy-efficient edge inference (Hanafy et al., 2021) characterizes latency-accuracy-energy tradeoffs across dozens of pretrained models on edge accelerators, the kind of device profile that Dělen’s adaptation policies draw on.
Energy Efficiency
Fairly sharing an accelerator means accounting for energy as well as time. Energy Time Fairness (Liang et al., 2023) shows that classical fair-share schedulers equalize time slices across tenants but can let some applications consume far more than their share of energy, because GPU workloads vary widely in power draw depending on model size. ETF instead schedules for a configurable balance between time-fairness and energy-fairness, so that sharing an accelerator does not quietly favor one axis over the other.
Energy efficiency at the edge also depends on where power comes from. FM-CAC (Yang et al., 2026) approaches this from the AI side. A time-series foundation model forecasts grid carbon intensity, and that forecast drives when a battery-buffered device draws energy from the grid rather than its own battery. The forecast also selects the software pipeline variant and the hardware operating point at each moment, which trades energy savings against accuracy in the same way that CarbonEdge trades carbon savings against added latency (see also Grid-Interactive Data Centers).
