Multi-Model Running Latency Optimization in an Edge Computing Paradigm

Li, Peisong, Wang, Xinheng ORCID: 0000-0001-8771-8901, Huang, Kaizhu, Huang, Yi, Li, Shancang and Iqbal, Muddesar (2022) Multi-Model Running Latency Optimization in an Edge Computing Paradigm. SENSORS, 22 (16). 6097-.

Access the full-text of this item by clicking on the Open Access link.

PDF
Multi-Model Running Latency Optimization in an Edge Computing Paradigm.pdf - Open Access published version
Download (2MB) | Preview

Official URL: http://dx.doi.org/10.3390/s22166097

Abstract

Recent advances in both lightweight deep learning algorithms and edge computing increasingly enable multiple model inference tasks to be conducted concurrently on resource-constrained edge devices, allowing us to achieve one goal collaboratively rather than getting high quality in each standalone task. However, the high overall running latency for performing multi-model inferences always negatively affects the real-time applications. To combat latency, the algorithms should be optimized to minimize the latency for multi-model deployment without compromising the safety-critical situation. This work focuses on the real-time task scheduling strategy for multi-model deployment and investigating the model inference using an open neural network exchange (ONNX) runtime engine. Then, an application deployment strategy is proposed based on the container technology and inference tasks are scheduled to different containers based on the scheduling strategies. Experimental results show that the proposed solution is able to significantly reduce the overall running latency in real-time applications.

Item Type:	Article
Uncontrolled Keywords:	edge computing, latency optimization, multi-model, task scheduling, autonomous driving, AI
Divisions:	Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	24 Apr 2023 14:26
Last Modified:	24 Apr 2023 14:26
DOI:	10.3390/s22166097
Open Access URL:	https://doi.org/10.3390/s22166097
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3169928