Please wait a minute...
  2020, Vol. 1 Issue (3): 243-257    doi: 10.23919/ICN.2020.0020
    
Deep reinforcement learning based computation offloading and resource allocation for low-latency fog radio access networks
G. M. Shafiqur Rahman*(),Tian Dang(),Manzoor Ahmed()
Key Laboratory of Universal Wireless Communications (Ministry of Education), Beijing University of Posts and Telecommunications, Beijing 100876, China
Download: PDF (1638 KB)      HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

Fog Radio Access Networks (F-RANs) have been considered a groundbreaking technique to support the services of Internet of Things by leveraging edge caching and edge computing. However, the current contributions in computation offloading and resource allocation are inefficient; moreover, they merely consider the static communication mode, and the increasing demand for low latency services and high throughput poses tremendous challenges in F-RANs. A joint problem of mode selection, resource allocation, and power allocation is formulated to minimize latency under various constraints. We propose a Deep Reinforcement Learning (DRL) based joint computation offloading and resource allocation scheme that achieves a suboptimal solution in F-RANs. The core idea of the proposal is that the DRL controller intelligently decides whether to process the generated computation task locally at the device level or offload the task to a fog access point or cloud server and allocates an optimal amount of computation and power resources on the basis of the serving tier. Simulation results show that the proposed approach significantly minimizes latency and increases throughput in the system.



Key wordsfog radio access networks      computation offloading      mode selection      resource allocation      distributed computation      low latency      deep reinforcement learning     
Received: 17 October 2020      Online: 19 August 2021
Corresponding Authors: G. M. Shafiqur Rahman     E-mail: shafiq.it@hotmail.com;tiandang@bupt.edu.cn;manzoor. achakzai@gmail.com
About author: G. M. Shafiqur Rahman received the MS degree from Beijing University of Posts and Telecommunications (BUPT), China in 2016 and currently is pursuing the PhD degree in the Key Laboratory of Universal Wireless Communications (Ministry of Education) at BUPT. His research interest includes heterogeneous networks, cloud computing based radio access networks, device-to-device communications, deep reinforcement learning, and the applications of stochastic geometry in fog radio access networks.|Tian Dang received the BS degree from Beijing University of Posts and Telecommunications, China in 2015. She is currently pursuing the PhD degree in the Key Laboratory of Universal Wireless Communications (Ministry of Education) at BUPT. Her research interests include the joint radio communication, caching, and computing resource allocation optimization in fog radio access networks.|Manzoor Ahmed received the bachelor degree with distinction from Balochistan University of Engineering and Technology, Khuzdar, Pakistan in 1996 and the master degree from Baluchistan University of Information Technology, Engineering and Management Sciences, Quetta, Pakistan in 2010. He has been pursuing the PhD degree at Beijing University of Posts and Telecommunications, Beijing, China, since 2011. He is serving in National Telecommunication Corporation and has worked in many different technical positions and received several national and international trainings since 2000. His research interests include the non-cooperative and cooperative game theoretic-based resource management in hierarchical heterogeneous networks, interference management in small cell networks, and 5G networks.
Cite this article:

G. M. Shafiqur Rahman,Tian Dang,Manzoor Ahmed. Deep reinforcement learning based computation offloading and resource allocation for low-latency fog radio access networks. , 2020, 1: 243-257.

URL:

http://icn.tsinghuajournals.com/10.23919/ICN.2020.0020     OR     http://icn.tsinghuajournals.com/Y2020/V1/I3/243

Fig. 1 F-RAN architecture with DRL for computation offloading. Here, CP represents cloud server and BBU represents baseband unit.
Fig. 2  DRL-based learning process and computation offloading for IoT devices.
ParameterValue
Number of fog access points L10
Number of remote radio heads J5
Number of UEs K30
Noise power (dBm/Hz)-140
Channel bandwidth B (MHz)10
Pathloss model128+37×log10?d
Size of replay memory ND2000
Learning rate α0.01
Size of minibatch M32
Discount factor ξ0.9
Table 1 Summary of simulation parameters.
Fig. 3 Evaluation of cost function with the number of epochs under different batch sizes.
Fig. 4 Evaluation of cost function with the number of epochs under different learning rates.
Fig. 5 Total loss during the training process.
Fig. 6 Evaluation of convergence performance of the algorithms with number of episodes.
Fig. 7 Evaluation of generated latency under different QoS requirements.
Fig. 8 Offloading delay versus number of computation tasks under different execution modes.
Fig. 9 Performance evaluation of power allocation under different schemes.
Fig. 10 Performance evaluation of the distributed computation of resource allocation for minimizing latency under different approache.
Fig. 11 Comparison of performance of the proposed scheme with different baselines.
SchemeNumber of epochsTime (s)
DRL27 000109.1269
Q-learning29 453153.1838
Fixed39 000231.6821
Random44 301286.8108
Table 2 Comparison of performance among different schemes.
[1]   Cao B., Li Y. X., Zhang L., Zhang L., Mumtaz S., Zhou Z. Y., and Peng M. G., When internet of things meets blockchain: Challenges in distributed consensus, IEEE Netw., vol. 33, no. 6, pp. 133-139, 2019.
[2]   Min M. H., Xiao L., Chen Y., Cheng P., Wu D., and Zhuang W. H., Learning-based computation offloading for IoT devices with energy harvesting, IEEE Trans. Veh. Technol., vol. 68, no. 2, pp. 1930-1941, 2019.
[3]   Zhao J. H., Li Q. P., Gong Y., and Zhang K., Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks, IEEE Trans. Veh. Technol., vol. 68, no. 8, pp. 7944-7956, 2019.
[4]   Cao B., Zhang L., Li Y., Feng D. Q., and Cao W., Intelligent offloading in multi-access edge computing: A state-of-the-art review and framework, IEEE Commun. Mag., vol. 57, no. 3, pp. 56-62, 2019.
[5]   Peng M. G., Yan S., Zhang K. C., and Wang C. G., Fog-computing-based radio access networks: Issues and challenges, IEEE Netw., vol. 30, no. 4, pp. 46-53, 2016.
[6]   Dang T. and Peng M. G., Joint radio communication, caching, and computing design for mobile virtual reality delivery in fog radio access networks, IEEE J. Sel. Areas Commun., vol. 37, no. 7, pp. 1594-1607, 2019.
[7]   Chen L. X., Zhou P., Gao L., and Xu J., Adaptive fog configuration for the industrial internet of things, IEEE Trans. Ind. Informat., vol. 14, no. 10, pp. 4656-4664, 2018.
[8]   Xiang H. Y., Peng M. G., Sun Y. H., and Yan S., Mode selection and resource allocation in sliced fog radio access networks: A reinforcement learning approach, IEEE Trans. Veh. Technol., vol. 69, no. 4, pp. 4271-4284, 2020.
[9]   Sun Y. H., Peng M. G., and Mao S. W., Deep reinforcement learning-based mode selection and resource management for green fog radio access networks, IEEE Internet Things J., vol. 6, no. 2, pp. 1960-1971, 2019.
[10]   Tan L. T. and Hu R. Q., Mobility-aware edge caching and computing in vehicle networks: A deep reinforcement learning, IEEE Trans. Veh. Technol., vol. 67, no. 11, pp. 10 190-10 203, 2018.
[11]   Wei Y. F., Yu F. R., Song M., and Han Z., Joint optimization of caching, computing, and radio resources for fog-enabled IoT using natural actor-critic deep reinforcement learning, IEEE Internet Things J., vol. 6, no. 2, pp. 2061-2073, 2019.
[12]   Mnih V., Kavukcuoglu K., Silver D., Rusu A. A., Veness J., Bellemare M. G., Graves A., Riedmiller M., Fidjeland A. K., Ostrovski G., et al., Human-level control through deep reinforcement learning, Nature, vol. 518, no. 7540, pp. 529-533, 2015.
[13]   Wu C., Yoshinaga T., Ji Y. S., Murase T., and Zhang Y., A reinforcement learning-based data storage scheme for vehicular ad hoc networks, IEEE Trans. Veh. Technol., vol. 66, no. 7, pp. 6336-6348, 2017.
[14]   Liu B. H., Liu C. X., Peng M. G., Liu Y. Q., and Yan S., Resource allocation for non-orthogonal multiple access-enabled fog radio access networks, IEEE Trans. Wirel. Commun., vol. 19, no. 6, pp. 3867-3878, 2020.
[15]   Liu B. H., Liu C. X., and Peng M. G., Resource allocation for energy-efficient MEC in NOMA-enabled massive IoT networks, IEEE J. Sel. Areas Commun., .
doi: 10.1109/JSAC.2020.3018809
[16]   Liu L. Q., Chang Z., Guo X. J., Mao S. W., and Ristaniemi T., Multiobjective optimization for computation offloading in fog computing, IEEE Internet Things J., vol. 5, no. 1, pp. 283-294, 2018.
[17]   Zhao Z. Y., Bu S. Q., Zhao T. Z., Yin Z. P., Peng M. G., Ding Z. G., and Quek T. Q. S., On the design of computation offloading in fog radio access networks, IEEE Trans. Veh. Technol., vol. 68, no. 7, pp. 7136-7149, 2019.
[18]   Zhang L., Cao B., Li Y., Peng M. G., and Feng G., A multi-stage stochastic programming based offloading policy for fog enabled IoT-eHealth, IEEE J. Sel. Areas Commun., .
doi: 10.1109/JSAC.2020.3020659
[19]   Du J. B., Zhao L. Q., Feng J., and Chu X. L., Computation offloading and resource allocation in mixed fog/cloud computing systems with min-max fairness guarantee, IEEE Trans. Commun., vol. 66, no. 4, pp. 1594-1608, 2018.
[20]   Liu Y. M., Yu F. R., Li X., Ji H., and Leung V. C. M., Distributed resource allocation and computation offloading in fog and cloud networks with non-orthogonal multiple access, IEEE Trans. Veh. Technol., vol. 67, no. 12, pp. 12 137-12 151, 2018.
[21]   Cao B., Xia S. C., Han J. W., and Li Y., A distributed game methodology for crowdsensing in uncertain wireless scenario, IEEE Trans. Mobile Comput., vol. 19, no. 1, pp. 15-28, 2020.
[22]   Sun Y. H., Peng M. G., Zhou Y. C., Huang Y. Z., and Mao S. W., Application of machine learning in wireless networks: Key techniques and open issues, IEEE Commun. Surv. Tutor., vol. 21, no. 4, pp. 3072-3108, 2019.
[23]   Lei L., Xu H. J., Xiong X., Zheng K., Xiang W., and Wang X. B., Multi-user resource control with deep reinforcement learning in IoT edge computing, arXiv preprint arXiv: 1906.07860, 2019.
[24]   Chen X. F., Zhang H. G., Wu C., Mao S. W., Ji Y. S., and Bennis M., Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning, IEEE Internet Things J., vol. 6, no. 3, pp. 4005-4018, 2019.
[25]   Huang L., Bi S. Z., and Zhang Y. J. A., Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks, IEEE Trans. Mobile Comput., vol. 19, no. 311, pp. 2581-2593, 2020.
[26]   Liu Y., Yu H. M., Xie S. L., and Zhang Y., Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks, IEEE Trans. Veh. Technol., vol. 68, no. 11, pp. 11 158-11 168, 2019.
[27]   Ye H., Li G. Y., and Juang B. H. F., Deep reinforcement learning based resource allocation for V2V communications, IEEE Trans. Veh. Technol., vol. 68, no. 4, pp. 3163-3173, 2019.
[28]   Li R. P., Zhao Z. F., Sun Q., I C. L., Yang C. Y., Chen X. F., Zhao M. J., Zhang H. G., Deep reinforcement learning for resource management in network slicing, IEEE Access, vol. 6, pp. 74 429-74 441, 2018.
[29]   Meng F., Chen P., Wu L. N., and Cheng J. L., Senior, power allocation in multi-user cellular networks: Deep reinforcement learning approaches, IEEE Trans. Wirel. Commun., vol. 19, no. 10, pp. 6255-6267, 2020.
[30]   Li X. J., Fang J., Cheng W., Duan H. P., Chen Z., and Li H. B., Intelligent power control for spectrum sharing in cognitive radios: A deep reinforcement learning approach, IEEE Access, vol. 6, pp. 25 463-25 473, 2018.
[31]   He X. M., Wang K., Huang H. W., Miyazaki T., Wang Y. X., and Guo S., Green resource allocation based on deep reinforcement learning in content-centric IoT, IEEE Trans. Emerg. Top. Comput., vol. 8, no. 3, pp. 781-796, 2020.
[32]   Yousefpour A., Ishigaki G., Gour R., and Jue J. P., On reducing IoT service delay via fog offloading, IEEE Internet Things J., vol. 5, no. 2, pp. 998-1010, 2018.
[33]   Sutton R. S. and Barto A. G.. Introduction to Reinforcement Learning, 2nd ed. Cambridge, MA, USA: MIT Press, 1998.
[34]   Zhu H., Cao Y., Wei X., Wang W., Jiang T., and Jin S., Caching transient data for internet of things: A deep reinforcement learning approach, IEEE Internet Things J., vol. 6, no. 2, pp. 2074-2083, 2019.
No related articles found!