Please wait a minute...
  2021, Vol. 2 Issue (2): 150-162    doi: 10.23919/ICN.2021.0010
Regular Articles     
Distributed reinforcement learning based framework for energy-effcient UAV relay against jamming
Weihang Wang,Zefang Lv,Xiaozhen Lu,Yi Zhang*(),Liang Xiao()
Department of Information and Communication Engineering, Xiamen University, Xiamen 361005, China
Beijing Key Laboratory of Mobile Computing and Pervasive Device, Beijing 100190, China
Download: PDF (885 KB)      HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

Unmanned aerial vehicle (UAV) network is vulnerable to jamming attacks, which may cause severe damage like communication outages. Due to the energy constraint, the source UAV cannot blindly enlarge the transmit power, along with the complex network topology with high mobility, which makes the destination UAV unable to evade the jammer by flying at will. To maintain communication with a limited battery capacity in the UAV networks in the presence of a greedy jammer, in this paper, we propose a distributed reinforcement learning (RL) based energy-efficient framework for the UAV networks with constrained energy under jamming attacks to improve the communication quality while minimizing the total energy consumption of the network. This framework enables each relay UAV to independently select its transmit power based on historical state-related information without knowing the moving trajectory of other UAVs as well as the jammer. The location and battery level of each UAV need not be shared with other UAVs. We also propose a deep RL based anti-jamming relay approach for UAVs with portable computation equipment like Raspberry Pi to achieve higher and faster performance. We study the Nash equilibrium (NE) and the performance bounds based on the formulated power control game. Simulation results show that the proposed schemes can reduce the bit error rate (BER) and reduce energy consumption of the UAV network compared with the benchmark method.



Key wordsunmanned aerial vehicles      relay      jamming      reinforcement learning     
Received: 15 December 2020      Online: 19 August 2021
Fund:  National Natural Science Foundation of China(61971366);Fundamental Research Funds for the central universities(20720200077)
Corresponding Authors: Yi Zhang     E-mail: yizhang@xmu.edu.cn.,;lxiao@xmu.edu.cn
About author: Weihang Wang received the BS degree in electronic and information engineering from Hefei University of Technology, China in 2018. She is currently working toward the MS degree at the Department of Information and Communication Engineering, Xiamen University, China. Her research interests include network security, privacy, and wireless communications.|Zefang Lv received the BS degree from Shandong University in 2016 and the MS degree from North China Electric Power University in 2020. She is currently pursuing the PhD degree at the Department of Information and Communication Engineering, Xiamen University, China. Her research interests include network security and wireless communications.|Xiaozhen Lu received the BS degree in communication engineering from Nanjing University of Posts and Telecommunications, Nanjing, China, in 2017. She is currently pursuing the PhD degree at the Department of Information and Communication Engineering, Xiamen University, China. Her research interests include network security and wireless communications.|Yi Zhang received the BS degree in software engineering from Xiamen University in 2014. He received the MS and PhD degrees in communication engineering from Taiwan University in 2016 and 2020, respectively. He was with Quanzhou Institute of Equipment Manufacturing, Chinese Academy of Sciences from 2016 to 2017. He is currently an assistant professor at the Department of Information and Communication Engineering, Xiamen University. His research interests include mobile and wireless networking, fog/edge computing, and game theoretical models for communications networks.|Liang Xiao is currently a professor at the Department of Information and Communication Engineering, Xiamen University, China. She has served as an associate editor of IEEE Trans. Information Forensics and Security and guest editor of IEEE Journal of Selected Topics in Signal Processing. She is the recipient of the best paper award for 2016 INFOCOM Big Security WS and 2017 ICC. She received the BS degree in communication engineering from Nanjing University of Posts and Telecommunications in 2000, the MS degree in electrical engineering from Tsinghua University in 2003, and the PhD degree in electrical engineering from Rutgers University in 2009. She was a visiting professor with Princeton University, Virginia Tech, and University of Maryland, College Park.
Cite this article:

Weihang Wang,Zefang Lv,Xiaozhen Lu,Yi Zhang,Liang Xiao. Distributed reinforcement learning based framework for energy-effcient UAV relay against jamming. , 2021, 2: 150-162.

URL:

http://icn.tsinghuajournals.com/10.23919/ICN.2021.0010     OR     http://icn.tsinghuajournals.com/Y2021/V2/I2/150

Fig. 1 RL-based UAV-aided wireless relay networks against jamming attacks.
SymbolDescription
NNumber of relay UAVs
x(k)[0,X]Relay power of UAV i at a time slot k
y(k)[0,Y]Jamming power
p(k)[0,P]Transmit power of the source
r(k)RSSI of the message received by UAV i
z(k)Channel gain from the source to UAV i
h(k)Channel gain from UAV i to the destination
g^(k), g(k)Channel gain from jammer to the {destination UAV, UAV i}
b(k), b~(k){Measured, estimated} battery level of a relay UAV
ϑBattery threshold
?Minimum SINR for successful transmission
εMaximum BER for successful transmission
ϕ(k)BER of the message received by UAV i from the source
ρi(k)BER of the message received by the destination from UAV i
l(k)Jamming power received by UAV i
E(k)Energy consumption of UAV i
Table 1 List of key notations.
Fig. 2 Illustration of DREAR for UAV networks.
Fig. 3 Simulation settings for performance evaluations.
Fig. 4 Performance of the deep RL-based energy-efficient UAV relay scheme averaged over 50 episodes for the UAV network with 3 relays in the theoretical analysis scenario compared with the performance bound.
Fig. 5 Performance of the RL-based energy-efficient UAV relay schemes averaged over 50 episodes for the UAV network with 3 relays against a greedy jammer.
[1]   M. Mozaffari, W. Saad, M. Bennis, Y. H. Nam, and M. Debbah, A tutorial on UAVs for wireless networks: Applications, challenges, and open problems, IEEE Commun. Surv. Tuts., vol. 21, no. 3, pp. 2334-2360, 2019.
[2]   Y. Zeng, R. Zhang, and T. J. Lim, Wireless communications with unmanned aerial vehicles: Opportunities and challenges, IEEE Commun. Mag., vol. 54, no. 5, pp. 36-42, 2016.
[3]   L. Gupta, R. Jain, and G. Vaszkun, Survey of important issues in UAV communication networks, IEEE Commun. Surv. Tuts., vol. 18, no. 2, pp. 1123-1152, 2016.
[4]   S. C. Lv, L. Xiao, Q. Hu, X. S. Wang, C. Z. Hu, and L. M. Sun, Anti-jamming power control game in unmanned aerial vehicle networks, in Proc. IEEE Global Communications Conf., Singapore, 2017, pp. 1-6.
[5]   H. Sedjelmaci, S. M. Senouci, and N. Ansari, A hierarchical detection and response system to enhance security against lethal cyber-attacks in UAV networks, IEEE Trans. Syst., Man, Cybern.: Syst., vol. 48, no. 9, pp. 1594-1606, 2018.
[6]   S. D’Oro, E. Ekici, and S. Palazzo, Optimal power allocation and scheduling under jamming attacks, IEEE/ACM Trans. Netw., vol. 25, no. 3, pp. 1310-1323, 2017.
[7]   L. Y. Zhang, Z. Y. Guan, and T. Melodia, United against the enemy: Anti-jamming based on cross-layer cooperation in wireless networks, IEEE Trans. Wirel. Commun., vol. 15, no. 8, pp. 5733-5747, 2016.
[8]   P. Gu, C. Q. Hua, R. Khatoun, Y. Wu, and A. Serhrouchni, Cooperative antijamming relaying for control channel jamming in vehicular networks, IEEE Trans. Veh. Technol., vol. 67, no. 8, pp. 7033-7046, 2018.
[9]   L. Xiao, X. Z. Lu, D. J. Xu, Y. L. Tang, L. Wang, and W. H. Zhuang, UAV relay in VANETs against smart jamming with reinforcement learning, IEEE Trans. Veh. Technol., vol. 67, no. 5, pp. 4087-4097, 2018.
[10]   L. Xiao, D. H. Jiang, D. J. Xu, H. Z. Zhu, Y. Y. Zhang, and H. V. Poor, Two-dimensional antijamming mobile communication based on reinforcement learning, IEEE Trans. Veh. Technol., vol. 67, no. 10, pp. 9499-9512, 2018.
[11]   K. Anazawa, P. Li, T. Miyazaki, and S. Guo, Trajectory and data planning for mobile relay to enable efficient internet access after disasters, in Proc. IEEE Global Communications Conf., San Diego, CA, USA, 2015, pp. 1-6.
[12]   K. Li, W. Ni, X. Wang, R. P. Liu, S. S. Kanhere, and S. Jha, Energy-efficient cooperative relaying for unmanned aerial vehicles, IEEE Trans. Mobile Comput., vol. 15, no. 6, pp. 1377-1386, 2016.
[13]   Y. F. Chen, W. Feng, and G. Zheng, Optimum placement of UAV as relays, IEEE Commun. Lett., vol. 22, no. 2, pp. 248-251, 2018.
[14]   Y. F. Chen, N. Zhao, Z. G. Ding, and M. S. Alouini, Multiple UAVs as relays: Multi-hop single link versus multiple dual-hop links, IEEE Trans. Wirel. Commun., vol. 17, no. 9, pp. 6348-6359, 2018.
[15]   L. Zhang, L. Huang, B. Li, M. Huang, J. W. Yin, and W. M. Bao, Fast-moving jamming suppression for UAV navigation: A minimum dispersion distortionless response beamforming approach, IEEE Trans. Veh. Technol., vol. 68, no. 8, pp. 7815-7827, 2019.
[16]   Z. C. Xiao, B. Gao, S. C. Liu, and L. Xiao, Learning based power control for mmWave massive MIMO against jamming, in Proc. IEEE Global Communications Conf., Abu Dhabi, United Arab Emirates, 2018, pp. 1-6.
[17]   N. Gao, Z. J. Qin, X. J. Jing, Q. Ni, and S. Jin, Anti-intelligent UAV jamming strategy via deep Q-Networks, IEEE Trans. Commun., vol. 68, no. 1, pp. 569-581, 2020.
[18]   L. Xiao, C. X. Xie, M. H. Min, and W. H. Zhuang, User-centric view of unmanned aerial vehicle transmission against smart attacks, IEEE Trans. Veh. Technol., vol. 67, no. 4, pp. 3420-3430, 2018.
[19]   T. S. Rappaport, Wireless Communications: Principles andPractice. Englewood Cliffs, NJ, USA: Prentice-Hall, 1996.
[20]   H. van Hasselt, A. Guez, and D. Silver, Deep reinforcement learning with double Q-learning, in Proc. 30th AAAI Conf. Artificial Intelligence, Phoenix, Arizona, 2016, pp. 2094-2100.