Reinforcement learning has recently been studied in various fields and also used to optimally control IoT devices supporting the expansion of Internet connection beyond the usual standard devices. In this paper, we try to allow multiple reinforcement learning agents to learn optimal control policy on their own IoT devices of the same type but with slightly different dynamics. For such multiple IoT devices, there is no guarantee that an agent who interacts only with one IoT device and learns the optimal control policy will also control another IoT device well. Therefore, we may need to apply independent reinforcement learning to each IoT device individually, which requires a costly or time-consuming effort. To solve this problem, we propose a new federated reinforcement learning architecture where each agent working on its independent IoT device shares their learning experience (i.e., the gradient of loss function) with each other, and transfers a mature policy model parameters into other agents. They accelerate its learning process by using mature parameters. We incorporate the actor–critic proximal policy optimization (Actor–Critic PPO) algorithm into each agent in the proposed collaborative architecture and propose an efficient procedure for the gradient sharing and the model transfer. Using multiple rotary inverted pendulum devices interconnected via a network switch, we demonstrate that the proposed federated reinforcement learning scheme can effectively facilitate the learning process for multiple IoT devices and that the learning speed can be faster if more agents are involved.
Recently, with the advent of various Internet of Things (IoT) applications, a massive amount of network traffic is being generated. A network operator must provide different quality of service, according to the service provided by each application. Toward this end, many studies have investigated how to classify various types of application network traffic accurately. Especially, since many applications use temporary or dynamic IP or Port numbers in the IoT environment, only payload-based network traffic classification technology is more suitable than the classification using the packet header information as well as payload. Furthermore, to automatically respond to various applications, it is necessary to classify traffic using deep learning without the network operator intervention. In this study, we propose a traffic classification scheme using a deep learning model in software defined networks. We generate flow-based payload datasets through our own network traffic pre-processing, and train two deep learning models: 1) the multi-layer long short-term memory (LSTM) model and 2) the combination of convolutional neural network and single-layer LSTM models, to perform network traffic classification. We also execute a model tuning procedure to find the optimal hyper-parameters of the two deep learning models. Lastly, we analyze the network traffic classification performance on the basis of the F1-score for the two deep learning models, and show the superiority of the multi-layer LSTM model for network packet classification.
Nowadays, Reinforcement Learning (RL) is applied to various real-world tasks and attracts much attention in the fields of games, robotics, and autonomous driving. It is very challenging and devices overwhelming to directly apply RL to real-world environments. Due to the reality gap simulated environment does not match perfectly to the real-world scenario and additional learning cannot be performed. Therefore, an efficient approach is required for RL to find an optimal control policy and get better learning efficacy. In this paper, we propose federated reinforcement learning based on multi agent environment which applying a new federation policy. The new federation policy allows multi agents to perform learning and share their learning experiences with each other e.g., gradient and model parameters to increase their learning level. The Actor-Critic PPO algorithm is used with four types of RL simulation environments, OpenAI Gym's CartPole, MoutainCar, Acrobot, and Pendulum. In addition, we did real experiments with multiple Rotary Inverted Pendulum (RIP) to evaluate and compare the learning efficiency of the proposed scheme with both environments.
1 January 2022 in Computer Science and Information Systems
This paper proposes a deep recurrent neural network (RNN)-based traffic classification scheme (deep RNN-TCS) for classifying applications from traffic patterns in a hybrid edge computing and cloud computing architecture.We can also classify traffic from a cloud server, but there will be a time delay when packets transfer to the server. Therefore, the traffic classification is possible almost in realtime when it performed on edge computing nodes. However, training takes a lot of time and needs a lot of computing resources to learn traffic patterns. Therefore, it is efficient to perform training on cloud server and to perform serving on edge computing node. Here, a cloud server collects and stores output labels corresponding to the application packets. Then, it trains those data and generates inferred functions. An edge computation node receives the inferred functions and executes classification. Compared to deep packet inspection (DPI), which requires the periodic verification of existing signatures and updated application information (e.g., versions adding new features), the proposed scheme can classify the applications in an automated manner. Also, deep learning can automatically make classifiers for traffic classification when there is enough data. Specifically, input features and output labels are defined for classification as traffic packets and target applications, respectively, which are created as two-dimensional images. As our training data, traffic packets measured at Universitat Politecnica de Catalunya Barcelonatech were utilized. Accordingly, the proposed deep RNN-TCS is implemented using a deep long short-term memory system. Through extensive simulation-based experiments, it is verified that the proposed deep RNN-TCS achieves almost 5% improvement in accuracy (96% accuracy) while operating 500 times faster (elapsed time) compared to the conventional scheme.
26 September 2023 in IEEE Internet of Things Journal
Virtual network embedding (VNE) is a promising technique enabling 5G networks to satisfy the given requirements of each service via network virtualization (NV). For better performance of the embedding algorithm, it is necessary to automatically detect the network status and provide an optimal embedding decision. However, existing virtual network embedding (VNE) algorithms disregard the long-term effect by focusing on selecting only one virtual network request (VNR) from the waiting queue, without considering all waiting virtual network requests concurrently. In this study, we propose a hierarchical cooperative multiagent reinforcement learning (MARL) algorithm to optimize the VNE problem by maximizing average revenue, minimizing average cost, and also improving the request acceptance ratio. The proposed algorithm applies two acrshort RL algorithms: 1) two-level hierarchical RL (HRL) to efficiently solve the problem by dividing it into subproblems and 2) multiagent-based cooperative acrshort RL to improve algorithm performance through the cooperation of multiple agents. In order to evaluate and analyze the proposed scheme from the long-term perspective, four performance parameters are evaluated: 1) revenue; 2) cost; 3) revenue-to-cost ratio; and 4) acceptance ratio. The simulation results demonstrate that the proposed VNE algorithm based on hierarchical and acrshort MARL outperforms the existing acrshort RL-based approaches.