Faced with fluctuating network traffic and unknown underlying network traffic dynamics, developing an effective orchestration model with low network cost is still a critical issue in Network Functions Virtualization (NFV)-enabled networks. Thus we propose a Deep Reinforcement Learning based Quality of Service (QoS)/Quality of Experience (QoE)-Aware Adaptive Online Orchestration (DRL-QOR) approach to adapt to the real- time network variations. We formulate the stochastic resource optimization as a Parameterized Action Markov Decision Process (PAMDP), with QoE and specific QoS requirements as key factors in formulating the reward function, aiming to maximize QoE while satisfying QoS constraints. Then we propose DRL-QOR to solve the Non-deterministic Polynomial hard (NP-hard) problem with consideration of improving the long-term profits, where deep neural network combinatorial optimization theory is extended under the constraints of the binary integer programming model. Extensive experimental results in real USANET topology demonstrate that our proposed DRL-QOR converges fast during the training process. Compared with other benchmarks that only consider the current system performance, it shows good performance in QoE provisioning and QoS requirements maintenance for orchestrating SFCs.

In software defined network/network function virtualization (SDN/NFV)-enabled cloud environment, cloud services can be implemented as service function chains (SFCs), which consist of a series of ordered virtual network functions. However, due to fluctuations of cloud traffic and without knowledge of cloud computing network configuration, designing SFC optimization approach to obtain flexible cloud services in dynamic cloud environment is a pivotal challenge. In this paper, we propose a fault tolerance oriented SFC optimization approach based on deep reinforcement learning. We model fault tolerance oriented SFC elastic optimization problem as a Markov decision process, in which the reward is modeled as a weighted function, including minimizing energy consumption and migration cost, maximizing revenue benefit and load balancing. Then, taking binary integer programming model as constraints of quality of cloud services, we design optimization approaches for single-agent double deep Q-network (SADDQN) and multi-agent DDQN (MADDQN). Among them, MADDQN decentralizes training tasks from control plane to data plane to reduce the probability of single point of failure for the centralized controller. Experimental results show that the designed approaches have better performance. MADDQN can almost reach the upper bound of theoretical solution obtained by assuming a prior knowledge of the dynamics of cloud traffic.

Network function virtualization (NFV) is considered to be a promising paradigm for reducing network infrastructure cost. However, the mapping from network physical resources to service function chain (SFC), which is an ordered sequence of virtual network functions, is challenging, especially when a dynamic wireless network is involved. In this article, we propose a novel lightweight SFC embedding framework based on reinforcement learning in the NFV-enabled wireless network for reducing the end-to-end wired and wireless delay and leveraging the SFC acceptance ratio. We model the embedding problem as a Markov decision process, in which the reward is modeled as an adjustable weighted function, including combined delay (wired and wireless delay) and remaining resource. Accordingly, a lightweight and adjustable $Q$ -learning-based SFC embedding strategy is proposed. To validate the effectiveness of our proposed framework, extensive simulations are performed by using real data traces and compared with a classical genetic algorithm (GA) and a cluster-based method MAPLE. The results of the simulation show that the proposed framework is efficient and performs better than both GA and MAPLE in terms of delay and service acceptance probability. The prototype of the framework is also built for demonstrating the feasibility of the proposed framework.

Software defined network (SDN) is a promising technology which can reduce network management complexity through the decoupling of the control plane and data plane. Due to large number of switches in the data plane, distributed and multiple controllers are necessary in the control plane for managing the switches. The switch controller mapping strategy for identifying the mapping relationships between the switch and controller is crucial in order to optimize the network performance. Considering the dynamics of the network behavior, it is quite important and challenging to develop models to reflect the network topology dynamics and to propose method for solving the long-term network performance optimization. Inspired by the recent advances in Artificial Intelligence (AI), in this paper, we propose a Deep Reinforcement Learning (DRL) based strategy for solving the switch controller mapping problem. A DRL based mapping strategy is proposed, in which Markov Decision Process (MDP) formulation is devised and Deep Q-network (DQN) is proposed to achieve the maximization of long-term system performance by leveraging network latency, load balancing and system stability. Extensive simulations show that the DQN based algorithm can achieve the best system stability results while maintaining moderate switch controller latency and system equilibrium performance comparing with the optimization which only considers current system performance for switch controller mapping decision, and the optimization approaches which generate mapping decisions purely based on latency or load balancing separately.

Owing to software defined network function virtualization (SDNFV), network services can be implemented as service function chains (SFCs) in SDNFV-enabled Cloud Computing. SFCs consist of a series of ordered virtual network functions (VNFs). Due to the dynamic of underlying network state and the unpredictability of network traffic, the traditional SFC orchestrating (SFCO) approaches based on centralized placement and single-path routing lead to low availability of network resources, making it difficult to effectively manage and utilize complex and heterogeneous network resources. To address the above challenges, we propose a queue-aware SFCs orchestrating and adaptive parallel traffic scheduling optimization approach. First, the SFCO problem is modeled as a stochastic optimization problem, and the Lyapunov optimization theory is used to transform and decompose the SFCO problem to decouple the time coupling of optimal decision-making. An automatic decentralized algorithm based on queue model is proposed to orchestrate SFCs using information of local and its immediate one-hop neighbors. Furthermore, an adaptive parallel traffic scheduling optimization algorithm based on deep reinforcement learning is proposed, according to the decision output of the distributed SFC algorithm and current network state, network traffic is allocated to multiple paths for parallel transmission, which improves the availability of network resources and network performance. Experimental results show that, compared with the benchmarks, the average queue depth of the designed approach is reduced by $42.18\% \sim 69.97\%$ , the average cost of the designed approach is reduced by $16.1\% \sim 55.6\%$ , the average throughput is improved by $2.41\%\sim 10.07\%$ , the average link resource utilization rate is improved by about $6.9\%\sim 28.3\%$ , the average round-trip delay is shortened by $17.1\%\sim 24.1\%$ , and the average packet loss rate is reduced by $39.4\%\sim 51.7\%$ .

Network functions virtualization (NFV) solves the problem of tight coupling between network functions and hardware. While the Service Function Chaining (SFC) is the basic business model and provides customized services for network functions in NFV. However, the existing network function deployment and migration mechanisms are relatively static and rigid. For example, problems such as network load imbalance, deployment nodes or virtual network function instances failing, and virtual function elastic scaling cannot be performed. Aiming at the above problems, this paper proposes the decision-making optimization algorithm for virtual network function migration based on deep reinforcement learning. The algorithm takes network and node resources as constraints, establishes a migration decision model including state space, behavior space, and reward mechanism, and realizes flexible adaptation of virtual network functions. In the algorithm, migration decision-making based on Deep Q Network (DQN) includes offline training and online decision-making. In the offline training stage, DQN is used to train the strategy network model. The online migration stage is based on the trained strategy network model, which uses the current network state as the model input to dynamically generate the migration strategy. Finally, the network performs the migration of virtual network functions online. This algorithm optimizes the node load and solves the problem of service interruption caused by the excessive load of a single node, failure of links, and virtual function instances in the network. Finally, we build a practical prototype system and demonstrate the performance of the proposed DQN approach. According to the network status, the prototype system realizes the on-demand dynamic migration of virtual network functions. And the experimental results verify that the proposed elastic adaptation mechanism can effectively solve the migration problems, improve the quality of service, and enhance the scalability and flexibility of the network.