Interest in deploying deep reinforcement learning (DRL) models on low-power edge devices, such as Autonomous Mobile Robots (AMRs) and Internet of Things (IoT) devices, has seen a significant rise due to the potential of performing real-time inference by eliminating the latency and reliability issues incurred from wireless communication and the privacy benefits of processing data locally. Deploying such energy-intensive models on power-constrained devices is not always feasible, however, which has led to the development of model compression techniques that can reduce the size and computational complexity of DRL policies. Policy distillation, the most popular of these methods, can be used to first lower the number of network parameters by transferring the behavior of a large teacher network to a smaller student model before deploying these students at the edge. This works well with deterministic policies that operate using discrete actions. However, many real-world tasks that are power constrained, such as in the field of robotics, are formulated using continuous action spaces, which are not supported. In this work, we improve the policy distillation method to support the compression of DRL models designed to solve these continuous control tasks, with an emphasis on maintaining the stochastic nature of continuous DRL algorithms. Experiments show that our methods can be used effectively to compress such policies up to 750% while maintaining or even exceeding their teacher’s performance by up to 41% in solving two popular continuous control tasks.

Deploying deep reinforcement learning on resource-constrained devices remains a significant challenge due to the energy-intensive nature of the sequential decision-making process. Model compression can reduce the spatial (e.g. storage, memory) requirements of a policy network, but this does not always translate to a proportional increase in inference speed and computational efficiency. We introduce a novel temporal compression paradigm that improves the efficiency more directly, by reducing the number of predictions needed to complete a task. This method, based on policy distillation, allows a student model to learn when a change of action will be required by observing sequences of identical actions in the trajectories of an existing teacher model. At each decision, the student can then predict both an action and how many times to perform this action consecutively. This approach allows any existing policy for discrete action spaces to be optimized for energy efficiency through both spatial and temporal compression simultaneously. Experiments on devices ranging from a microcontroller and smartphone processor to a data centre GPU show how this method can decrease the average time it takes to predict an action by up to 13.5 times, compared to 4 times through spatial compression alone, while maintaining a similar average return as the original teacher. In practice, this allows complex models to be deployed on ultra-low-power devices, enabling them to conserve energy by remaining in sleep mode for longer periods, and still achieve high runtime and task performance.

Deep Reinforcement Learning (DRL) has achieved remarkable success across various domains, yet deploying these powerful models on resource-constrained edge devices remains challenging due to their high computational requirements. To address these challenges, model compression techniques have been developed to reduce the size and computational cost of neural networks. Compression of DRL policy networks is still relatively understudied, however, with most research adapting supervised learning methods without fully leveraging DRL's unique characteristics. This gap limits the potential for smaller, more efficient DRL agents to be deployed at the edge. Moreover, not all DRL policies are compatible with existing compression methods, necessitating the development of new techniques tailored to their specific requirements. This work hypothesizes that, by understanding and exploiting the unique properties of DRL models, novel compression methods can be developed that achieve higher compression rates and enable the compression of a wider range of models than previously possible. Through a series of methodological contributions, we demonstrate that compression ratios exceeding a thousand times are possible in extreme cases, without significant loss of effectiveness. We explored several dimensions of policy compression, including the reduction of parameter count, precision, and the frequency of decision-making. First, we introduce a novel distillation loss for actor-critic models, incorporating the teacher's state-value predictions as an auxiliary task, to enable student models to learn a more informed internal representation. To facilitate low-precision deployment, we propose Quantization-aware Policy Distillation (QPD), an algorithm that enables stable training of 8-bit integer policy networks. Additionally, we developed new ways to compress policies for continuous control tasks, highlighting how maintaining the stochastic nature of the original policy is crucial for retaining the most effective policies after distillation. We also investigate the impact of pruning on policy adaptability, finding that while pruning can impair generalization, informed pre-training and task-relevant pruning can mitigate these effects. Temporal distillation is then introduced, a novel compression paradigm that reduces the computational cost of policy execution by enabling the student to predict sequences of repeated actions with a single decision. Finally, for partially observable environments, we propose a method for compressing RNN-based policies by preserving the temporal dependencies in the training data used for distillation. Overall, our contributions advance the feasibility of deploying complex DRL models on highly resource-constrained edge devices. This unlocks new possibilities for real-time sequential decision-making in domains such as autonomous robotics and network management systems.