Deep neural networks (DNNs) typically have a single exit point that makes predictions by running the entire stack of neural layers. Since not all inputs require the same amount of computation to reach a confident prediction, recent research has focused on incorporating multiple “exits” into the conventional DNN architecture. Early-exit DNNs are multi-exit neural networks that attach many side branches to the conventional DNN, enabling inference to stop early at intermediate points. This approach offers several advantages, including speeding up the inference process, mitigating the vanishing gradients problems, reducing overfitting and overthinking tendencies. It also supports DNN partitioning across devices and is ideal for multi-tier computation platforms such as edge computing. This article decomposes the early-exit DNN architecture and reviews the recent advances in the field. The study explores its benefits, designs, training strategies, and adaptive inference mechanisms. Various design challenges, application scenarios, and future directions are also extensively discussed.

In computer vision applications, mobile devices can transfer the inference of Convolutional Neural Networks (CNNs) to the cloud due to their computational restrictions. Nevertheless, besides introducing more network load concerning the cloud, this approach can make unfeasible applications that require low latency. A possible solution is to use CNNs with early exits at the network edge. These CNNs can pre-classify part of the samples in the intermediate layers based on a confidence criterion. Hence, the device sends to the cloud only samples that have not been satisfactorily classified. This work evaluates the performance of these CNNs at the computational edge, considering an object detection application. For this, we employ a MobiletNetV2 with early exits. The experiments show that the early classification can reduce the data load and the inference time without imposing losses to the application performance.

Deep Neural Networks (DNNs) have demonstrated exceptional performance in diverse tasks. However, deploying DNNs on resource-constrained devices presents challenges due to energy consumption and delay overheads. To mitigate these issues, early-exit DNNs (EE-DNNs) incorporate exit branches within intermediate layers to enable early inferences. These branches estimate prediction confidence and employ a fixed threshold to determine early termination. Nonetheless, fixed thresholds yield suboptimal performance in dynamic contexts, where context refers to distortions caused by environmental conditions, in image classification, or variations in input distribution due to concept drift, in NLP. In this article, we introduce Upper Confidence Bound in EE-DNNs (), an online algorithm that dynamically adjusts early exit thresholds based on context. leverages confidence levels at intermediate layers and learns without the need for true labels. Through extensive experiments in image classification and NLP, we demonstrate that achieves logarithmic regret, converging after just a few thousand observations across multiple contexts. We evaluate for image classification and text mining. In the latter, we show that can reduce cumulative regret and lower latency by approximately 10%–20% without compromising accuracy when compared to fixed threshold alternatives. Our findings highlight as an effective method for enhancing EE-DNN efficiency.

Redes Neurais Profundas com Saídas Antecipadas (EE-DNNs) inserem ramos laterais que permitem a inferência local quando a confiança ultrapassa um limiar pré-definido, reduzindo a dependência da nuvem. No entanto, um limiar fixo não se adapta às variações contextuais do mundo real. Este trabalho investiga a adaptação dinâmica dos limiares utilizando algoritmos de multi-armed bandits (MABs). Além disso, um buffer de entradas finito é introduzido para equilibrar o compromisso entre acurácia e latência, considerando a confiança e o tamanho da fila. Os resultados experimentais demonstram que os limiares ajustados por MABs convergem rapidamente em diversos contextos, enquanto o buffer garante um equilíbrio eficiente entre acurácia e latência.

As redes neurais profundas (DNNs) são sensíveis a imagens com distorção, tendo sua acurácia reduzida. Este trabalho analisa como soluções de DNNs com saídas antecipadas (EE-DNNs) podem resolver esse problema. As EE-DNNs possuem ramos laterais inseridos em suas camadas intermediárias para classificar antecipadamente amostras na borda e evitar envios para a nuvem. Além disso, múltiplos ramos laterais podem compor um comitê e produzir coletivamente uma inferência mais precisa. Os resultados, em termos de acurácia, mostram que EE-DNNs e o comitê são tão sensíveis quanto DNNs convencionais. Entretanto, dado o uso da borda, essas abordagens conseguem reduzir o tempo de inferência e o número de operações para classificar imagens.