Machine learning techniques, particularly those based on neural networks, are always more often used at the edge of the network by IoT nodes. Unfortunately, the computation capabilities demanded by those applications, together with their energy efficiency related constraints, exceed those exposed by embedded general-purpose processors. For this reason, the use of domain-specific hardware accelerators is considered the most viable solution to the unsustainable “Turing tariff” of general-purpose hardware. Starting from the observation that memory and communication traffic account for a large fraction of the overall latency and energy in deep neural network (DNN) inferences, this paper proposes a new compression technique aimed at (i) reducing the memory footprint for storing the model parameters of a DNN, and (ii) improving DNN inference latency and energy on resource-constrained IoT devices. The proposed compression technique, namely, LineCompress, is applied on a set of representative convolutional neural networks (CNNs) for object recognition mapped on a state-of-the-art domain-specific hardware accelerator targeted for resource-constrained IoT devices. We show that, on average, 7.4× memory footprint reduction can be obtained, thus reducing the memory and communication traffic that result to 77% and 87% inference latency and energy reduction, respectively, trading-off efficiency vs. accuracy.

The need to efficiently execute different Deep Neural Networks (DNNs) on the same computing platform, coupled with the requirement for easy scalability, makes Multi-Chip Module (MCM)-based accelerators a preferred design choice. Such an accelerator brings together heterogeneous sub-accelerators in the form of chiplets, interconnected by a Network-on-Package (NoP). This paper addresses the challenge of selecting the most suitable sub-accelerators, configuring them, determining their optimal placement in the NoP, and mapping the layers of a predetermined set of DNNs spatially and temporally. The objective is to minimise execution time and energy consumption during parallel execution while also minimising the overall cost, specifically the silicon area, of the accelerator. This paper presents MOHaM, a framework for multi-objective hardware-mapping co-optimisation for multi-DNN workloads on chiplet-based accelerators. MOHaM exploits a multi-objective evolutionary algorithm that has been specialised for the given problem by incorporating several customised genetic operators. MOHaM is evaluated against state-of-the-art Design Space Exploration (DSE) frameworks on different multi-DNN workload scenarios. The solutions discovered by MOHaM are Pareto optimal compared to those by the state-of-the-art. Specifically, MOHaM-generated accelerator designs can reduce latency by up to 96% and energy by up to 96.12%.

Currently, there is a growing trend of outsourcing the execution of DNNs to cloud services. For service providers, managing multitenancy and ensuring high-quality service delivery, particularly in meeting stringent execution time constraints, assumes paramount importance, all while endeavoring to maintain cost-effectiveness. In this context, the utilization of heterogeneous multi-accelerator systems becomes increasingly relevant. This paper presents RELMAS, a low-overhead deep reinforcement learning algorithm designed for the online scheduling of DNNs in multi-tenant environments, taking into account the dataflow heterogeneity of accelerators and memory bandwidths contentions. By doing so, service providers can employ the most efficient scheduling policy for user requests, optimizing Service-Level-Agreement (SLA) satisfaction rates and enhancing hardware utilization. The application of RELMAS to a heterogeneous multi-accelerator system composed of various instances of Simba and Eyeriss sub-accelerators resulted in up to a 173% improvement in SLA satisfaction rate compared to state-of-the-art scheduling techniques across different workload scenarios, with less than a 1.5% energy overhead.

Mapping a deep neural network (DNN) layer onto domain-specific accelerators can require an intractable number of choices regarding loop factorization, ordering, and spatial unrolling. Determining the optimal mapping that achieves the best figures in terms of latency and energy efficiency can be difficult due to the vast number of possible candidates that need to be exhaustively evaluated. Many techniques have been recently proposed for fast and efficient mapping space exploration; some of them adopt a black-box optimization approach, others make assumptions on the underlying accelerator memory hierarchy or require time-consuming model retraining. We propose an integer linear programming (ILP) approach and formulate a mathematical model, namely LEMON, that takes into account number of accesses to each buffer, energy costs and buffer bandwidths in the accelerator and is flexible enough to work with different memory hierarchies. Compared with state-of-the-art techniques, LEMON achieves up to 83% energy-delay product reduction when compared to another ILP-based approach (CoSA) and 27% when compared to a genetic algorithm approach (GAMMA).

Deep neural network (DNN) hardware accelerators enable the execution of complex DNN inferences on resource-constrained IoT devices. Inference performance and energy figures depend on how the DNN layers are mapped into the accelerator and how the architecture of the accelerator fits the variety of layers’ shapes of the actual DNN. The mapping determines the execution order of the operations, both temporally and spatially. Thus, selecting the best mapping that allows fitting the DNN model to the specific accelerator is of paramount importance to meet the strong constraints imposed by resource-scarce IoT platforms. Although several mapping space exploration techniques have been proposed in the literature, they are focused on determining the best mapping for a given layer, for a given architecture, and for optimizing a single objective. This article largely extends the scope of the exploration by considering the huge design space spanned by mapping related and architectural parameters, considering all the layers of the DNN, and optimizing multiple objectives simultaneously. We present EPOCA, end-to-end Pareto optimization of DNN accelerators, whose goal is to determine the accelerator’s architecture and the mapping for each layer that optimizes end-to-end and in a multiobjective fashion a set of conflicting design criteria. We assess EPOCA on different DNN models on a parameterized hardware accelerator designed for IoT applications and compare them with a state-of-the-art mapping space explorer, considering the area, inference latency, and inference energy as optimization metrics. We show that the set of Pareto solutions found by EPOCA provides the designer with a range of choices from which to select the best tradeoff with respect to the specific application.