Wavelength-selective thermal emitters (WS-TEs) have been frequently designed to achieve desired target emissivity spectra, as a typical emissivity engineering, for broad applications such as thermal camouflage, radiative cooling, and gas sensing, etc. However, previous designs require prior knowledge of materials or structures for different applications and the designed WS-TEs usually vary from applications to applications in terms of materials and structures, thus lacking of a general design framework for emissivity engineering across different applications. Moreover, previous designs fail to tackle the simultaneous design of both materials and structures, as they either fix materials to design structures or fix structures to select suitable materials. Herein, we employ the deep Q-learning network algorithm, a reinforcement learning method based on deep learning framework, to design multilayer WS-TEs. To demonstrate the general validity, three WS-TEs are designed for various applications, including thermal camouflage, radiative cooling and gas sensing, which are then fabricated and measured. The merits of the deep Q-learning algorithm include that it can (1) offer a general design framework for WS-TEs beyond one-dimensional multilayer structures; (2) autonomously select suitable materials from a self-built material library and (3) autonomously optimize structural parameters for the target emissivity spectra. The present framework is demonstrated to be feasible and efficient in designing WS-TEs across different applications, and the design parameters are highly scalable in materials, structures, dimensions, and the target functions, offering a general framework for emissivity engineering and paving the way for efficient design of nonlinear optimization problems beyond thermal metamaterials.