The reliability of neural network (NN) accelerators is one key to ensuring inference accuracy. Relying solely on the robustness of NN can only tolerate transient faults, and once the circuit encounters permanent faults, it will lead to a serious accuracy drop. We, by employing processing elements (PEs) checking and re-scheduling of threads of functional units, propose a new fault-tolerant mechanism named PerFT-N for neural processing units (NPUs). Compared to previous NPU fault-tolerant methods, PerFT-N recovers the execution facing permanent faults with minimal hardware overhead. Specifically, we utilize the existing resources to achieve fault detection and localization, while achieving recovery by re-linking unfailed threads. Furthermore, an instruction-based programmable permanent fault emulation scheme is deployed on the FPGA platform for fast verification. Experimental results demonstrate that the PerFT-N architecture works in the extreme case with 98.5% of failed functional units while incurring the physical overheads of 2.7% in area and 3.6% in power consumption under 40nm SMIC standard cell library.