Consensus in group decision-making has become a hotspot to ensure the agreement opinions of decision makers (DMs). The irrational behaviors of DMs, such as confidence, will impact the consensus results, which should be considered. In addition, the existing self-confidence level directly given by DMs rather than exacted from evaluation information may generate malicious manipulation. Furthermore, double-hierarchy hesitant fuzzy linguistic term set (DHHFLTS) is an effective tool to express the complex evaluations of DMs. In this paper, the endo-confidence of DHHFLTS to reflect confidence of DMs from the perspective of evaluation information is defined. Then, we propose a novel consensus model with endo-confidence of DMs based on DHHFLTSs. First, some improved operators of DHHFLTSs are developed. Second, the weight is determined based on both entropy and endo-confidence. Due to the fact that the consensus threshold should decrease as the endo-confidence increases, we give a novel method to obtain the consensus threshold considering endo-confidence level. Moreover, the two-stage adjustment mechanism is presented for non-consensus DMs and the selection process is constructed. Finally, an illustrative example is carried out to demonstrate the feasibility of the proposed model, and a series of comparative analysis is used to show its stability.