(1) Background: Mutation-specific T cell receptor (TCR)-based adoptive T cell therapy represents a truly tumor-specific immunotherapeutic strategy. However, isolating neoepitope-specific TCRs remains a challenge. (2) Methods: We investigated, side by side, different TCR repertoires—patients’ peripheral lymphocytes (PBLs) and tumor-infiltrating lymphocytes (TILs), PBLs of healthy donors, and a humanized mouse model—to isolate neoepitope-specific TCRs against eight neoepitope candidates from a colon cancer and an ovarian cancer patient. Neoepitope candidates were used to stimulate T cells from different repertoires in vitro to generate neoepitope-specific T cells and isolate the specific TCRs. (3) Results: We isolated six TCRs from healthy donors, directed against four neoepitope candidates and one TCR from the murine T cell repertoire. Endogenous processing of one neoepitope, for which we isolated one TCR from both human and mouse-derived repertoires, could be shown. No neoepitope-specific TCR could be generated from the patients’ own repertoire. (4) Conclusion: Our data indicate that successful isolation of neoepitope-specific TCRs depends on various factors such as the heathy donor’s TCR repertoire or the presence of a tumor microenvironment allowing neoepitope-specific immune responses of the host. We show the advantage and feasibility of using healthy donor repertoires and humanized mouse TCR repertoires to generate mutation-specific TCRs with different specificities, especially in a setting when the availability of patient material is limited.