Background: In the second-line treatment of advanced pancreatic cancer (APC), there is only one approved regimen based on the phase III NAPOLI-1 trial. However, for patients progressing after Nab-paclitaxel and Gemcitabine (Nab-P/Gem) or Nab-P combinations, second-line treatment were very limited. Methods: This is a retrospective single-center analysis of patients. Our aim was to determine the effectiveness and tolerability of a novel regimen, gemcitabine plus Anlotinib and anti-PD1, in APC patients and to compare it with oxaliplatin, irinotecan, leucovorin, and fluorouracil (FOLFIRINOX) in the second-line setting who have failed on the first-line Nab-P combinations. Results: In total, twenty-three patients received Gemcitabine plus Anlotinib and anti-PD1 in the second-line, 28 patients were treated with FOLFORINOX. There was no significant difference in overall survival (OS) or progression free survival (PFS) for either of the two sequences (p > 0.05). Patients who received Gemcitabine plus Anlotinib and anti-PD1 had a median PFS of 4.0 months (95% CI: 1.1–6.9) versus 3.5 months (95% CI 1.8–5.2) in FOLFORINOX group (p = 0.953). The median OS of Gemcitabine plus Anlotinib and anti-PD1 was 9.0 months (95% CI: 4.0–13.7) and 8.0 months (95% CI: 5.5–10.5) in FOLFORINOX group (p = 0.373). Grade ≥3 treatment-emergent adverse events (AEs) occurred for 13% of patients with Gemcitabine plus Anlotinib and anti-PD1 and 40% for FOLFORINOX. Conclusion: Our data confirms the effectiveness of Gemcitabine plus Anlotinib and anti-PD1 as a well-tolerated regimen in the second-line treatment of APC and extends available data on its use as a second-line treatment option when compared with FOLFIRINOX.