(1) Background: desmoid tumors (DTs) are common in patients with familial adenomatous polyposis (FAP). An active surveillance approach has been recently proposed as a valuable alternative to immediate treatment in some patients. However, no clear indication exists on which patients are suitable for active surveillance, how to establish the cut-off for an active treatment, and which imaging technique or predictive factors should be used during the surveillance period. (2) Results: we retrospectively analyzed 13 FAP patients with DTs. A surveillance protocol consisting of scheduled follow-up evaluations depending on tumor location and tissue thickening, abdominal computed tomography (CT) scan/Magnetic resonance imaging (MRI) allowed prompt intervention in 3/11 aggressive intra-abdominal DTs, while sparing further interventions in the remaining cases, despite worrisome features detected in three patients. Moreover, we identified a possible predictive marker of tumor aggressiveness, i.e., the “average monthly growth rate” (AMGR), which could distinguish patients with very aggressive/life-threatening tumor behavior (AMGR > 0.5) who need immediate active treatment, from those with stable DTs (AMGR < 0.1) in whom follow-up assessments could be delayed. (3) Conclusion: surveillance protocols may be a useful approach for DTs. Further studies on larger series are needed to confirm the usefulness of periodic CT scan/MRI and the value of AMGR as a prognostic tool to guide treatment strategies.