In this paper, we investigate the mechanism of atomic force microscopy in tapping mode (AFM-TM) under the Casimir and van der Waals (VdW) forces. The dynamic behavior of the system is analyzed through a nonlinear dimensionless mathematical model. Numerical tools as Poincaré maps, Lyapunov exponents, and bifurcation diagrams are accounted for the analysis of the system. With that, the regions in which the system presents chaotic and periodic behaviors are obtained and investigated. Moreover, the fractional calculus is introduced into the mathematical model, employing the Riemann-Liouville kernel discretization in the viscoelastic term of the system. The 0-1 test is implemented to analyze the new dynamics of the system, allowing the identification of the chaotic and periodic regimes of the AFM system. The dynamic results of the conventional (integer derivative) and fractional models reveal the need for the application of control techniques such as Optimum Linear Feedback Control (OLFC), State-Dependent Riccati Equations (SDRE) by using feedback control, and the Time-Delayed Feedback Control. The results of the control techniques are efficient with and without the fractional-order derivative.