The autofocusing properties of Hermite-Gaussian vortex beams (HGVBs) with quadratic phase modulation (QPM) are studied using a split-step Fourier transform algorithm within fractional systems characterized by variable coefficients and potentials. Initially, the autofocusing behavior of HGVBs in free space driven by QPM is analyzed, revealing that the focusing characteristics are influenced by the Lévy index, topological charge, and QPM coefficient. The propagation dynamics are then examined under varying diffraction modulations. Cosine modulation induces phenomena such as oscillation, periodic evolution with single or dual focusing, with the focusing ability and period decreasing as modulation frequency rises. Under power function modulation, the beam stabilizes after focusing, with the focusing time and stabilization structure width governed by the modulation coefficient. The influence of external potentials is explored, where a parabolic potential induces self-focusing and defocusing, with the focusing effect being dependent on both the Lévy index and parabolic coefficient. The focusing characteristics of the beam in the frequency domain are opposite to those in the spatial domain. In an annular potential, the beam exhibits periodic reflection, ultimately evolving into chaos at higher Lévy indices. The focusing dynamics is also investigated when the Lévy index is less than 1. These findings highlight new opportunities for optical switching and manipulation.

This article investigates the transmission characteristics of Laguerre–Gaussian (LG) beams under cosine modulation, power function modulation and linear modulation based on the variable coefficient fractional Schrödinger equation (FSE), respectively. In the absence of modulation, the LG beam undergoes diffraction-induced expansion as the transmission distance increases, with the degree of spreading increasing with a rising Lévy index. Under the cosine modulation, the evolution of the beam exhibits a periodic inversion, where the higher modulation frequency leads to a shorter oscillation period. The oscillation amplitude enlarges with a higher Lévy index and lower modulation frequency. When taking a power function modulation into account, the beam gradually evolves into a stable structure over propagation, with its width broadening with a growing Lévy index and modulation coefficient. In a linear modulation, the propagation of the LG beam forms a “trumpet-like” structure due to an accelerated diffraction effect. Notably, the transmission of the beam is not affected by the radial and azimuthal indices, but its ring number and phase singularity are changed correspondingly. The beam behaves in a similar evolutionary law under different modulations when the Lévy index is below 1. These findings offer valuable insights for applications in optical manipulation and communication.