Litchi pomace, a by-product of litchi processing, is rich in dietary fiber. Soluble and insoluble dietary fibers were extracted from litchi pomace, and insoluble dietary fiber was modified by ultrasonic enzymatic treatment to obtain modified soluble and insoluble dietary fibers. The structural, physicochemical, and functional properties of the dietary fiber samples were evaluated and compared. It was found that all dietary fiber samples displayed typical polysaccharide absorption spectra, with arabinose being the most abundant monosaccharide component. Soluble dietary fibers from litchi pomace were morphologically fragmented and relatively smooth, with relatively high swelling capacity, whereas the insoluble dietary fibers possessed wrinkles and porous structures on the surface, as well as higher water holding capacity. Additionally, soluble dietary fiber content of litchi pomace was successfully increased by 6.32 ± 0.14% after ultrasonic enzymatic modification, and its arabinose content and apparent viscosity were also significantly increased. Further, the soluble dietary fibers exhibited superior radical scavenging ability and significantly stimulated the growth of probiotic bacterial species. Taken together, this study suggested that dietary fiber from litchi pomace could be a promising ingredient for functional foods industry.