The purpose of this study was to explore the effects of xylose with different polymerizations on growth performance, intestinal barrier function, and gut microbial composition in weaned piglets. A total of 144 weaned piglets were assigned to 3 dietary treatments in a completely randomized design according to their body weight and sex. Dietary treatments included a corn-soybean meal basal diet (CON) and 2 additional diets formulated with 1% arabinoxylan (AX) and 1% xylo-oligosaccharide (XOS), respectively. Results showed that dietary supplementation of XOS or AX reduced diarrhea incidence of weaned piglets compared with the CON group (p < 0.05). XOS or AX increased the ileal villus height and intestinal activity of antioxidases in weaned piglets compared with the CON group (p < 0.05). XOS or AX reduced the ileal and colonic IL-6 content and increased the colonic sIgA and IL-10 concentrations in weaned piglets compared with the CON group (p < 0.05). XOS or AX increased the total organic acids concentrations in the ileum and in vitro fermentation (p < 0.05). XOS increased the abundance of Lactobacillus and Bifidobacterium in the ileal digesta (p < 0.05), while AX increased the population of Lactobacillus in the ileal digesta and the abundance of Bifidobacterium in the colonic digesta of weaned piglets (p < 0.05). In conclusion, both XOS and AX reduce diarrhea incidence and improve antioxidant capacity, immune function, and populations of beneficial bacteria, while microbial fermentation of XOS with a lower polymerization and molecular mass can produce more organic acids and an increased abundance of Lactobacillus and Bifidobacterium in the upper gut of weaned pigs compared with AX.