Sub-seafloor sediment is the largest microbial habitat on Earth. The study of microbes in sub-seafloor sediment is largely limited by the technical challenge of acquiring ambient microbial DNA because of sediment heterogeneity. Changes in the extraction method, even just by one step, can affect the extraction yields for complicated sediment samples. In this work, sub-seafloor sediment samples from the Baltic Sea with high organic carbon content were used to evaluate the influence of different grinding beads on DNA extraction. We found that the grinding beads can affect the DNA extraction from the organic-matter- and biosiliceous-clay-rich samples. A mixture of 0.5-mm and 0.1-mm grinding beads exhibited higher DNA yields and recovered more unique taxa than other bead combinations, such as Stenotrophomonas from Gammaproteobacteria and Leptotrichia from Fusobacteria; therefore, these beads are more suitable than the others for DNA extraction from the samples used in this study. This advantage might be magnified in samples with high biomass. On the contrary, the use of only small beads might lead to underestimation for certain Gram-positive strains. Overall, the discovery of abundant widespread deep biosphere clades in our samples indicated that our optimized DNA extraction method successfully recovered the in situ microbial community.