Polystyrene-organo-montmorillonite (PS-OMMT) nanocomposite particles were prepared via emulsion polymerization of styrene in the presence of montmorillonite modified with an anionic surfactant, sodium lauryl sulfonate (SLS), and its tribological properties as an additive to polyalphaolefin (PAO) were tested. The results of Fourier transform infrared spectroscopy (FT-IR), X-ray diffraction (XRD) and thermogravimetric analysis (TGA) showed that SLS molecules resided in the montmorillonite (MMT) interlayer space. The effects of OMMT on the morphology and properties of the nanocomposites were also investigated. Gel permeation chromatography (GPC) and dynamic light scattering (DLS) demonstrate that the presence of OMMT can effectively reduce the average molecular weight and average particle size of PS. XRD and transmission electron microscopy (TEM) of the PS-OMMT nanocomposites indicate that exfoliated and intercalated structures form and that the MMT layers either are partly embedded inside the PS particles or remain on their surface. Compared with pure PS, the PS-OMMT nanocomposites possessed higher stability to thermal decomposition and higher glass transition temperatures. Adding nanocomposite particles reduces the friction coefficient, and thus, the antiwear properties of the PAO are significantly improved. The PS-OMMT-3 (3 wt % of OMMT based on styrene) particles have the best tribological performance and maintained a stable, very low coefficient of friction of 0.09.