With laminated structure like graphite, hexagonal boron nitride nanoparticle (nano-hBN) performed excellent anti-wear and friction-reducing properties. In this paper, hBN nanofluids were prepared using sodium polyacrylate (PAAS) as dispersant in water. Four-ball tribometer and pin-on-disk tribometer were employed to evaluate the tribological performance and lubrication behavior of the above-prepared lubricating fluids during steel/steel friction. The worn surface of balls and disks was analyzed by scanning electron microscope (SEM), energy dispersive spectrometer (EDS) and x-ray photoelectron spectrometer (XPS). The minimum lubricating film thickness was calculated by Hamrock and Dowson equation. Results showed that 0.7 wt% was the optimum concentration of nano-hBN in water which significantly reduced the coefficient of friction (COF) and wear scar diameter (WSD) by about 29% and 15%, respectively. The lubricating film thickness was lower than both nanoparticles diameter and worn surface roughness. From the above it was concluded that the worn area could be divided into direct contact, particle lubricating and liquid lubricating area with rolling mehanism, interlayer sliding mechansim, polishing mechansim and mending mechansim of nanoparticles. A protective film was formed on friction surface by PAAS molecule, FeOOH and FeO due to the absorption effect and tribochemical reaction between the nanofluids and worn surface, which further reduced the friction and wear rate.