In this article, the effect of two different methods of cooling enhancement, including the use of nanofluid as the coolant and sinusoidal wavy walls, on the performance of rectangular microchannel heat sinks (MCHSs) has been numerically investigated. Three-dimensional simulations based on the finite-volume approach are carried out using three different wave amplitudes (50, 100, and 200 µm) and wavelengths (1.3, 2, and 4 mm). Water-based nanofluid containing Al₂O₃ nanoparticles with volume fraction of 1% is employed as the coolant, and its influence on the hydraulic and thermal performance of the MCHS is compared with the water. The simulation of nanofluid flow is performed by both the single-phase method and the two-phase mixture method. Comparison between predicted results and experimental data from literature is carried out, which indicates the two-phase method is more precise than the single-phase method. Results showed that the enhanced microchannels with higher wave amplitudes and lower wavelengths performed better in terms of heat transfer, and among the investigated geometries, the wavelength of 1.3 mm and the amplitude of the 200 µm show the highest cooling performance. Using these optimal geometrical values in the wavy-walled microchannel heat sink (WMCHS), the amount of heat absorbed by the water-Al₂O₃ with volume fraction of 1% increases by 43% compared with the water flow in the straight-walled microchannel heat sink (SMCHS) at Reynolds number of 500. In addition, in a given geometry with fixed length and cross section area, employing wavy walls has a greater effect on heat transfer enhancement than using nanofluid instead of water. Employing optimal wavy walls in the MCHS, the amount of absorbed heat increases by 30% when the coolant is pure water and by 21% when the coolant is nanofluid. Whereas if nanofluid is used instead of water, the heat absorbed increases by 18% in SMCHS and 10% in WMCHS. Although both methods of cooling enhancement increase the pressure drop as well, the negative effect of the increase in pressure drop is less than the positive effect of increasing heat transfer in both methods.