The identification of advanced fibrosis by applying noninvasive tests is still a key component of the diagnostic algorithm of NAFLD. The aim of this study is to assess the concordance between the FIB-4 and liver stiffness measurement (LSM) in patients referred to two liver centers for the ultrasound-based diagnosis of NAFLD. Fibrosis 4 Index for Liver Fibrosis (FIB-4) and LSM were assessed in 1338 patients. A total of 428 (32%) had an LSM ≥ 8 kPa, whereas 699 (52%) and 113 (9%) patients had an FIB-4 < 1.3 and >3.25, respectively. Among 699 patients with an FIB-4 < 1.3, 118 (17%) had an LSM ≥ 8 kPa (false-negative FIB-4). This proportion was higher in patients ≥60 years, with diabetes mellitus (DM), arterial hypertension or a body mass index (BMI) ≥ 27 kg/m². In multiple adjusted models, age ≥ 60 years (odds ratio (OR) = 1.96, 95% confidence interval (CI) 1.19–3.23)), DM (OR = 2.59, 95% CI 1.63–4.13), body mass index (BMI) ≥ 27 kg/m² (OR = 2.17, 95% CI 1.33–3.56) and gamma-glutamyltransferase ≥ 25 UI/L (OR = 2.68, 95% CI 1.49–4.84) were associated with false-negative FIB-4. The proportion of false-negative FIB-4 was 6% in patients with none or one of these risk factors and increased to 16, 31 and 46% among those with two, three and four concomitant risk factors, respectively. FIB-4 is suboptimal to identify patients to refer to liver centers, because about one-fifth may be false negative at FIB-4, having instead an LSM ≥ 8 KPa.