Evaluating the degree of improvement of an impaired freshwater ecosystem resembles the statistical null-hypothesis testing through which the prevailing conditions are compared against a reference state. The pillars of this process involve the robust delineation of what constitutes an achievable reference state; the establishment of threshold values for key environmental variables that act as proxies of the degree of system impairment; and the development of an iterative decision-making process that takes advantage of monitoring data to assess the system-restoration progress and revisit management actions accordingly. Drawing the dichotomy between impaired and non-impaired conditions is a challenging exercise that is surrounded by considerable uncertainty stemming from the variability that natural systems display over time and space, the presence of ecosystem feedback loops (e.g., internal loading) that actively influence the degree of recovery, and our knowledge gaps about biogeochemical processes directly connected to the environmental problem at hand. In this context, we reappraise the idea of probabilistic water quality criteria, whereby the compliance rule stipulates that no more than a stated number of pre-specified water quality extremes should occur within a given number of samples collected over a compliance assessment domain. Our case study is the Bay of Quinte, Ontario, Canada; an embayment lying on the northeastern end of Lake Ontario with a long history of eutrophication problems. Our study explicitly accounts for the covariance among multiple water quality variables and illustrates how we can assess the degree of improvement for a given number of violations of environmental goals and samples collected from the system. The present framework offers a robust way to impartially characterize the degree of restoration success and minimize the influence of the conflicting perspectives among decision makers/stakeholders and conscious (or unconscious) biases pertaining to water quality management.