A new nonparametric estimator of Shannon's entropy on a countable alphabet is proposed and analyzed against the well-known plug-in estimator. The proposed estimator is developed based on Turing's formula, which recovers distributional characteristics on the subset of the alphabet not covered by a size-n sample. The fundamental switch in perspective brings about substantial gain in estimation accuracy for every distribution with finite entropy. In general, a uniform variance upper bound is established for the entire class of distributions with finite entropy that decays at a rate of O(ln(n)/n) compared to O([ln(n)]2/n) for the plug-in. In a wide range of subclasses, the variance of the proposed estimator converges at a rate of O(1/n), and this rate of convergence carries over to the convergence rates in mean squared errors in many subclasses. Specifically, for any finite alphabet, the proposed estimator has a bias decaying exponentially in n. Several new bias-adjusted estimators are also discussed.
1 January 2007 in Journal of Hydraulic Engineering
The objective of this paper is to describe an autoregressive regression framework with respect to metered sewer flow and simultaneous rainfall data series. The main purpose is to produce more reliable estimates for rain derived inflow and infiltration (RDII) characteristics in a sanitary sewer basin. The discussion is centered at three specific points: (1) a justification for an autoregressive error structure; (2) a modeling setup to capture the RDII characteristics for rain events of varying duration and intensity; and (3) a statistical quantification of RDII reduction, by a sewer rehabilitation project, via tests based on two independent data series.
1 August 2007 in Journal of Quantitative Linguistics
A simple frequentist's justification of Turing's formula, an improvement to Turing's formula by means of reduced bias, a clarification of the relationships among various objects related to Turing's formula, a conservative confidence interval to Turing's target, and a conservative testing procedure using observed rank-frequencies under a hypothesized known infinite-dimensional multinomial distribution are given in this paper. As an example, the authorship of the nine-stanza poem “Shall I Die?” is tested against Shakespeare's canon and statistically significant evidence is found for a difference in word type usage.
In this letter, we introduce an estimator of Küllback-Leibler divergence based on two independent samples. We show that on any finite alphabet, this estimator has an exponentially decaying bias and that it is consistent and asymptotically normal. To explain the importance of this estimator, we provide a thorough analysis of the more standard plug-in estimator. We show that it is consistent and asymptotically normal, but with an infinite bias. Moreover, if we modify the plug-in estimator to remove the rare events that cause the bias to become infinite, the bias still decays at a rate no faster than [ ]. Further, we extend our results to estimating the symmetrized Küllback-Leibler divergence. We conclude by providing simulation results, which show that the asymptotic properties of these estimators hold even for relatively small sample sizes.
Zhang in 2012 introduced a nonparametric estimator of Shannon’s entropy, whose bias decays exponentially fast when the alphabet is finite. We propose a methodology to estimate the bias of this estimator. We then use it to construct a new estimator of entropy. Simulation results suggest that this bias adjusted estimator has a significantly lower bias than many other commonly used estimators. We consider both the case when the alphabet is finite and when it is countably infinite.
1 March 2005 in Journal of Environmental Engineering
Sanitary sewer overflows (SSOs) are a major environmental issue. One of the major factors causing SSOs is the rain-derived inflow and infiltration (RDII) to a separate sanitary sewer system. If a wastewater collection system is not well maintained, cumulative system-wide RDII could easily cause the wastewater conveyance and treatment capacity to be overwhelmed, and thus lead to SSOs. Monitoring system condition is a key component in system management. The industry’s standard approaches to system monitoring include the practice of collecting and analyzing continuous rainfall and flow data at certain key locations in the system to estimate the level of RDII. However, the writer is of the opinion that the current standard analytical methodologies of the industry can be significantly improved. This paper introduces a basic regression approach with autoregressive errors to support statistical inferences with respect to the level of RDII.