Site response is a critical consideration when assessing earthquake hazards. Site characterization is key to understanding site effects as influenced by seismic site conditions of the local geology. Thus, a number of geophysical site characterization methods were developed to meet the demand for accurate and cost-effective results. As a consequence, a number of studies have been administered periodically as blind trials to evaluate the state-of-practice on-site characterization. We present results from the Consortium of Organizations for Strong Motion Observation Systems (COSMOS) blind trials, which used data recorded from surface-based microtremor array methods (MAM) at four sites where geomorphic conditions vary from deep alluvial basins to an alpine valley. Thirty-four invited analysts participated. Data were incrementally released to 17 available analysts who participated in all four phases: (1) two-station arrays, (2) sparse triangular arrays, (3) complex nested triangular or circular arrays, and (4) all available geological control site information including drill hole data. Another set of 17 analysts provided results from two sites and two phases only. Although data from one site consisted of recordings from three-component sensors, the other three sites consisted of data recorded only by vertical-component sensors. The sites cover a range of noise source distributions, ranging from one site with a highly directional microtremor wave field to others with omni-directional (azimuthally distributed) wave fields. We review results from different processing techniques (e.g., beam-forming, spatial autocorrelation, cross-correlation, or seismic interferometry) applied by the analysts and compare the effectiveness between the differing wave field distributions. We define the M index as a quality index based on estimates of the time-averaged shear-wave velocity of the upper 10 (V_S10), 30 (V_S30), 100 (V_S100), and 300 (V_S300) meters and show its usefulness in quantitative comparisons of V_S profiles from multiple analysts. Our findings are expected to aid in building an evidence-based consensus on preferred cost-effective arrays and processing methodology for future studies of seismic site effects.