This paper reports on a benchmark study based on small-scale (1:50) measurements of a single, oscillating water column chamber mounted sideways in a long flume. The geometry of the OWC chamber is extracted from a barge-like, attenuator-type floating concept “KNSwing” with 40 chambers targeted for deployment in the Danish part of the North Sea. In addition to traditional two-way energy extraction we also consider one-way energy extraction with passive venting and compare chamber response, pressures and total absorbed energy between the two methods. A blind study was established for the numerical modeling, with participants applying several implementations of weakly nonlinear potential flow theory and commercial Navier–Stokes solvers (CFD). Both compressible and incompressible models were used for the air phase. Potential flow calculations predict more energy absorption near the chamber resonance for one-way absorption than for two-way absorption, but the opposite is found from the experimental measurements. This outcome is mainly attributed to energy losses in the experimental passive valve system, but this conclusion must be confirmed by better experimental measurements. Modeling the one-way valve in CFD proved to be very challenging and only one team was able to provide results which were generally closer to the experiments. The study illustrates the challenges associated with both numerical and experimental analysis of OWC chambers. Air compressibility effects were not found to be important at this scale, even with the large volume of additional air used for the one-way case.