When developing fabrics for applications in which evaporative cooling and drying play an important role, e.g., sports or occupational applications, the drying performance of fabrics is commonly determined using fast and easy-to-perform benchmark methods. The measurement conditions in these methods, however, differ significantly from the drying conditions on the human body surface, where drying is obstructed on one side of the fabric through contact with the skin and at the same time enhanced due to contact with the heated surface (skin). The aims of this study were to understand and quantify the fabric drying process at the skin interface considering these real-use effects based on tests applying two-sided drying, one-sided drying, one-sided drying on a heated surface, and one-sided drying on a heated surface in the stretched state, and to relate these to existing standard methods. The findings showed that contact with a solid heated surface such as the skin and the stretched state of the fabric both make a significant contribution (p < 0.05) to the drying rate compared to two-sided drying in standard climatic conditions. The corresponding drying rates observed for a range of typical fabrics used in leisure and sports as a first layer next to the skin were found to be 1.6 (±0.2), 1.1 (±0.2), 7.9 (±2.1), and 10.6 (±0.8) g/m² min for two-sided drying, one-sided drying, one-sided drying on a heated surface, and one-sided drying on a heated surface in the stretched state, respectively. These findings are of great importance for human thermal modelling, including clothing models, where the drying process significantly contributes to the heat and mass transfer in the skin–clothing–environment system.