Multimodal classification often suffers from recognition reliability that is asymmetric across data sources and classes, and its evaluation is frequently complicated by information leakage from overlapping sampling windows. This paper proposes a class-wise-optimized reliability fusion model (CORF), using the classification of multimodal driver responses under four controlled weather conditions as a validation case. Electroencephalogram, electrocardiogram, and vehicle signals were recorded for 30 participants, and two leakage-free protocols were adopted: leave-one-subject-out (LOSO) cross-validation and a purged temporal-block cross-validation, with all preprocessing, probability calibration, and weight estimation refitted inside every fold. Under LOSO, CORF achieved an accuracy of 0.356 (chance = 0.25) and a 0.630 macro-average area under the curve (AUC), whereas the originally used random overlapping-window split inflated accuracy to 0.92; the fused adverse-class probability discriminated adverse- from clear-weather windows with an AUC of 0.73. The fusion retains a symmetric reliability-weighting structure across classes, and its moderate symmetry-breaking difficulty emphasis significantly improved the most challenging adverse-weather class over equal-weight fusion (snow F1 +9.8 percentage points, Holm-corrected p < 0.001) at a small, statistically non-significant overall accuracy cost. CORF therefore provides a probability-calibrated, interpretable mechanism for controlling class-specific performance tradeoffs, highlighting the necessity of leakage-free validation in multimodal physiological classification.