Research on computer-aided arrhythmia classification is actively conducted, but the limited generalization capacity constrains its applicability in practical clinical settings. One of the primary challenges in deploying such techniques in real-world scenarios is the inter-patient variability and the consequent performance degradation. In this study, we leverage our previous innovation, the n-beat-score map (n-BSM), to introduce an adversarial framework to mitigate the issue of poor performance in arrhythmia classification within the inter-patient paradigm. The n-BSM is a 2D representation of the ECG signal, capturing its constituent beat characteristics through beat-score vectors derived from a pre-trained beat classifier. We employ adversarial learning to eliminate patient-dependent features during the training of the beat classifier, thereby generating the patient-independent n-BSM (PI-BSM). This approach enables us to concentrate primarily on the learning characteristics associated with beat type rather than patient-specific features. Through a beat classifier pre-trained with adversarial learning, a series of beat-score vectors are generated for the beat segments that make up a given ECG signal. These vectors are then concatenated chronologically to form a PI-BSM. Utilizing PI-BSMs as the input, an arrhythmia classifier is trained to differentiate between distinct types of rhythms. This approach yields a 14.27% enhancement in the F1-score in the MIT-BIH arrhythmia database and a 4.97% improvement in cross-database evaluation using the Chapman–Shaoxing 12-lead ECG database.