Industrial Control Systems (ICS) are increasingly integrated with Information Technology (IT) systems, blending Operational Technology (OT) and IT components. This evolution introduces new cyber-attack risks, necessitating specialized security measures like Intrusion Detection Systems (IDS). This paper presents our work on both developing an experimental protocol and conducting tests of various IDS types in a digital substation hardware in the loop (HIL) testbed, offering insights into their performance in realistic scenarios. Our findings reveal significant variations in IDS effectiveness against industrial-specific cyber-attacks, with IT-specific IDSs struggling to detect certain attacks and changing testlab conditions affecting the assessment of ICS-specific IDSs. The challenges faced in creating valid and reliable evaluation metrics underscore the complexities of replicating operational ICS conditions. This research enhances our understanding of IDS effectiveness in ICS settings and underscores the importance of further experimental research in HIL testlab environments.