Ultra-short-term HRV features assess minor autonomous nervous system variations such as variations resulting from cognitive stress peaks during demanding tasks. Several studies compare ultra-short-term and short-term HRV measurements to investigate their reliability. However, existing experiments are conducted in low cognitively demanding environments. In this paper, we propose to evaluate these measurements’ reliability under cognitively demanding tasks using a near real-life setting. For this purpose, we selected 31 HRV features, extracted from data collected from 21 programmers performing code comprehension, and compared them across 18 different time frames, ranging from 3 min to 10 s. Statistical significance and correlation tests were performed between the features extracted using the larger window (3 min) and the same features extracted with the other 17 time frames. We paired these analyses with Bland–Altman plots to inspect how the extraction window size affects the HRV features. The main results show 13 features that presented at least 50% correlation when using 60-second windows. The HF and mNN features achieved around 50% correlation using a 30-second window. The 30-second window was the smallest time frame considered to have reliable measurements. Furthermore, the mNN feature proved to be quite robust to the shortening of the time resolution.

Ultra-short term HRV features are becoming increasingly popular due to the fact that they do not need long time periods for their assessment and, therefore, can be used in nearly real-time cognitive load assessment scenarios where the standard 1-min to 5-min time frames are not applicable. Several authors focused on the assessment of the validity of these features by comparing them to the accepted 5-min short term features, showing that the accuracy of these features decreases with the decrease of the analysis window length. However, there is one question that, to the best of our knowledge, has not been fully addressed yet. How does the reduction of the analysis window affect the classification process during cognitive demanding tasks? In this paper we propose the use of 18 different time frames, ranging from 3 minutes to 10 seconds, to extract HRV features from data collected out of 21 subjects during code comprehension tasks. The HRV features are then associated with a code section, gazed during an experiment run, and statistical transformations are computed to built the several datasets, where each section gazed is a sample. A Support Vector Machine (SVM) classifier was trained for each different dataset using a Leave-One-Subject-Out cross-validation procedure, following 3 distinct approaches. The classifier’s goal is to discriminate between low and high complexity code sections analysed by the subjects during the experiment. The F1-Scores ranged from 0.79 to 0.64, indicating that it’s possible to achieve similar, but lower classification results using smaller time frames, with a consistent increase of the variability in the performance evidenced by a higher standard deviation of F1-Scores in the smaller time frames.