Yernar Seksenbayev

Mr. Yernar Seksenbayev

Digital Sciences and Artificial Intelligence, L.N. Gumilyov Eurasian National University, Nur-Sultan, Astana, Kazakhstan
+1
orcid

Research Focus & Metrics

Academic Career Length

1Year (2026-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
Verified
Journal Article
Open access
PDF file
SKSaule Kudubayeva
Saule Kudubayeva
Yernar SeksenbayevYernar Seksenbayev
Yernar Seksenbayev
Aigerim YerimbetovaAigerim Yerimbetova
Aigerim Yerimbetova
Elmira Nurbekkyzy DaiyrbayevaElmira Nurbekkyzy Daiyrbayeva
Elmira Nurbekkyzy Daiyrbayeva
BBBakzhan Bakzhan
Bakzhan Bakzhan
MTMUSSA TURDALYULY
MUSSA TURDALYULY
Multimodal human–AI systems generally consider facial expressions and body motions as separate input streams, leading to disjointed interpretations and diminished emotional coherence. To overcome this issue, we offer the Engagement-Safe Expressive Alignment (ESEA) paradigm and the Unified Visual Synchrony (UVS) framework as its computational implementation. UVS models the coherence between facial expressions and gestures, offering an interpretable visual synchrony signal that can function as adaptive feedback in human–AI interactions. The framework’s key component is the Consistency Index for Affective Synchrony (CIAS), which correlates brief visual segments with scalar synchrony scores through a common latent representation. Facial and gestural signals are processed by modality-specific projection networks into a unified latent space, and CIAS is derived from the similarity and short-term temporal consistency of these latent trajectories. The synchrony index is regarded as an estimation of affective visual coherence within the ESEA paradigm. We formalize the UVS/CIAS framework and conduct a comparative experimental evaluation utilizing matched and mismatched face–gesture segments derived from rendered dialog footage. Utilizing ROC analysis, score distribution comparisons, temporal visualizations, and negative control tests, we illustrate that CIAS effectively captures structured face–gesture alignment that surpasses similarity-based baselines, while also delivering a persistent, time-resolved synchronization signal. These findings establish CIAS as a principled and interpretable feedback signal for future affect-aware, engagement-focused multimodal agents.

19 July 2026 in Technologies
Verified
Journal Article
Open access
PDF file
Yernar SeksenbayevYernar Seksenbayev
Yernar Seksenbayev
SKSaule Kudubayeva
Saule Kudubayeva
ABAbdykarim Baimankulov
Abdykarim Baimankulov
Aigerim YerimbetovaAigerim Yerimbetova
Aigerim Yerimbetova
Elmira Nurbekkyzy DaiyrbayevaElmira Nurbekkyzy Daiyrbayeva
Elmira Nurbekkyzy Daiyrbayeva
BBBakzhan Bakzhan
Bakzhan Bakzhan
We introduce the Probabilistic Multimodal Consistency Index (PMCI), a prototype measure of probabilistic semantic agreement used to quantify how much two modalities share in terms of mutually compatible affective evidence. PMCI does not model feature fusion or emotion classification directly; instead, each modality is mapped to a different probability distribution over learnable latent affective-agreement anchors, and the agreement of those distributions is quantified via the Jensen–Shannon divergence. Consequently, we propose PMCI not as a substitute for a discriminative model of pair matching, but rather as an auxiliary diagnostic index of cross-modal affective agreement. Experiments were conducted on pose-based facial and body keypoint sequences obtained from the RAVDESS and MELD datasets. In the updated RAVDESS setup, we extended the cache of preprocessed keypoint files to include all 24 actors, and we sampled four distinct pose-derived 12-frame crops per source video. This resulted in a total of 11,520 pose-derived windows, with the actor label determining the train/validation/test splits for the RAVDESS dataset. MELD contained 402 filtered pose-derived windows and served as an auxiliary in-the-wild validation setting with additional noise. When applying the updated RAVDESS standard pair-matching, DirectCosine_K0 achieved ROC–AUC = 0.943 and PR–AUC = 0.917, demonstrating that it is indeed the best exact pair-matching baseline and that PMCI-based configurations are not as accurate as DirectCosine_K0 when performing common exact face–body pair discrimination. Under this protocol, PMCI_K8, PMCI_K16, and PMCI_K32 produced ROC–AUC scores of 0.851, 0.805, and 0.887, respectively. Running a one-window-per-source-video experiment with 100 repetitions yielded similar results in the following order: DirectCosine_K0 = 0.942 plus-minus 0.009, PMCI_K16 = 0.807 plus-minus 0.016, and PMCI_K32 = 0.893 plus-minus 0.012, which shows that the results obtained were not the result of only repeated temporal crops. In the DirectCosine-mined hard negative setting, PMCI produced modest diagnostic separation above chance, where Direct+PMCI_K32 yielded ROC–AUC = 0.582. In a suite of shuffle control, permutation control, model initialization control, temperature control, anchor usage control, control latency, and pose perturbation control experiments, the diagnostic stability of PMCI was tested, confirming that PMCI is sensitive to both pose-estimation accuracy and domain shift. Finally, PMCI must be interpreted as a probabilistic diagnostic index of cross-modal affective agreement that is based on pose, not as a general-purpose emotion-recognition system.

31 August 2026 in Technologies
Verified
Journal Article
Open access
PDF file
Yernar SeksenbayevYernar Seksenbayev
Yernar Seksenbayev
SKSaule Kudubayeva
Saule Kudubayeva
OSOlga Salykova
Olga Salykova
Repeated dyadic interactions are characterized not only by average multimodal signal levels but also by how cross-modal consistency evolves over time. This study investigated whether the joint temporal organization of within-person face–body cross-modal consistency and probabilistic cross-modal affective agreement, quantified by the Consistency Index for Affective Synchrony (CIAS) and the Probabilistic Multimodal Consistency Index (PMCI), yielded reproducible session-level dynamic representations associated with repeated dyadic interaction. The primary analysis included 24 improvised and 17 naturalistic sessions, with 12 dyads in each condition. Time-varying signals from both partners were summarized through dynamic-state occupancy, dwell structure, transition characteristics, and temporal variability and evaluated using relational permutation analysis with shared-participant and technical controls. In naturalistic interaction, repeated sessions of the same dyad were significantly more similar (β = −2.0180, FDR q = 0.0114), with significant dyad-specificity beyond shared-participant similarity (Δβ = −2.4670, q = 0.0033). Representation ablations showed that PMCI provided the dominant state-separation structure, whereas CIAS contributed modest and condition-dependent complementary information. Most notably, frozen participant-disjoint evaluation on 73 naturalistic sessions from six dyads and 11 previously unseen participants retained significant dyad-associated structure and achieved session-weighted nearest-dyad accuracy of 0.8356 and macro-dyad accuracy of 0.8512. Additional robustness analyses showed that the principal pattern persisted across state-count specifications, temporal-window settings, partner-order reversal, and explicit technical-quality adjustment. Overall, the findings support the existence of reproducible session-level multimodal dynamic structure associated with repeated dyadic interaction and its transfer to previously unseen participants, while not implying direct interpersonal synchrony or psychologically defined interaction states.

30 June 2026 in EKTU Journal of Information and Communication Sciences
Verified
Journal Article
Open access
PDF file
KSKudubayeva Saule
Kudubayeva Saule
Yernar SeksenbayevYernar Seksenbayev
Yernar Seksenbayev
EDElmira Daiyrbayeva
Elmira Daiyrbayeva
AYAigerim Yerimbetova
Aigerim Yerimbetova
ZBZholdas Buribayev
Zholdas Buribayev
This paper proposes a methodological approach for evaluating the temporal stability of emotional states in multimodal data. Unlike previous studies, we introduce a multi-level formalization (signal →function →latent space →prediction) and a set of indicators including the number of class switches, episode duration, and the Hurst exponent. Stability is assessed at the level of trajectories in a polar space interpreted on the basis of Plutchik’s model. The study proposes a system of metrics describing temporal stability and provides a comparative analysis of these metrics across several multimodal architectures. The evaluation protocol includes speaker-independent splits, standard metrics (F1/UAR, CCC/MAE), and temporal stability indicators. As an addi-tional validation step, a text-only pilot experiment was conducted on a balanced small subset of the MELD dataset. The proposed evaluation approach makes it possible to characterize the temporal stability properties of models more accurately. The practicalvalue of the work lies in improving the predictability and interpretability of behavior in human-computer interaction systems and assistive technologies.

1 October 2026 in Information
Verified
Journal Article
Open access
PDF file
BZBotagoz Zhussupova
Botagoz Zhussupova
SKSaule Kudubayeva
Saule Kudubayeva
RTRakhila Turebayeva
Rakhila Turebayeva
SASaid Alimbayev
Said Alimbayev
Yernar SeksenbayevYernar Seksenbayev
Yernar Seksenbayev
Automatically mapping written Kazakh to dictionary-linked Kazakh Sign Language (KSL) lexical signs is difficult because inflected forms often do not match canonical entries in limited digital lexicons. Surdo Med combines Kazakh natural-language processing, Dimskis notation, reusable avatar animation components, and a web interface for medical-domain lexical learning. This study evaluates morphology-aware lexical concept recovery rather than complete Kazakh-to-KSL translation. The primary benchmark uses naturally occurring inflected tokens from the official UD Kazakh-KTB test split that map uniquely to a frozen medical concept inventory. Across 373 supported inflected tokens, direct lookup recovered 1.61% of target concepts, a rule-based baseline recovered 45.31%, and contextual Stanza lemmatization recovered 97.86% (sentence-cluster bootstrap 95% CI: 96.29–99.19%). Recovery remained 96.85% after deduplicating surface–lemma pairs and 94.80% when macro-averaged across 52 lexical concepts. An external analysis using published KSL lexical material showed 20.0% overlap with the medical inventory; within supported items, Stanza recovered 6/6 root tokens versus 3/6 for direct lookup. These results support morphology-aware normalization for dictionary-linked KSL concept recovery. Avatar linguistic quality and educational effectiveness require separate human evaluation.