Sami Ul Haq

Mr. Sami Ul Haq

orcid

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
16 August 2022
Verified
Conference Paper
Closed access
SHSami Ul Haq
Sami Ul Haq
SRSadaf Abdul Rauf
Sadaf Abdul Rauf
ASArslan Shaukat
Arslan Shaukat
MAMuhammad Hassan Arif
Muhammad Hassan Arif
Context-aware neural machine translation has attracted much attention recently by promising sophisticated contextual information integration into conventional neural machine translation. However, context-aware NMT is challenged with effective context aggregation and increased training time due integration of extra information. In this work, we study the effect of encoding selective contextual information using pre-trained models for effective contextual integration and performance optimization. We conduct experiments on different context selection methods and quantify that encoding selected context significantly reduces the training time while maintaining superiority over sentence-level NMT models. Specifically, we experimented on IWSLT English↔German translation task and show encoding selected keywords as context is sufficient and obtains best translation results.

22 June 2026 in Preprints.org
Verified
Preprint
Open access
PDF file
SHSami Ul Haq
Sami Ul Haq
Sheila CastilhoSheila Castilho
Sheila Castilho
YGYvette Graham
Yvette Graham
Multimodal machine translation aims to integrate visual context, such as images, with textual data to improve the translation of ambiguous source text. However, the evaluation of these systems still largely relies on traditional automatic metrics, which do not account for additional modalities during evaluation. The lack of dedicated evaluation methods often results in inconsistent findings and creates uncertainty regarding the actual contribution of visual context in translation. In this work, we examine the performance of state-of-the-art trained and untrained automatic evaluation metrics, particularly when comparing multimodal and text-only systems. Our evaluation focuses on whether existing metrics are sensitive enough to distinguish between multimodal and text-only machine translation systems. We further investigate whether automatically generated image descriptions can serve as effective contextual signals for improving metric sensitivity to multimodal tasks. Our results show that incorporating such visual information into supervised metrics yields better alignment with human judgments. While all metrics successfully distinguished image-aware from image-agnostic systems on general test sets, both n-gram–based and embedding-based metrics struggled on a contrastive evaluation set designed to capture context-dependent errors. Furthermore, we discuss how the presence of visual context influences human evaluators' judgments, as ratings were often substantially revised, further emphasising the critical role of context in the evaluation of multimodal machine translation systems.

15 July 2026 in Information
Verified
Journal Article
Open access
PDF file
SHSami Ul Haq
Sami Ul Haq
Sheila CastilhoSheila Castilho
Sheila Castilho
YGYvette Graham
Yvette Graham
Multimodal machine translation (MMT) aims to integrate visual context with textual data to improve the translation of ambiguous source text, such as the inclusion of an image as additional context. However, the evaluation of systems largely still relies on automatic metrics designed to evaluate text alone, and do not account for additional modalities during evaluation. The lack of dedicated MMT evaluation methods often results in inconsistent findings and creates uncertainty regarding the actual contribution of visual context in translation. In this work, we examine the performance of state-of-the-art trained and untrained evaluation metrics, particularly when comparing multimodal and text-only systems. Our evaluation focuses on the degree to which existing metrics are sensitive enough to distinguish between multimodal and text-only machine translation systems. We further investigate the potential for automatically generated image descriptions to serve as effective contextual signals for improving metric sensitivity to multimodal tasks. Our results show that incorporating such visual information into supervised metrics yields better alignment with human judgment. While all metrics successfully distinguished image-aware from image-agnostic systems on general test sets, both n-gram-based and embedding-based metrics struggled with respect to contrastive evaluation designed to capture context-dependent errors. Furthermore, we discuss how the presence of visual context may influence human evaluator judgment, observing that given the opportunity, human ratings are often substantially revised, further emphasizing the critical role of context in the evaluation of MMT.

1 January 2025
Verified
Conference Paper
Open access
PDF file
SHSami Ul Haq
Sami Ul Haq
Sheila CastilhoSheila Castilho
Sheila Castilho
YGYvette Graham
Yvette Graham
Sami Haq, Sheila Castilho, Yvette Graham. Proceedings of the Tenth Conference on Machine Translation. 2025.

17 September 2025 in arXiv
Verified
Preprint
Open access
PDF file
SHSami Ul Haq
Sami Ul Haq
Sheila CastilhoSheila Castilho
Sheila Castilho
YGYvette Graham
Yvette Graham
Machine Translation (MT) has achieved remarkable performance, with growing interest in speech translation and multimodal approaches. However, despite these advancements, MT quality assessment remains largely text centric, typically relying on human experts who read and compare texts. Since many real-world MT applications (e.g Google Translate Voice Mode, iFLYTEK Translator) involve translation being spoken rather printed or read, a more natural way to assess translation quality would be through speech as opposed text-only evaluations. This study compares text-only and audio-based evaluations of 10 MT systems from the WMT General MT Shared Task, using crowd-sourced judgments collected via Amazon Mechanical Turk. We additionally, performed statistical significance testing and self-replication experiments to test reliability and consistency of audio-based approach. Crowd-sourced assessments based on audio yield rankings largely consistent with text only evaluations but, in some cases, identify significant differences between translation systems. We attribute this to speech richer, more natural modality and propose incorporating speech-based assessments into future MT evaluation frameworks.