The cascade approach to Speech Translation (ST) is based on a pipeline that concatenates an Automatic Speech Recognition (ASR) system followed by a Machine Translation (MT) system. Nowadays, state-of-the-art ST systems are populated with deep neural networks that are conceived to work in an offline setup in which the audio input to be translated is fully available in advance. However, a streaming setup defines a completely different picture, in which an unbounded audio input gradually becomes available and at the same time the translation needs to be generated under real-time constraints. In this work, we present a state-of-the-art streaming ST system in which neural-based models integrated in the ASR and MT components are carefully adapted in terms of their training and decoding procedures in order to run under a streaming setup. In addition, a direct segmentation model that adapts the continuous ASR output to the capacity of simultaneous MT systems trained at the sentence level is introduced to guarantee low latency while preserving the translation quality of the complete ST system. The resulting ST system is thoroughly evaluated on the real-life streaming Europarl-ST benchmark to gauge the trade-off between quality and latency for each component individually as well as for the complete ST system.

Although Long-Short Term Memory (LSTM) networks and deep Transformers are now extensively used in offline ASR, it is unclear how best offline systems can be adapted to work with them under the streaming setup. After gaining considerable experience on this regard in recent years, in this paper we show how an optimized, low-latency streaming decoder can be built in which bidirectional LSTM acoustic models, together with general interpolated language models, can be nicely integrated with minimal perfomance degradation. In brief, our streaming decoder consists of a one-pass, real-time search engine relying on a limited-duration window sliding over time and a number of ad hoc acoustic and language model pruning techniques. Extensive empirical assessment is provided on truly streaming tasks derived from the well-known LibriSpeech and TED talks datasets, as well as from TV shows on a main Spanish broadcasting station.

The application of large language models (LLMs) to speech translation (ST) or, in general, to machine translation (MT) has recently provided excellent results, superseding conventional encoder-decoder MT systems in the general domain. However, this is not clearly the case when LLMs as MT systems are translating medical-related materials. In this respect, the provision of multilingual training materials for oncology professionals is a goal of the EU project Interact-Europe in which this work was framed. To this end, cross-language technology adapted to the oncology domain was developed, evaluated and deployed for multilingual interspecialty medical education. More precisely, automatic speech recognition (ASR) and MT models were adapted to the oncology domain to translate English pre-recorded training videos, kindly provided by the European School of Oncology (ESO), into French, Spanish, German and Slovene. In this work, three categories of MT models adapted to the medical domain were assessed: bilingual encoder-decoder MT models trained from scratch, pre-trained large multilingual encoder-decoder MT models, and multilingual decoder-only LLMs. The experimental results underline the competitiveness in translation quality of LLMs compared to encoder-decoder MT models. Finally, the ESO speech dataset, comprising roughly 1000 videos and 745 h for the training and evaluation of ASR, MT and ST models, was publicly released for the scientific community.

Older adults often need guidance when visiting new buildings for the first time. However, indoor navigation remains challenging due to the lack of Global Positioning System (GPS) availability, visually repetitive corridors, and frequent location failures. This article presents a multimodal indoor navigation assistant that combines graph-based route planning with visual landmark verification to provide step-by-step guidance. The environment is modelled as a directed graph whose nodes are annotated with semantic landmarks, and the graph is constructed primarily from a video of the building, reducing the need for 3D scanners, beacons, or other specialised instruments. Routes are calculated using Dijkstra’s shortest-path algorithm over the semantic graph. During navigation, camera frames are analysed using a restricted vision-language recognition strategy that only considers candidate landmarks from the current and next nodes, reducing false detections and improving interpretability. To increase robustness, a temporary voting mechanism was introduced to confirm node transitions, as well as a hierarchical redirection strategy with local and global recovery. The system is implemented in two modes: handheld mode with visual cues using augmented reality arrows, mini map and voice instructions, and hands-free mode with front camera using voice instructions and keywords. Evaluation involved preliminary technical testing in the United Kingdom followed by formal user validation in Spain. During these trials, participants reported high usability, strong confidence and safety, and increased perceived independence.

Online video lecture repositories are rapidly growing and becoming established as fundamental knowledge assets. However, most lectures are neither transcribed nor translated because of the lack of cost-effective solutions that can give accurate enough results. In this paper, we describe a system architecture that supports the cost-effective transcription and translation of large video lecture repositories. This architecture has been adopted in the EU project transLectures and is now being tested on a repository of more than 9000 video lectures at the Universitat Politecnica de Valencia. Following a brief description of this repository and of the transLectures project, we describe the proposed system architecture in detail. We also report empirical results on the quality of the transcriptions and translations currently being maintained and steadily improved.