This paper presents the research and development of the prototype of the assistive mobile information robot (AMIR). The main features of the presented prototype are voice and gesture-based interfaces with Russian speech and sign language recognition and synthesis techniques and a high degree of robot autonomy. AMIR prototype’s aim is to be used as a robotic cart for shopping in grocery stores and/or supermarkets. Among the main topics covered in this paper are the presentation of the interface (three modalities), the single-handed gesture recognition system (based on a collected database of Russian sign language elements), as well as the technical description of the robotic platform (architecture, navigation algorithm). The use of multimodal interfaces, namely the speech and gesture modalities, make human-robot interaction natural and intuitive, as well as sign language recognition allows hearing-impaired people to use this robotic cart. AMIR prototype has promising perspectives for real usage in supermarkets, both due to its assistive capabilities and its multimodal user interface.

Приведен аналитический обзор разновидностей интегральных (end-to-end) систем для распознавания речи, методов их построения и оптимизации. Рассмотрены модели на основе коннекционной временной классификации (CTC), модели на основе механизма внимания и модели, построенные с использованием условных случайных полей (CRF). Также описаны возможности интеграции с языковыми моделями на этапе декодирования, демонстрирующие существенное сокращение ошибки распознавания для интеграционных моделей. Обзор исследований, проводимых в данной предметной области, показывает, что интегральные системы позволяют достичь результатов, сравнимых с результатами стандартных систем, но с применением более простой конфигурации и быстрой работой системы, как при обучении, так и при декодировании. Рассмотрены популярные и развивающиеся библиотеки и инструментарии для построения интегральных систем распознавания речи.

We show the results of studying models of the Russian language constructed with recurrent artificial neural networks for systems of automatic recognition of continuous speech. We construct neural network models with different number of elements in the hidden layer and perform linear interpolation of neural network models with the baseline trigram language model. The resulting models were used at the stage of rescoring the N best list. In our experiments on the recognition of continuous Russian speech with extra-large vocabulary (150 thousands of word forms), the relative reduction in the word error rate obtained after rescoring the 50 best list with the neural network language models interpolated with the trigram model was 14%.