Aly Magassouba

Dr. Aly Magassouba

orcid

Research Focus & Metrics

Academic Career Length

12Years (2015-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 July 2019 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
AMAly Magassouba
Aly Magassouba
KSKomei Sugiura
Komei Sugiura
AQAnh Trinh Quoc
Anh Trinh Quoc
HKHisashi Kawai
Hisashi Kawai
In this paper, we address multimodal language understanding with unconstrained fetching instruction for domestic service robots. A typical fetching instruction such as “Bring me the yellow toy from the white shelf” requires to infer the user intention, i.e., what object (target) to fetch and from where (source). To solve the task, we propose a Multimodal Target-source Classifier Model (MTCM), which predicts the region-wise likelihood of target and source candidates in the scene. Unlike other methods, MTCM can handle region-wise classification based on linguistic and visual features. We evaluated our approach that outperformed the state-of-the-art method on a standard data set. We also extended MTCM with Generative Adversarial Nets (MTCM-GAN), and enabled simultaneous data augmentation and classification.

21 June 2018 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
AMAly Magassouba
Aly Magassouba
KSKomei Sugiura
Komei Sugiura
HKHisashi Kawai
Hisashi Kawai
This paper focuses on a multimodal language understanding method for "Carry and Place" tasks with domestic service robots. We address the case of ambiguous instructions, that is when the target area is not specified. For instance "Put away the milk and cereal." is a natural instruction where there is ambiguity on the target area, considering daily life environments. Conventionally, this instruction can be disambiguated from a dialogue system, but at the cost of time and cumbersomeness. Instead, we propose a multimodal approach, where the instructions are disambiguated from the robot state and environment context. We develop MultiModal Classifier Generative Adversarial Network (MMC-GAN) to predict the likelihood of the different target areas considering the robot physical limitation and the target clutter. Our approach, MMC-GAN, significantly improves accuracy compared to baseline methods using instructions only or simple deep neural networks.

25 June 2021 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
AMAly Magassouba
Aly Magassouba
KSKomei Sugiura
Komei Sugiura
HKHisashi Kawai
Hisashi Kawai
Navigation guided by natural language instructions is particularly suitable for Domestic Service Robots that interacts naturally with users. This task involves the prediction of a sequence of actions that leads to a specified destination given a natural language navigation instruction. The task thus requires the understanding of instructions, such as “Walk out of the bathroom and wait on the stairs that are on the right”. The Visual and Language Navigation remains challenging, notably because it requires the exploration of the environment and at the accurate following of a path specified by the instructions to model the relationship between language and vision. To address this, we propose the CrossMap Transformer network, which encodes the linguistic and visual features to sequentially generate a path. The CrossMap transformer is tied to a Transformer-based speaker that generates navigation instructions. The two networks share common latent features, for mutual enhancement through a double back translation model: Generated paths are translated into instructions while generated instructions are translated into path. The experimental results show the benefits of our approach in terms of instruction understanding and instruction generation.

14 December 2023 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
MDMélodie Daniel
Mélodie Daniel
AMAly Magassouba
Aly Magassouba
MAMiguel Aranda
Miguel Aranda
LLLaurent Lequièvre
Laurent Lequièvre
Juan Antonio Corrales RamonJuan Antonio Corrales Ramon
Juan Antonio Corrales Ramon
YMYoucef Mezouar
Youcef Mezouar
Robotic manipulation of deformable linear objects (DLOs) has great potential for applications in diverse fields such as agriculture or industry. However, a major challenge lies in acquiring accurate deformation models that describe the relationship between robot motion and DLO deformations. Such models are difficult to calculate analytically and vary among DLOs. Consequently, manipulating DLOs poses significant challenges, particularly in achieving large deformations that require highly accurate global models. To address these challenges, this letter presents MultiAC6: a new multi Actor-Critic framework for robot action space decomposition to control large 3D deformations of DLOs. In our approach, two deep reinforcement learning (DRL) agents orient and position a robot gripper to deform a DLO into the desired shape. Unlike previous DRL-based studies, MultiAC6 is able to solve the sim-to-real gap, achieving large 3D deformations up to 40 cm in real-world settings. Experimental results also show that MultiAC6 has a 66% higher success rate than a single-agent approach. Further experimental studies demonstrate that MultiAC6 generalizes well, without retraining, to DLOs with different lengths or materials.

21 July 2020 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
TOTadashi Ogura
Tadashi Ogura
AMAly Magassouba
Aly Magassouba
KSKomei Sugiura
Komei Sugiura
THTsubasa Hirakawa
Tsubasa Hirakawa
TYTakayoshi Yamashita
Takayoshi Yamashita
HKHisashi Kawai
Hisashi Kawai
Domestic service robots (DSRs) are a promising solution to the shortage of home care workers. However, one of the main limitations of DSRs is their inability to interact naturally through language. Recently, data-driven approaches have been shown to be effective for tackling this limitation; however, they often require large-scale datasets, which is costly. Based on this background, we aim to perform automatic sentence generation of fetching instructions: for example, "Bring me a green tea bottle on the table." This is particularly challenging because appropriate expressions depend on the target object, as well as its surroundings. In this paper, we propose the attention branch encoder-decoder network (ABEN), to generate sentences from visual inputs. Unlike other approaches, the ABEN has multimodal attention branches that use subword-level attention and generate sentences based on subword embeddings. In experiments, we compared the ABEN with a baseline method using four standard metrics in image captioning. Results show that the ABEN outperformed the baseline in terms of these metrics.

3 January 2020 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
AMAly Magassouba
Aly Magassouba
KSKomei Sugiura
Komei Sugiura
HKHisashi Kawai
Hisashi Kawai