Hearing loss and chronic vestibular pathologies require brain adaptive mechanisms supported by a cross-modal cortical plasticity. They are often accompanied by cognitive deficits. Spatial memory is a cognitive process responsible for recording information about the spatial environment and spatial orientation. Visual-spatial working memory (VSWM) is a kind of short-term working memory that allows spatial information to be temporarily stored and manipulated. It can be conditioned by hearing loss and also well-compensated chronic vestibular deficit. Vestibular rehabilitation and hearing aid devices or training are able to improve the VSWM. We studied 119 subjects suffering from perinatal or congenital hearing loss, compared with 532 healthy subjects and 404 patients with well-compensated chronic vestibular deficit (CVF). VSWM was evaluated by the eCorsi test. The subjects suffering from chronic hearing loss and/or unilateral or bilateral vestibular deficit showed a VSWM less efficient than healthy people, but much better than those with CVF, suggesting a better multimodal adaptive strategy, probably favored by a cross-modal plasticity which also provides habitual use of lip reading. The sport activity cancels the difference with healthy subjects. It is therefore evident that patients with this type of deficit since childhood should be supported and advised on a sport activity or repeated vestibular stimulation.