With the advance in automatic speech recognition, voice user interface has gained popularity recently. Since the COVID-19 pandemic, VUI is increasingly preferred in online communication due to its non-contact. Additionally, various ambient noise impedes the public applications of voice user interfaces due to the requirement of audio-only speech recognition methods for a high signal-to-noise ratio. In this paper, we present Wavoice, the first noise-resistant multi-modal speech recognition system that fuses two distinct voice sensing modalities, i.e., millimeter-wave (mmWave) signals and audio signals from a microphone, together. One key contribution is that we model the inherent correlation between mmWave and audio signals. Based on it, Wavoice facilitates the real-time noise-resistant voice activity detection and user targeting from multiple speakers. Furthermore, we elaborate on two novel modules into the neural attention mechanism for multi-modal signals fusion, and result in accurate speech recognition. Extensive experiments verify Wavoice's effectiveness under various conditions with the character recognition error rate below 1% in a range of 7 meters. Wavoice outperforms existing audio-only speech recognition methods with lower character error rate and word error rate. The evaluation in complex scenes validates the robustness of Wavoice.

Internet of Things (IoT) devices are hindered from communicating with their neighbors by incompatible protocols or electromagnetic interference. Existing solutions adopting physical covert channels have limitations in receiver distinction, additional hardware, conditional placement, or physical contact. Our system, Deaf-Aid, utilizes the stealthy speaker-to-gyroscope channel to build robust protocol-independent communication with automatic receiver identification. Deaf-Aid exploits ultrasonic signals at a frequency corresponding to the target receiver, forcing the gyroscope inside to resonate, so as to convey information. We probe the relationship among axes in a gyroscope to surmount frequency offset ingeniously and support multi-channel communication. Meanwhile, Deaf-Aid identifies the receivers automatically via device fingerprints constituted by the diversity of resonant frequency ranges. Furthermore, we entitle Deaf-Aid the capability of mobile communication which is an essential demand for IoT devices. We address the challenge of accurate signals recovery from motion interference. Extensive evaluations demonstrate that Deaf-Aid yields 47bps with BER lower than 1% under motion interference. To our best knowledge, Deaf-Aid is the first work to enable stealthy mobile IoT communication on the basis of inertial motion sensors.

In cyber-physical systems, inertial sensors are the basis for identifying motion states and making actuation decisions. However, extensive studies have proved the vulnerability of those sensors under acoustic transduction attacks, which leverage malicious acoustics to trigger sensor measurement errors. Unfortunately, the threat from such attacks is not assessed properly because of the incomplete investigation on the attack's potential, especially towards multiple-degree-of-freedom systems, e.g., drones. To thoroughly explore the threat of acoustic transduction attacks, we revisit the attack model and design a new yet practical acoustic modulation-based attack, named KITE. Such an attack enables stable and controllable injections, even under frequency offset based distortions that limit the effect of prior attacking approaches. KITE exploits the potential threat of transduction attacks without the need of strengthening attackers' abilities. Furthermore, we extend the attack surface to multiple-degree-of-freedom (MDOF) systems, which are more widely deployed but ignored by prior work. Our study also covers the scenario of attacking moving targets. By revealing the practical threat from acoustic transduction attacks, we appeal for both the attention to their harm and necessary countermeasures.

IMU-based eavesdropping has brought growing concerns over smartphone users’ privacy. In such attacks, adversaries utilize IMUs that require zero permissions for access to acquire speeches. A common countermeasure is to limit sampling rates (within 200 Hz) to reduce overlap of vocal fundamental bands (85-255 Hz) and inertial measurements (0-100 Hz). Nevertheless, we experimentally observe that IMUs sampling below 200 Hz still record adequate speech-related information because of aliasing distortions. Accordingly, we propose a practical side-channel attack, InertiEAR, to break the defense of sampling rate restriction on the zero-permission eavesdropping. It leverages IMUs to eavesdrop on both top and bottom speakers in smartphones. In the InertiEAR design, we exploit coherence between responses of the built-in accelerometer and gyroscope and their hardware diversity using a mathematical model. The coherence allows precise segmentation without manual assistance. We also mitigate the impact of hardware diversity and achieve better device-independent performance than existing approaches that have to massively increase training data from different smartphones for a scalable network model. These two advantages re-enable zero-permission attacks but also extend the attacking surface and endangering degree to off-the-shelf smartphones. InertiEAR achieves a recognition accuracy of 78.8% with a cross-device accuracy of up to 49.8% among 12 smartphones.

Eavesdropping via inertial measurement units (IMUs) has brought growing concerns over smartphone users’ privacy. In such attacks, adversaries utilize IMUs, including accelerometers and gyroscopes, which require zero permissions for access to acquire speeches. A common countermeasure is to limit sampling rates (within 200 Hz) to reduce overlap of vocal fundamental bands (85$\sim$255 Hz) and inertial measurements (0$\sim$100 Hz). Nevertheless, we observe that IMUs sampling below 200 Hz still record adequate speech-related information because of aliasing distortions. Accordingly, we propose a practical side-channel attack, namely InertiEAR, to break the defense of sampling rate restriction on the zero-permission eavesdropping. It leverages accelerometers and gyroscopes jointly to eavesdrop on both top and bottom speakers in smartphones. We exploit coherence between responses of the built-in accelerometer and gyroscope using a mathematical model. The coherence allows precise segmentation without manual assistance. We also mitigate the impact of hardware diversity and achieve better device-independent performance than existing approaches that have to massively increase training data from different smartphones for a scalable network model. These two advantages re-enable zero-permission attacks but also extend the attacking surface and endangering degree to off-the-shelf smartphones. InertiEAR achieves the recognition accuracy of 78.8% with the cross-device accuracy of up to 60.9% among 12 smartphones.

Ultrasonic microphone jammers protect speech privacy from being eavesdropped by leveraging microphones' non-linearity. However, existing jammers merely introduce independent noises and are vulnerable to capable adversaries who adopt advanced denoising techniques. We propose a novel jammer, namely MicFrozen. It reduces the signal-to-noise ratio (SNR) at the adversary's microphone from two perspectives, i.e., cancelling speech signals and adding noises that are difficult to be removed. It effectively cancels out the protected speech signals at the adversary without compromising the delivery of the signal to the targeted individual. MicFrozen further adds coherent noises that are coupled with the speech signals to resist removal by the adversary. Extensive evaluations show that MicFrozen can cause a low SNR (-13.6 dB) at the adversary and up to 96.9% of speech signals are unrecognized at the adversary even if state-of-the-art denoising techniques are adopted by the adversary. Comprehensive experiments demonstrate the effectiveness of MicFrozen confronted by capable adversaries.