We present SpeakingFaces as a publicly-available large-scale multimodal dataset developed to support machine learning research in contexts that utilize a combination of thermal, visual, and audio data streams; examples include human–computer interaction, biometric authentication, recognition systems, domain transfer, and speech recognition. SpeakingFaces is comprised of aligned high-resolution thermal and visual spectra image streams of fully-framed faces synchronized with audio recordings of each subject speaking approximately 100 imperative phrases. Data were collected from 142 subjects, yielding over 13,000 instances of synchronized data (∼3.8 TB). For technical validation, we demonstrate two baseline examples. The first baseline shows classification by gender, utilizing different combinations of the three data streams in both clean and noisy environments. The second example consists of thermal-to-visual facial image translation, as an instance of domain transfer.

This paper presents the preliminary prototype design and implementation of the Nazarbayev University (NU) Hand, a new semi-anthropomorphic multigrasp robotic hand. The hand is designed to be an end effector for industrial and service robots. The main objective is to develop a low-cost, low-weight and easily manufacturable robotic hand with a sensor module allowing acquisition of data for autonomous intelligent object manipulation. 3D printing technologies were extensively used in the implementation of the hand. Specifically, the structure of the hand is printed using a 3D printer as a complete assembly voiding the need of using fasteners and bearings for the assembly of the hand and decreasing the total weight. The hand also incorporates a sensor module containing a LIDAR, digital camera and non-contact infrared temperature sensor for intelligent automation. As an alternative to teach pendants for the industrial manipulators, a teaching glove was developed, which acts as the primary human machine interface between the user and the NU Hand. The paper presents an extensive performance characterization of the robotic hand including finger forces, weight, audible noise level during operation and sensor data acquisition.

This paper presents an intelligent object manipulation framework for industrial tasks, which integrates a sensor-rich multi-fingered robot hand, an industrial robot manipulator, a conveyor belt and employs machine learning algorithms. The framework software architecture is implemented using a Windows 7 operating system with RTX real-time extension for synchronous handling of peripheral devices. The framework uses Scale Invariant Feature Transform (SIFT) image processing algorithm, Support Vector Machine (SVM) machine learning algorithm and 3D point cloud techniques for intelligent object recognition based on RGB camera and laser rangefinder information from the robot hand end effector. The objective is automated manipulation of objects with different shapes and poses with minimum programming effort applied by a user.

This paper describes an inertial motion capture based arm gesture recognition system for the high-level control of a mobile manipulator. Left arm kinematic data of the user is acquired by an inertial motion capture system (Xsens MVN) in real-time and processed to extract supervisory user interface commands such as "Manipulator On/Off", "Base On/Off" and "Operation Pause/Resume" for a mobile manipulator system (KUKA youBot). Principal Component Analysis and Linear Discriminant Analysis are employed for dimension reduction and classification of the user kinematic data, respectively. The classification accuracy for the six class gesture recognition problem is 95.6 percent. In order to increase the reliability of the gesture recognition framework in real-time operation, a consensus voting scheme involving the last ten classification results is implemented. During the five-minute long teleoperation experiment, a total of 25 high-level commands were recognized correctly by the consensus voting enhanced gesture recognizer. The experimental subject stated that the user interface was easy to learn and did not require extensive mental effort to operate.