Summary: In this paper we describe an FPGA implementation of the Scale Invariant Feature Transform (SIFT) algorithm. The FPGA is required as its a lightweight device which makes it ideal for vision‐guided hybrid neuro‐prostheses utilised for upper limbs replacement. SIFT point detection is needed for computation of coordinates of the object‐to‐grasp in a wearable multi‐camera system. A modified SIFT algorithm is proposed and an implementation of it into C/C++ language on Xilinx ZCU102 FPGA board. The proposed hybrid hardware/software solution is compared to other hardware or hybrid implementations of the SIFT algorithm and with the baseline software detector OpenSIFT. The algorithm optimised for FPGA gives an average precision of 0.84 and the average recall of 0.94 in SIFT‐point detection compared to the baseline.The proposed solution has lower dissipated power than other solutions like CPU or GPU, and it has better computational speed.This solution allows for processing medium‐sized images in real‐time with low power consumption.

The present paper proposes an implementation of a hybrid hardware–software system for the visual servoing of prosthetic arms. We focus on the most critical vision analysis part of the system. The prosthetic system comprises a glass-worn eye tracker and a video camera, and the task is to recognize the object to grasp. The lightweight architecture for gaze-driven object recognition has to be implemented as a wearable device with low power consumption (less than 5.6 W). The algorithmic chain comprises gaze fixations estimation and filtering, generation of candidates, and recognition, with two backbone convolutional neural networks (CNN). The time-consuming parts of the system, such as SIFT (Scale Invariant Feature Transform) detector and the backbone CNN feature extractor, are implemented in FPGA, and a new reduction layer is introduced in the object-recognition CNN to reduce the computational burden. The proposed implementation is compatible with the real-time control of the prosthetic arm.