Nature has a large repertoire of animals that take advantage of naturally abundant polarization phenomena. Among them, the mantis shrimp possesses one of the most advanced and elegant visual systems nature has developed, capable of high polarization sensitivity and hyperspectral imaging. Here, we demonstrate that by shifting the design paradigm away from the conventional paths adopted in the imaging and vision sensor fields and instead functionally mimicking the visual system of the mantis shrimp, we have developed a single-chip, low-power, high-resolution color-polarization imaging system. Our bio-inspired imager captures co-registered color and polarization information in real time with high resolution by monolithically integrating nanowire polarization filters with vertically stacked photodetectors. These photodetectors capture three different spectral channels per pixel by exploiting wavelength-dependent depth absorption of photons. Our bio-inspired imager comprises 1280 by 720 pixels with a dynamic range of 62 dB and a maximum signal-to-noise ratio of 48 dB. The quantum efficiency is above 30% over the entire visible spectrum, while achieving high polarization extinction ratios of $\sim 40$ on each spectral channel. This technology is enabling underwater imaging studies of marine species, which exploit both color and polarization information, as well as applications in biomedical fields.

A novel bioinspired polarization-sensitive imager enables passive underwater geolocalization with an accuracy of 61 km.

Polarization is one of the three fundamental properties of light, along with color and intensity, yet most vertebrate species, including humans, are blind with respect to this light modality. In contrast, many invertebrates, including insects, spiders, cephalopods, and stomatopods, have evolved to detect polarization information with high-dynamic-range photosensitive cells and utilize this information in visually guided behavior. In this paper, we present a high-dynamic-range polarization imaging sensor inspired by the visual system of the mantis shrimp. Our bioinspired imager achieves 140 dB dynamic range and 61 dB maximum signal-to-noise ratio across $384\times 288$ pixels equipped with logarithmic photodiodes. Contrary to state-of-the-art active pixel sensors, where photodiodes in individual pixels operate in reverse bias mode and yield up to $\sim 60\mathrm{dB}$ dynamic range, our pixel has a logarithmic response by operating individual photodiodes in forward bias mode. This novel pixel circuitry is monolithically integrated with pixelated polarization filters composed of 250-nm-tall × 75-nm-wide aluminum nanowires to enable snapshot polarization imaging at 30 frames per second. This sensor can enable many automotive and remote sensing applications, where high-dynamic-range imaging augmented with polarization information can provide critical information during hazy or rainy conditions.

We have developed a near-infrared (NIR) fluorescence goggle system based on the complementary metal–oxide–semiconductor active pixel sensor imaging and see-through display technologies. The fluorescence goggle system is a compact wearable intraoperative fluorescence imaging and display system that can guide surgery in real time. The goggle is capable of detecting fluorescence of indocyanine green solution in the picomolar range. Aided by NIR quantum dots, we successfully used the fluorescence goggle to guide sentinel lymph node mapping in a rat model. We further demonstrated the feasibility of using the fluorescence goggle in guiding surgical resection of breast cancer metastases in the liver in conjunction with NIR fluorescent probes. These results illustrate the diverse potential use of the goggle system in surgical procedures.

Image-guided surgery can enhance cancer treatment by decreasing, and ideally eliminating, positive tumor margins and iatrogenic damage to healthy tissue. Current state-of-the-art near-infrared fluorescence imaging systems are bulky and costly, lack sensitivity under surgical illumination, and lack co-registration accuracy between multimodal images. As a result, an overwhelming majority of physicians still rely on their unaided eyes and palpation as the primary sensing modalities for distinguishing cancerous from healthy tissue. Here we introduce an innovative design, comprising an artificial multispectral sensor inspired by the Morpho butterfly’s compound eye, which can significantly improve image-guided surgery. By monolithically integrating spectral tapetal filters with photodetectors, we have realized a single-chip multispectral imager with $1000\times$ higher sensitivity and $7\times$ better spatial co-registration accuracy compared to clinical imaging systems in current use. Preclinical and clinical data demonstrate that this technology seamlessly integrates into the surgical workflow while providing surgeons with real-time information on the location of cancerous tissue and sentinel lymph nodes. Due to its low manufacturing cost, our bio-inspired sensor will provide resource-limited hospitals with much-needed technology to enable more accurate value-based health care.

A color/near-infrared camera inspired by the mantis shrimp visual system enables tumor detection and lymph node mapping during cancer surgery.