Superior vision in nocturnal insects inspires new night vision technologies

At night—because of the paucity of light—human visual capacities are severely reduced, with a complete loss of the ability to see color and a dramatic loss in the ability to see fine spatial and temporal details. However, this is not the case for many nocturnal animals (e.g., insects). Nocturnal animals typically have highly sensitive eye designs and visual neural circuitry that is optimized for the extraction of reliable information from dim and noisy visual images. Although we are only now at the threshold of understanding the neural mechanisms responsible for reliable nocturnal vision, there is growing evidence to suggest that the neural summation of photons in space and time is critically important.^{1, 2} Artificial imaging systems that operate in dim light—like nocturnal eyes—also suffer from limitations in detection reliability. These systems should thus also benefit from the same strategies for improving their reliability (i.e., the summation of photons in space and time).

In our recent work—particularly on fast-flying moths and bees, and on ball-rolling dung beetles—we have studied the visual abilities of nocturnal animals. In particular, we have shown that they are able to distinguish colors³—see Figure 1(A)—detect faint movements,¹ learn visual landmarks⁴—see Figure 1(B)—orient to the faint pattern of polarized light produced by the Moon,⁵ and navigate using the stars.⁶ These impressive visual capabilities are the result of exquisitely adapted eyes and visual systems (the product of millions of years of evolution).

Figure 1. Nocturnal insects with sensitive vision. (A) The European elephant hawkmoth, Deilephila elpenor, whose sensitive superposition eyes allow it to distinguish color at night. (B) The central American halictid bee, Megalopta genalis, whose sensitive apposition eyes allow it to forage at night (by visually learning landmarks along the foraging route and around the nest entrance). Photographs provided with kind permission of Michael Pfaff (A) and Ajay Narendra (B).

The strategy for improving nocturnal vision involves improving sensitivity by increasing the size of the visual ‘pixel’ and by lengthening the visual exposure time. Even though vision in dim light thereby becomes necessarily coarser and slower, it also becomes significantly more reliable.^{1, 2} In our most recent work,⁷ it was therefore our idea to take visual solutions—which result from millions of years of biological evolution—and incorporate them in a computer algorithm that radically improves the quality of video sequences collected in dim light. Our algorithm thus mimics several of the key steps of nocturnal visual processing.⁸

The first step in our algorithm (see Figure 2) involves an amplification of the dim image sequence. Such amplification (or gain) is also typical of the photoreceptor responses of nocturnal insects.⁹ The downside of this amplification is that it amplifies both the signal and the noise, and thus does not improve the signal-to-noise ratio per se. To negotiate this issue, we include another crucial step in the algorithm that involves spatial and temporal summation. This step results in signal information (which is correlated between frames in the video sequence) being summed and reinforced, and noise (which is uncorrelated) being averaged out. The result is a dramatic improvement in the signal-to-noise ratio. Moreover, because this summation is conducted separately (and in parallel) for the red, green, and blue channels, the summation result preserves color information (as evident in the output video frame in Figure 2). A similar, separate and parallel, summation of color channels in nocturnal animals is likely the reason that many can see color at night.

Figure 2. The main steps in the night vision algorithm. The algorithm consists of a contrast enhancement step (amplification) and a noise-reduction step that is based on optimal spatial and temporal summation. The algorithm adapts automatically to the local intensity distribution and preserves as much of the structure in the scene as possible. It also adapts to the local noise level, which depends on the ambient light level. Furthermore, it sharpens edges in the output at appropriate places to prevent over-smoothing (via a lateral inhibition algorithm that is similar to lateral inhibition in the retina). A still frame (upper left) from a dim input video sequence can be compared with a still frame (upper right) from the significantly enhanced output video sequence, i.e., which resulted from processing with the algorithm.

A disadvantage of the even radial spatial summation, however, is that it does not preserve edges and boundaries (i.e., the contrast elements of visual scenes that define and localize objects). To deal with this problem, our algorithm performs a real-time structure tensor analysis of the intensity structure (in space and time) of the scene contained within the film. In addition, local smoothing kernels for each location in the film frame are constructed. These kernels adapt to the local spatiotemporal intensity structure. They are large (i.e., summing more) in directions where there are no spatiotemporal edges, and small (i.e., summing less) in directions where there are strong edges. Moreover, because dimmer video sequences require greater extents of summation, the volumes of these kernels become large for videos filmed at lower light levels. The algorithm therefore adapts to light intensity in an automatic manner, much like an animal's visual system.

By mimicking the way nocturnal visual systems process information, our night-vision algorithm can be used to dramatically restore the quality of video sequences recorded in very dim light.⁸ Not only are spatial and temporal details—obscured by noise in the original video—restored, but color information is also preserved. Our hypothesis that spatial and temporal summation is a crucial strategy in nocturnal visual processing is thus very well supported by our algorithm. The output of the algorithm suggests that summation can greatly enhance nocturnal visual performance.

Insects have conquered the terrestrial world's dimmest habitats and have done so with some of the most sensitive visual systems found in the animal kingdom. In our work we have shown that the neural strategies they use—and the benefits to visual reliability they bestow—are directly applicable to algorithms for processing video sequences captured in very dim light. As we continue to study how the brain of nocturnal insects processes and optimizes the visual information it receives, we will hopefully be in a position to further improve our algorithm. We plan to add components that are necessary for tasks such as object recognition and distance discrimination.

I am grateful to many institutions that have supported this work over the years, including the Swedish Research Council, the US Air Force Office of Scientific Research, the Royal Physiographic Society of Lund, the Crafoord Foundation, the Wenner-Gren Foundation, the Swedish International Development Agency, Toyota Research and Development, and the University of Lund.