A camera that never overexposes an image

With the advent of affordable digital camera technologies, photography today is more accessible than ever. However, taking successful pictures in the dark or through a window is difficult for professional and amateur photographers alike, because of the high brightness contrast in the surroundings. To work around this problem, we can use computational photography, specifically, high dynamic range (HDR) photography, which enables imaging of both very bright and very dim light sources, without loss of quality.

Broadly speaking, traditional HDR photography techniques fall into two categories: multi-capture and single-capture fusion. Multi-capture fusion¹ involves taking several images at different exposure levels, and then fusing these together. This approach is less effective when imaging moving objects. Moreover, any movement of the camera can detrimentally alter the results, so a ‘deghosting’ technique (removing the transparency) is usually necessary. Single-capture fusion² requires a camera whose exposure time differs from pixel to pixel. This technique was first proposed 15 years ago, and Sony and Fujifilm have since developed cameras that work on this basis. However, although it is possible to obtain an HDR image in a single shot using this technique, during capture there are nevertheless a large number of over- or underexposed pixels. As as result, spatial resolution is sacrificed, so that there is a trade-off between time and resolution when choosing an HDR approach.

To improve these methods, we have developed an HDR³ technique where the sensor circuit is tweaked to protect the camera from overexposure. Conventional camera sensors are overexposed after receiving an excess amount of light, causing their well capacitors to ‘overflow.’ Our modulo camera solves this saturation problem by resetting the sensor capacitors whenever they get full. More specifically, we give a reset controller to each pixel of the camera sensor. During exposure time, whenever a sensor capacitor is filled, a reset signal is sent to discharge the capacitor of the pixel (or clear the pixel value). In this way, the pixel is always ‘active’ during exposure. If the ground truth signal value is a large number that can be represented by a long string of binary digits, our camera only takes the least significant bits. For example, with an eight-bit camera sensor (a bit depth of 8), whenever a pixel is filled with eight bits of information, its capacitor is reset to zero and starts another round of filling. At the end of exposure, this modulo camera keeps the least significant eight-bit information of all the pixels. Comparably, a conventional camera gives eight 1s, which result in many white pixels. Therefore, our approach preserves more information during capturing. This resetting function is exactly the same as a modulo operation in mathematics, hence the name ‘modulo camera’ for our approach.

Straight after capture, modulus images contain modulo fringes (artifacts) and are therefore not yet ready to be viewed (see the center panel of Figure 1). To recover the latent image, we apply an inverse modulo (or unwrapping) algorithm. Importantly, the modulo fringes only appear between neighboring pixels. Therefore, the recovering algorithm aims to remove spurious jumps in the image. Concretely, the problem is formulated as the energy minimization of a Markov random field (a graphical optimization model), and is solved by a graph cuts (max-flow/min-cut) algorithm. In the solver, the statistical properties of HDR images are also considered to ensure realistic results. In this way, we require only a single capture to create an HDR image, and there is no trade-off between time and resolution. This not only enables HDR photos free of blur, but also allows for the possibility of HDR video.

Figure 1. Images taken using conventional and modulo cameras. Left: An image taken by a conventional camera with a large region of saturation (overexposure). Center: A raw image recorded by a modulo camera. Additional information is preserved in the saturated region. Right: The final image recovered from the raw version using an inverse modulo algorithm.

In short, we have developed an HDR technique that protects a camera from overexposure by resetting its sensor capacitors whenever they receive excess light. We apply an algorithm to images captured by the camera to ensure no modulo fringes are left, and the result is an image without the degradation of overexposure.

There are numerous uses for this technology. No more will photographers—professional or amateur—have to fumble with aperture size and exposure length. The algorithm would enable the user to simply click the camera button and let the computer deal with the exposure problems. Furthermore, cameras serve as visual input for robots, and clarity in all lighting conditions is crucial to robotic vision (although good lighting cannot always be guaranteed). Our future work will focus on applying our real-time HDR camera to real-world applications. For example, when a self-driving car drives into a tunnel, an ordinary camera goes blind. The exit ahead is bright, and the surroundings are dark, so the camera cannot see both inside the tunnel and out of it. A real-time HDR camera could ensure safety in these conditions.