A better way of seeing

QUESTION: What do E. coli, an apple bruise, pesticide on a tobacco leaf, and the smell of bad fish have in common?

ANSWER: They all have a spectral signature, a fingerprint in visible or invisible reflected light that reveals their presence. You can't see fingerprints, but if you know where and how to detect them, the information they provide is invaluable.

The same is true of a spectral signature. Once you know what to look for and how to see it, you can unlock a treasure chest of useful information on food safety and quality.

Just as detectives use ultraviolet light or dusting powder to find fingerprints, food producers have turned to hyperspectral imaging, or HSI, to examine fruit, vegetables, meat, and even crops still in the field. HSI works in spectrums beyond the capabilities of the human eye and at speeds that would be impossible for any worker to match. It can detect a stone on a conveyor belt full of hazelnuts, determine if a cut of beef is properly marbled, and see bacterial contamination on a cut of chicken in real time. It returns consistent results that are as good as the sensors and algorithms that power it without getting tired, distracted, or sick.

"We are processing each individual item of the product on the conveyor," said Fatih Ömrüuzun, managing director of Visratek, a research and development company in Turkey specializing HSI applications for food safety and quality. "In the past, our customers were taking some samples from the product coming from the field and doing tests in the laboratory, which takes a day or two, then deciding if the entire product lot will be accepted or rejected. This is why hyperspectral imaging is critical. We are analyzing each individual hazelnut on the conveyor belt, not just some samples."

HSI is a technology that combines aspects of both machine vision and spectroscopy and is well suited to identifying desired characteristics of an object. In most applications, light from a natural or artificial source is reflected off the surface of the subject then captured and analyzed by the hyperspectral imager.

HSI creates a three-dimensional data cube with x and y axes as spatial dimensions of the subject, and spectral information as the third dimension. Think of a data cube as a book with the full-color image on the cover. The x and y axes would be the length and width of the cover. The spectral dimension is the number of pages in the book with each page representing a unique wavelength. In this metaphor, an image in the visible spectrum would be just a three-page book: red, green, and blue, with the full-color image on the cover. A hyperspectral imager, on the other hand, would create a book with hundreds of pages representing a contiguous portion of the electromagnetic spectrum. What sets HSI apart from machine vision is its ability to analyze the spectral characteristics of each pixel on the spatial plane, like taking a tiny core sample to examine the spectral fingerprint of an image in exacting detail. A study that used HSI to scan fish for E. coli contamination using select wavelengths (424, 451, 545, 585, and 610 nanometers) where the bacteria is visible, for example, was successful in detecting that bacteria about 90 percent of the time.

Other recent studies point to the expanding list of possible applications. HSI has been shown to accurately assess the total viable count of micro-organisms in beef, fish, chicken, and dried sausages; the volatile basic nitrogen content (a measure of freshness and decomposition) in fish, pork, chicken, duck, and shrimp; and the fat content, protein, and moisture in beef, lamb, and other meat. It can also quickly determine the soluble solid content—a measure of carbohydrates in many types of fruit and vegetables—and can discriminate among textural features that influence characteristics such as chewiness, gumminess, cohesiveness, and other attributes in a variety of foods.

***

The term "hyperspectral imaging" was first used in 1985 in a paper describing the early results of imaging spectrometry in support of the Landsat 1 satellite, launched in 1972. It was an idea ahead of its time, or at least ahead of the technology of the time. As Alexander Goetz, a pioneer in the development and use of HSI at the University of Colorado, Boulder, wrote in a 2009 paper summarizing 30 years of technological development, "When the first field spectral measurements were conducted in the early '70s and the promise of imaging spectrometry became apparent, the technology was not advanced enough for it to be implemented. In spite of the fact that humans had walked on the moon, essentially all image processing was carried out in large centralized computer centers and processing jobs were loaded on punch cards."

Given the technology's origin and development at NASA, it's not surprising that HSI was first used for remote sensing and astronomy. The ability to quickly identify mineral deposits from above is helpful to geologists, while scientists interested in ground cover can use spectral signatures to detect the species of individual trees.

HSI also found extensive use in military surveillance and detection on board planes and drones. In an odd twist, that use made it easily adaptable to the food industry.

"In terms of the harshness of the environment, a fast-moving aircraft or a predator drone are very similar to what you find in food processing applications," said David Bannon, CEO of Boston-based Headwall Photonics, a supplier of HSI systems for both military and food-processing, as well as many other demanding applications. "A pork-processing environment or a chicken-processing plant are very tough environments. They are damp and humid, there's a lot of vibration, a lot of things moving, a lot of caustic chemicals being used to clean. A lot of our experience from military and defense is in systems that cannot fail. We put together hardware-software solutions and moved them into a commercial environment where there's a lot of money at stake in a processing environment if your system goes down."

Headwall software analyzes the hyperspectral data and sends coordinates to a robot down the line to take action. Credit: Headwall Photonics

***

Three trends are shaping the future of HSI in food safety and quality: faster, cheaper, and easier to use.

A faster HSI imaging system would open a window on chemical processes not visible to current systems. Researchers in Spain, for example, recently combined dual-comb spectroscopy and video-rate imaging, making it possible to quickly acquire vast amounts of spectral information of an entire scene.

Dual-comb spectroscopy uses two optical sources, called frequency combs, that emit a spectrum of perfectly spaced frequencies, like teeth on a comb. Combined, the combs produce an interferogram that can be used to assess the spectral characteristics of a subject very quickly. In fact, the key development of the Spanish researchers was slowing down the interferogram so that a video camera could read it.

"The idea is quite straightforward, really," says team leader Pedro Martin-Mateos, an assistant professor at the Universidad de Carlos III de Madrid. "Dual-comb spectroscopy has been in use for a long time. The main issue is that the signals you need to read, you need to measure, have rates in the megahertz. So what we did is stretch dual-comb signals in a way in which a camera can detect. Before you had interferograms that repeated at one microsecond, and now we have interferograms that stretch up to one second, so in that way, with a very, very low frame rate or acquisition rate, you can detect interference between two combs and you can therefore perform dual-comb spectroscopy."

They used relatively inexpensive fiber-optic equipment to build the system, which splits a laser beam and sends outputs through acousto-optic modulators. The modulators allowed them to offset results by an arbitrarily low frequency, which slowed it down enough for the camera to record.

"There are acousto-optical modulators, a monochromatic laser, and fiber optic components from the communication industry, and that's pretty much it," said Martin-Mateos.

With the system tuned to the near infrared, the researchers were able to see ammonia escaping from a bottle. Martin-Mateos said the instrument could be adapted to terahertz or millimeter regions, increasing its usefulness in food inspection.

"There have been quite a lot of studies proving that the terahertz wave can be used, for example, to detect residues in food," he said. "There are quite a lot of papers proving that you can detect the quality of food, and even different species of wheat."

Currently, one drawback of HSI imagers is cost, which can easily top $100,000. Researchers at Duke University have taken a step toward putting an entire spectral imager on a single chip that would cost less than $100 and be the heart of a very fast, capable system.

The idea is based on plasmonics, nanoscale physical phenomena created when light interacts with metal. By creating silver cubes about a 100 nanometers wide and placing them on a transparent film nanometers above a thin layer of gold, researchers were able to trap frequency-specific energy created when light strikes the silver cubes and excites the electrons inside. By manipulating the size of the nanocubes and their distance from the gold layer, they could control the frequency of the nanocube's response. Changing the spacing between the silver nanoparticles controlled the intensity of light absorbed. Thus, changing the size and spacing of the cubes allows the researchers to tune the system to respond to different wavelengths.

That's an interesting phenomenon, but to make it an HSI system, the team needed a way to "read" the information from the nanocubes.

A metasurface of nanoparticles integrated with a pyroelectric AIN 100 nm film that is placed underneath the metasurface. The heat is initially concentrated in the area between the nanoparticle and gold film, then diffuses down, changing the polarization in the pyroelectric crystal and generating a voltage that is proportional to the temperature change in the material. Credit: Maiken Mikkelsen

"We chose these sort of almost forgotten materials, kind of old-school thermal detectors that are called pyroelectrics. They produce an electrical current in response to a temperature change and allow us to use a very simple architecture," said Maiken Mikkelsen, associate professor of electrical and computer engineering at Duke University. "We put our nanostructures on top of the pyroelectric material, in this case aluminum nitride. When light hits the material, it heats up and generates a current."

At that point, you have an on-chip HSI imager. Mikkelsen's team is still working on the chip, but nothing about its basic architecture is particularly complicated to manufacture. It would cost in the "tens of dollars" range, she says.

Mikkelsen's concept is tunable and very fast. The previous record for response time on any spectrally selective thermal camera, pyroelectric-based or not, was 337 microseconds. Her chip showed response times of 700 picoseconds.

"We saw speeds of five orders of magnitude higher compared to other spectrally selective detectors," she said.

A cheap, lightweight, chip-based HSI mounted on a drone and flown over crops could examine the spectral fingerprints of individual plants to precisely determine which ones need fertilizer, pesticide, or water. That kind of "smart" agriculture already happens, but it's extremely expensive in some countries and unavailable in others.

Of course as Headwall CEO Bannon notes, HSI technology is of little use to food producers if it can't be deployed, which is why Headwall's MV.X imager emphasizes end-user programmability: it offers end users the ability to change spectral parameters to reflect their own needs. "It can be easily tuned and updated to changing application requirements using a remote connection, and deliver real-time output, programmed in real time, based on the different rules for grading," said Bannon. "It is not a deployable or scalable strategy if we have to send an engineer with a PhD in optics out with every system."

The real promise of HSI to improve the safety and quality of our food will be realized when food producers can use the vast amount of information in a spectral signature as routinely as detectives read a fingerprint. Recent advances show that day is closer than ever.

Bob Whitby is freelance science writer based in Fayetteville, Arkansas.