3D human gesture recognition using integral imaging

Optical image sensing and visualization technologies in 3D have been researched extensively in fields as diverse as TV broadcasting, entertainment, medical sciences, and robotics.^1–4 One promising technology, integral imaging, is an autostereoscopic 3D imaging method that offers a passive and relatively inexpensive way to capture 3D information and visualize it optically or computationally.^5–7Integral imaging belongs to the broader class of multi-view imaging techniques that allow depth analysis from three points of view: stereo, time-of-flight, and structured-light strategies.⁸ Integral imaging has also been used for classification tasks.⁹ However, we are the first to apply it to action recognition.^{10, 11}

Integral imaging provides the 3D profile and range of the objects in a scene using an array of high-resolution imaging sensors or in a synthetic aperture mode (see Figure 1). When a single sensor captures multiple 2D images, it is possible to obtain larger field-of-view (FOV) 2D images. In the synthetic aperture integral imaging mode that we used, a series of sensors are distributed in a grid, or a single sensor is moved to the positions in the grid. The horizontal or vertical distance between two of these positions is called the pitch (p). The 3D image reconstruction can be achieved by computationally simulating the optical back-projection of the elemental images. In Figure 1, c_x and c_v are the horizontal and vertical sizes of the sensor and f its focal length. We used a computer-synthesized, virtual pinhole array to inversely map the elemental images into the object space (see Figure 1). Superimposing the properly shifted elemental images created the 3D reconstructed images.

Figure 1. Synthetic aperture integral imaging acquisition and computational reconstruction method. Pitch (p) is the distance between the sensor centers (left), c_x and c_v represent the horizontal and vertical sizes of the sensor, and f is the sensor focal length (right).

Our methodology is based on acquiring 3D videos of hand gestures using an integral imaging system formed by an array of 3×3 cameras. We analyzed the potential of gesture recognition using 3D integral imaging and compared the performance to 2D single-camera videos. We processed sectional reconstructed representations of the objects in the scene using gesture recognition strategies. We believe the experiments provide evidence of the feasibility of gesture recognition with integral imaging.

Our setup included a 3×3 array of Stingray F080B/C cameras. Using an IEEE 1394 high-speed communication serial bus, we captured nine synchronized videos at 15fps and a resolution of 1024×768 pixels. We subsequently rectified the acquired videos.¹² We then acquired two performances of three different gestures from 10 people. We captured the three gestures, which were made by extending the right arm: left, deny, and opening and closing the hand (see Figure 2).^10,11 Each camera lens was focused at a plane about 2m away. The depth of field allowed for all objects and people from 0.5 to 3.5m away to be in focus. The 10 people were about 2.5m in front of the camera array. We acquired their gestures in a laboratory with no other movements. We recorded 60 videos corresponding to the three actions the 10 people performed twice. The 3D volume for its first frame was reconstructed to infer the distance at which the hand was in focus. This distance was then used to reconstruct the volume for the remaining frames. We made the reconstruction from 1 to 3.5m in 10mm steps. Figure 2 shows the reconstructed plane where the action is in focus for the three different gestures we considered.

Figure 2. Three hand gestures used in our research. Left: Left gesture. Middle: Deny gesture. Right: Open/close gesture.

The method for characterizing and recognizing gestures can be summarized as follows:¹³ generating and characterizing spatiotemporal interest points (STIPs) for each video (see Figure 3);^10,11 quantizing the resulting descriptors in a number of visual words (also called the codebook); creating a bag-of-words (BoW) representation for each video using its STIPs and the resulting codebook; and classifying unseen videos from their BoW representations.

Figure 3. Examples of the spatiotemporal interest points (STIPs) detected (yellow circles) at three different frames in the ‘open’ gesture video. The different circle sizes represent each STIP's detection scale.

We generated STIPs and, from them, histograms of oriented gradients (HOG) and of optic flow (HOF). We quantized these histograms into visual words through k-means clustering. We represented each video by creating a histogram of codewords.¹⁴To estimate the gesture recognition performance, we followed a ‘leave-one-subject-out’ protocol.^{15, 16} We chose support vector machines as the pattern recognition method for classifying the gestures.¹⁷Our results showed¹⁰ that integral imaging outperformed acquisition with the central camera of the 3×3 array when comparing the best descriptor in each case (HOF for monocular and HOF+HOG for integral imaging).

In summary, 3D information shows potential in improving the accuracy of human gesture recognition. Integral imaging allows us to reconstruct a 3D scene for only the planes where the gesture preferentially appears. This opens the door to the application of recognition strategies that were not previously possible and eventually to substantially increased recognition capability. Our next step will be to computationally parallelize the entire process so that it can be applied nearly in real time and to use other gesture recognition descriptors that exploit the focusing capabilities of integral imaging.