Automatic omni-directional, high-speed, pan-tilt platform

Shooting videos is currently a task that has to be performed by human operators, i.e., the operator needs to direct the camera toward the given target and keep the target in the field of view. When the target is moving at high speed, however, it can be very difficult to maintain the correct pointing direction for the camera. One solution to this problem is to automatically control the camera's pointing direction with the use of automated pan-tilt platforms (and to use associated image processing methods). However, the slow response speeds of such mechanical approaches (known as ‘gaze control’) are a major drawback.

A simple approach that has previously been employed to speed up the response is the use of high-power motors that have the structure of a conventional pan-tilt platform.¹ The response time with this method, however, was not less than 50ms because of the considerable inertia of the mechanical structure. Another approach is to actuate the mirrors instead of the camera, and to control the gaze direction by changing the reflection angles (because the inertia of the mirrors tends to be much smaller than that of the camera). Indeed, in a commercial gaze controller (the Ocular Robotics RE01) a moving mirror (supported by a gimbal mechanism) is used to achieve a gaze-control range of 360° and a minimum response time of 2ms.² Even this response time, however, is limited by the inertia of the rather complex gimbal structure.

In another solution to the problem, we have previously proposed a 1ms-response-time automatic pan-tilt system that is based on a high-speed gaze controller and is known as a saccade mirror.^{3, 4} This mirror is composed of two rotating mirrors and a pupil-shift lens, as shown in Figure 1. The pupil-shift lens is used to transfer the entrance pupil so that it is in front of the camera lens, i.e., so that even a small mirror can reflect all of the incident rays that enter the camera lens. We have also previously reported a prototype of this system that had a practical viewing angle of up to 20° and a high-speed response time of 3.5ms.⁴ The inherent structure of this saccade mirror, however, means that it has a limited gaze-control range (up to 60° for each axis). This limitation can thus be an obstacle for some applications. For example, if a single saccade mirror were used to track a ball in a football field that has a width of more than 100m, it would be necessary to place the mirror at least 87m from the edge of the field (i.e., to cover the entire field). Therefore, depending on the structure of the stadium, there may or may not be sufficient space for placement of the saccade mirror system.

Figure 1. Schematic structure (a) and photograph (b) of the originally proposed 1ms automatic pan/tilt system, which consisted of two rotating mirrors and a pupil-shift lens.

In our more recent work, we have thus developed a new structure that enables the gaze direction of our mirror to be controlled within a 360° pan-angle range.^{5, 6} In this structure—the ‘Saccade Mirror 3’—we use three rotational mirrors to overcome the gaze-range limitation of our previous design. The basic structure of this system is illustrated in Figure 2(a), i.e., we use the first and second mirrors (M1 and M2) to mainly control the tilt rotation of the gaze vector, by regulating the incident angle of light to the third mirror (M3). We thus use M3 to mainly control the pan rotation. We also schematically illustrate this gaze-control mechanism in Figure 2(b).

Figure 2. (a) Schematic diagram of the Saccade Mirror 3 basic structure and (b) its gaze-control mechanism.⁶The system includes three rotational mirrors (M1, M2, and M3) that are used to control the tilt rotation of the gaze vector and the pan rotation. φ: Azimuth angle of the gaze direction. θ: Angle between the M3 plane and the rotational axis.

Although only two degrees of freedom (DOFs) are needed to control the gaze direction, our proposed structure actually has a third—redundant—DOF (because of the three independently rotatable mirrors). This redundancy arises because our system has singular postures, i.e., some DOFs are degenerate. For example, in the case where φ=0, π (φ is the azimuth angle of the gaze direction) and the angle of M1 is fixed, both M2 and M3 can only be used to control the gaze direction in the pan sense (i.e., not the tilt sense). Likewise, the DOFs for M1 and M3 are degenerate in the case where φ=±π/ 2. The redundant DOF of our proposed structure is therefore essential for avoiding a singular configuration.

In our work, we have also developed a prototype of the Saccade Mirror 3 (see Figure 3) to verify the design principle. This prototype unit was composed of a pupil-shift lens and three automated rotational mirrors. We designed the pupil-shift lens so that it was coupled with a camera lens. In addition, this pupil-shift lens had an angle of view of less than 20°, and it was compatible with a full high-definition resolution imager (1920 × 1080 pixel resolution) if the camera had no aberrations. All three mirrors had response times of less than 5ms, which meant that the response time of the whole system was also less than 5m (because each mirror is mechanically independent). We also equipped the prototype with two camera ports that shared the same gaze direction. We achieved this by splitting the optical path with a half-mirror. In this way, the high-speed vision system we used for image processing and the professional video camera we used for broadcasting could simultaneously share the same field of view.

Figure 3. Photograph of the Saccade Mirror 3 prototype.

To demonstrate the operation of our prototype, we thus coupled it with a high-speed vision system (which can acquire and process 1000 images per second). We then used the whole system in two separate tests, i.e., to track both a table tennis ball during a rally and a flying drone. The experimental setups for both these tests, as well as the results, are illustrated in Figure 4. We found that the prototype was able to successfully track both the high-speed targets over a wide area. Videos of these experiments are also available online.^{7, 8}

Figure 4. Illustration of the experimental setups (left) in which the Saccade Mirror 3 prototype was used for the tracking tests, i.e., to track (a) a table tennis ball during a rally and (b) a flying drone. The markings A, B, C, D in (b) indicate the flight path of the drone. The images (right) of the tracked objects were captured at 1000 frames/second, and are rotated in the figure so that they appear straight. t: Time.

In summary we have developed a new pan-tilt platform, known as Saccade Mirror 3, that can be used to automatically control the gaze direction of a camera over a 360° pan-angle range, and with a fast response time. We have also successfully verified our design with a prototype in two separate tracking experiments. In our ongoing activities we are working to apply our system to more specific applications, such as for sports broadcasting, and for vision systems in mobile robots and automobiles.