2.5D space: restoring image depth parameters from a single camera

Please, note that the article is automatically translated from Russian into English, so the translation may not be accurate.

Vadim Kharlanov

Such a stunt like tuning camera settings for optimum performance with a set of demonstration videos are actively used by manufacturers for advertising purposes, as it allows to effectively demonstrate the full range of intelligent camera features. In practice, the client is often disappointed by the results of the initial phase of the product. But is it all a bad thing? Let's try to understand.

The most urgent problems of the existing video analytics, which, with varying degrees of success trying to fight the developers of "smart" cameras are:

• to minimize the error probabilities of the first (pass system critical events) and the second kind (false positives) to achieve a high degree of confidence in the results;
• universality of the algorithms. It is important that the system functioned equally well as a demonstration installations, cameras, provided by the developer as an advertisement of its product, and in actual field conditions.

Specifying the spatial calibration

Quite often in the camera settings include the developer tools for the job of spatial calibration. The meaning of calibration is to determine as precisely as possible the actual relative position of the camera and the scene that this camera covers. For the full definition of the depth of the scene must have at least a pair of cameras, so for one camera introduces an element of interactivity, ie the user is encouraged to "help" system to calculate correctly the spatial component, using a defined set of tools. Unfortunately, very often the implementation of this process leaves much to be desired, because it requires the operator of the camera specific skills and knowledge from the field stereometrical and optics. The consequence is that as the settings remain the default settings, which, although they allow the intellect to operate the camera quite well, but at the same time do not reveal the full potential of the algorithms.

Benefits of calibrated cameras

Properly calibrated camera can operate with such features as:

• the actual size of objects present in the scene;
• real distance between the camera and scene elements, as well as between elements of a scene;
• the actual speed of objects.

Advantages are obvious: the security system it is possible to classify objects using their physical attributes.Consider a few examples from the field of automated perimeter.

Setting the minimum threshold height of an object, you can reduce or eliminate the priority registration system of small animals (squirrels, rabbits, foxes, dogs) and birds (Fig. 1).

Fig. 1. An object "rabbit" system is not classified as a violator of the perimeter of the parameter "minimum"

Establishing the maximum permissible speed allows you to record only the people and ignore the fall in the vision camera moving vehicles, as well as low-flying birds, bats, etc. (Fig. 2).

Fig. 2. Object "bird" is not classified by the system as the perpetrator of the perimeter of the parameter "maximum speed"

Determine the maximum allowable height of an object, you can effectively "fight" with objects that are present on stage, but moving out of the ground plane: insects on the lens, a bird flying in close proximity to the camera, etc.(Fig. 3).

Fig. 3. Object "insect" system is not classified as a violator of the perimeter of the parameter "maximum size"

In addition to all of the above spatial calibration has one (at first glance not obvious) advantage, which should also be mentioned. In some cases, it can significantly improve system performance. Total gain depends on the specific installation (defines the scene, which is analyzed by the camera) and on the specific implementation of intelligent algorithms. Most obviously illustrates the benefits of using spatial calibration of Fig. 4, which shows the registration algorithms, the horizon line. It is calculated in the calibration process and allows you to narrow your area of interest to the screen area, located below the line.

Fig. 4. An example of a visible horizon line (green line). The region above this line can be excluded from consideration analytical algorithm (highlighted in red)

Methods of calibration

Simple geometrical calculations show that to calculate the depth of the scene in case of a camera, provided that the known characteristics of the matrix and the camera lens, it is enough to have one of the following data sets:

• height and angle of the camera
• two straight lines parallel to each other in the real world, and the amount equal to the actual distance between these two lines.

There are other possible combinations of these parameters. So, if there are two parallel lines, then the calibration can successfully undertake, knowing only the height of the assembly chamber. However, the practical use of these approaches significantly complicates the understanding of the average user calibration procedure, which is already not trivial.

The advantage to this method is its simplicity - a user is required to enter only two values. However, it is applicable only to a relatively simple scene, where there is no significant extended space, such as indoors. The main problem - a quick set of measurement error with increasing distance from the camera. To reduce this error should be set camera angle with a truly astronomical accuracy.

The second method is most often used by the creators of calibration systems, despite the fact that it is more complicated from a user perspective. The difficulty lies in the need to highlight the image from the camera at least two lines that are parallel in the real world. The image data lines will converge, and the line, which owns all the points of intersection of all the "parallel" lines form a horizon line. As a set of parallel lines may have different extended objects that are present on stage: fences, walls, buildings, roads, etc. (Fig. 5).

Fig. 5. The fence forms a pair of lines that are parallel to each other in the real world

Explain to the user exactly what he has to select the image, very difficult. Even I'm not entirely sure I can adequately reveal the main thrust of the procedure described in the preceding paragraph. Therefore, most manufacturers go for a little trick and introduce the concept of "markers of height." To calibrate the user in most cases, you must specify only a couple of markers, marking the height of one and the same object at different distances from the camera lens, as well as giving the actual height of the marked object.

Fig. 6. The relationship between the parallel lines and markers of height

The illustration (Fig. 6) that by its very nature this method is no different from the above-mentioned method of allocation of parallel lines. Static objects can not always be present at the scene, so the use of markers of height can be calibrated by using the moving objects - their size observed in the process of displacement (Fig. 7).

Fig. 7. Example of calibration markers, height, where the basis for calibrating selected dynamic object

Problems of calibration

There are a number of features that may adversely affect the quality of calibration.

Firstly, it is worth noting that much depends on a specific user. Accuracy and precision placement of markers given numerical parameters are given to all operators. Oversights in the alignment determine those errors, which will be formed in the process of calculating the prospects.

Secondly, in order to maximally simplify the calibration procedure, developers often it is assumed that the reviewed camera scene has a smooth terrain without abrupt level changes. Thus, the camera will correctly handle the scene just within the same plane. For complex, multi-dimensional scenes (Fig. 8) or the calibration will be effective within one of the most important in terms of monitoring, plane, or do not apply.

Fig. 8. An example of a complex scene. Calibration is complicated by the presence of three different planes (highlighted in yellow, blue and green)

Examples of products

Fig. 9. IOImage Toolkit for interactive definition of scene depth

Commercial product, comprised of means for performing spatial calibration is an application company IOImage (Fig. 9). It is equipped with sufficiently powerful set of additional tools for interactive layout. Slight disadvantage of the application can assume the complexity of the proposed development tools, which to some extent offset by the manufacturer of their teaching demo.

Fig. 10. Tools for interactive devices MagicBox determine the depth of the scene

Another example of a software product that illustrates the different approach to the spatial calibration and implementing the principle of simplicity and minimalism can be considered tools included with the device software MagicBox (Fig. 10). All that is required from the user, is to place two markers of height, that to give greater associativity are presented in the form of "little people" - all very easy.

"Smart" cameras will justify the expectations

To date, given the level of existing hardware solutions and ongoing scientific developments, the use of spatial calibration - is perhaps the only real way to make intellectual stuffing chamber to become truly universal and highly effective means of self-monitoring, analysis and prevention. It is also hoped that this article will help at least to some extent to make a more justified as the hopes and expectations of buyers of intelligent video surveillance devices for these products, as well as hopes and expectations of manufacturers of "smart" devices to their users.