An Efficiency Analysis of Augmented Reality Marker Recognition Algorithm

Abstract The article reports on the investigation of augmented reality system which is designed for identification and augmentation of 100 different square markers. Marker recognition efficiency was investigated by rotating markers along x and y axis directions in range from −90° to 90°. Virtual simulations of four environments were developed: a) an intense source of light, b) an intense source of light falling from the left side, c) the non-intensive light source falling from the left side, d) equally falling shadows. The graphics were created using the OpenGL graphics computer hardware interface; image processing was programmed in C++ language using OpenCV, while augmented reality was developed in Java programming language using NyARToolKit. The obtained results demonstrate that augmented reality marker recognition algorithm is accurate and reliable in the case of changing lighting conditions and rotational angles - only 4 % markers were unidentified. Assessment of marker recognition efficiency let to propose marker classification strategy in order to use it for grouping various markers into distinct markers’ groups possessing similar recognition properties.


I. INTRODUCTION
Augmented reality (AR) is one of main future technological tools.It is more and more successfully applied to many areas of life [1], [2], for example: learning [3], playing games, in a military, architecture, science, medicine [4], and in mobile technology [5] - [7].Augmented reality can inject a joyful element/detail into everyday product.It adds a sense of excitement to the process which appeals to a greater number of customers, and this can be the aim of marketing [8].
AR system without accurate image registration will not be accepted in many applications [9].So one of the most important problems in AR to be perfected is real and augmented view matching, i.e., linking of different scene coordinates and their object coordinates with general coordinates [10].Moreover, in order to reach the most effective AR results it is also important to assess the camera and used locations of objects.The features for the analysis usually are taken from view textures or collected data from sensors, such as: GPS [11], compass, gyroscope, magnetometer and linear accelerometers [12].Non-marker AR systems are being oriented by objects, planes, their movements in the images although these kinds of AR systems are less accurate [1].
One of the most frequently used approach for image analysis in obtaining the necessary data are markers, providing benchmarks and distinguishing internal marker elements according to which the objects can be identified and focused on.To add more, markers are important AR tool because of their high degree of accuracy to capture [1], [4], [13].The system should also be able to calculate the position of the camera using the detected marker.Four known points are sufficient to calculate the pose of a camera uniquely [14] and the simplest shape to acquire them as a square [1].Also they allow pinpointing the coordinates for the environment and localizing the objects to be augmented [15].
It is of primary importance to define the conditions under which the AR system will operate correctly and to analyze the marker recognition algorithm effectiveness condition either [1], [13].Several factors are available which can be analyzed while correctly identifying markers, i.e. the distance to the marker [13], [16], rotational angles and lighting conditions [2].
Those are the reasons why the markers rotational angles, different conditions for lighting with even/uneven light and shadows are observed and investigated in the following material.The recognition of 100 different square markers rotated by angle from −90° to 90° along the two x and y axes is performed.Under these conditions efficiency of marker recognition algorithm is assessed.The efficiency of marker recognition index is proposed and basing on it the markers are divided into three classes.The recommendations for markers internal elements textures are presented.
Throughout the research an open source library for different functions is being used.Open source software is free and built by passion communities of developers, and that makes innovations appear faster than closed source software.Another great reason to choose open source libraries is flexibility, because anyone can insert new source and adjust the algorithm to adapt the necessary operating conditions.And finally an open source model can result in powerful, secure, useful, industrial-strength software.For example, open source library NyARToolKit [17] is constantly improved and widely applied therefore it was used here, too.

II. MARKER RECOGNITION ALGORITHM INVESTIGATION TECHNIQUE
In order to describe investigation technique of AR marker recognition algorithm primary we need to give comments on employed markers and their identity, then the outline selected marker recognition algorithm, afterwards present created virtual environments and finally in details present created AR system implementation.
In [1] it is shown that square marker with a thick border is the most accurate for marker-based systems.This is because of: • marker form -can be easily found even using different methods or algorithms; • color contrast -even with different cameras color differences are easily noticed; • border type -is more robust in most of the situations which can be applied.Square marker can be created with different number of internal elements (4 × 4, 5 × 5 and up to 177 × 177) which directly influences the processing time.That is why in current research we used 4 × 4 internal elements markers taken from the University of Utah developed databases [18].A composition frame of used markers is shown in Fig. 1.
All the markers have the same thickness border.Each employed marker has the ID assigned according to the Fig. 2 (where the first ID digit is the row number, the second -the column number), resulting in ID from 00 to 99.
All the markers are resistant to turns along z axis, i.e., turning the marker onwards (upwards) will not influence marker recognition accuracy [2].

B. Marker Recognition Algorithm
The marker recognition algorithm is based on NyARToolKit library which roots from ARToolKit library that was adopted for Java programming language, as well as for other programming languages like C # and operating systems (e. g.Android) that support virtual machines.The main advantage of the use of NyARToolKit library is in the fact that here user viewpoint tracking problem is solved.Functions from the library allow tracking the view angle without losing the virtual object.This ensures a view of clear augmented representation.
Original marker recognition algorithm was proposed in [16].Let us briefly comment its main processing stages.The algorithm starts with the input of image which afterwards using a lighting threshold is binarized and labeled.After that, when component list is received, starts contour detection.Based on contour detection results, sub-pixel corner detection is performed and found patterns are normalized.Based on camera intrinsic parameters and found components list marker regions list is built.Then a template matching is performed.Marker templates are taken from in advance prepared individual .pattfiles [19].Marker size and lines parameters list taken from sub-pixel corner detection is processed to compute homography.After applying camera transformation markers transformation is received and used for final recognition step.In the case of successful recognition the recognized marker's ID number is output, otherwise recognition failure is indicated.

C. Virtual Environment Setup and Implementation
Virtual scenes prepared for experimental investigation are shown in Fig. 3.The following objects (with their possible coordinate systems) can be seen: the scene, the marker with ID 00, a camera and light sources.In order to simplify image processing, camera coordinate system was reduced (fixed).
During the experimentation four virtual scenes different in illumination were considered: 1. Light -with natural ambient light (Fig. 3, a) [2]; 2. Twilight -with uniformly distributed shadow and blurred ambient lighting (Fig. 3, a). 3. Shadow 1 -with a bright light source from the left side (Fig. 3, b); 4. Shadow 2 -with the blurred light source from the left side (Fig. 3, b); Processing in virtual environment is realized as follows.Every 5-25 ms marker (starting from marker with ID 00) is rotated by 1° along x axis from −90° to 90° in the first virtual environment scene Shadow 1, after that the rotation is performed along y axis.Next marker is selected and processed with a similar sequence of actions.After all 100 markers processing the recognition results are recorded in a matrix of size 181 × 200 elements (each element holds recognized marker index or failure sign).The chosen directed light source is from the left side of camera.Permanent place is created to determine the changes in characteristics of lighting intensity while the degree is changed.Afterwards virtual environment scene is changed and all the processing is repeated.

D. Augmented Reality System Implementation
Augmented reality system can be created using computer and various gadgets with camera.Most popular gadgets for AR systems are: head-mounted, eyeglasses, contact lenses, virtual retinal display, handheld.In reality however, preparation for the test is difficult, because it has to have guaranteed coordinate system, completely controlled lighting conditions, maintain rotational angles along both axes and accuracy (every one degree), and all used tools errors shall not exceed the permissible.That is why, e.g., in [20] only few rotation angles (30°, 45°, 60°, 75°, 90°) were observed.All of the above mentioned problems are minimized or diminished if AR system testing is performed in a virtual environment.For investigation purposes created augmented reality system involves the following functional parts: 1. Image loading.
3. Image processing with features extraction.4. Augmented video data loading.5. AR system video streaming.Each part of the system according to its own operating specifics may be modified or revoked, depending on the desired results.
One of the ways to develop this kind of augmented reality system is shown in flowchart at Fig. 4. At Steps 1 and 2 libraries are initialized and image parameters are inputted.At Step 3 100 markers (shown in Fig. 2) are loaded.At Steps 4-7 virtual environment is generated, markers are rotated and processed (details were already discussed in Section II.B).
At Step 8 marker recognition is preformed (see Section II.B).At Steps 9-11 the coordinates and the object (in this case simple colorful cube with marker ID on top) are drawn, by showing if marker is detected and if detection is correct.At the final Step 12 all the data about markers recognition is written to a file.

A. Individual Marker Recognition Results
Experiments were carried out in four different virtual environment scenes.A total number of created marker images were 144,800, calculated by:   Fig. 5 presents the results on the total number of correct recognitions of individual markers evaluated under rotation (part a -along x axis, part b -along y axis) in four virtual environment scenes.
Table 1 summarizes the same investigation results presenting the total number of correctly recognized markers in four scenes while rotating markers along x and y axis.
The influence of the environment on the algorithm precision is small, as differences of the total number of correct recognitions of individual markers varies only up to one unit.Most of the markers (72 % in x axis and 73 % in y axis rotation direction) are resistant to the changes in all four environments (shown as dots in Fig. 5).
Markers were correctly identified by turning them in the range from −87° to 87°, an average of unrecognized rotational angles is just 7° per marker out of 181°.It could be noted that in all four considered virtual environment scenes, the same markers (see Fig. 6) were found to be the best (ID 03) and the worst (ID 29).

B. Assessment of Marker Recognition Efficiency
Minimal angle along which rotated in all directions (along x and y axis) in a specific virtual environment scene e marker m is still recognized should be expressed by: ( here: and α β are angles of rotations along x and y axis correspondingly; e -virtual environment scene index; m rtrue markers m recognition result (1 -recognized; 0unrecognized).
Maximal angle along which rotated in at least one direction (along x and y axis) in a specific virtual environment scene e marker m is still recognized: m m e r e Keep noted that there is no guarantee that specific marker rotated by angle from max ( ) m m e e +δ − δ -most optimistic (results will be acceptable at least in one rotation direction).In order to take into account variability of recognition results in all considered virtual environments let us introduce two more derivative parameters: ∆ is a maximal angle along which rotated in all considered directions and virtual environment scenes marker m is still recognized.Results computed according to (4) are presented graphically in Fig. 7.
Then angular discrepancy (sort of risk) associated with particular marker to be unrecognized could be expressed as: However such approach will yield ambiguous results (see Fig. 8, where ∆ mp chaotic changes are noticeable).
More consistent results could be achieved by two-fold sorting, namely: [ ] Marker sorting results according to (7) are illustrated in Fig. 9.The angular discrepancy associated with particular marker presented in Fig. 9 could be accorded with one more parameter -relative failure index -the total number of angular positions in which investigated recognition algorithm fails to give correct recognition results: The relative failure index aligned with previous data is presented in Fig. 10.

Accumulative sum of the relative failure index
is not linear (see Fig. 11) thus could serve as indicator for markers division into classes.

D. Marker Classification Results
According to the proposed strategy for markers classification, all considered marker sets can be divided into three marker classes: A, B and C, rated by marker recognition efficiency in decreasing order, correspondingly.A marker class is the most reliable, while B and C marker classes possess some associated risks.
Boundary between A and B marker classes is found based on most conservative criteria -min where L 1 and L 2 are two linear functions with corresponding parameters a n and b n , that fit by not overlapping pieces
here: N M -the number of used markers N M = 100; N R -the number of rotations, N R = 90 • 2 + 1 = 181; N A -the number of axis, N A = 2; N E -the number of environment scenes, N E = 4.

Fig. 5 .
Fig. 5. Limits of the total number of correct recognitions of individual markers evaluated under marker rotation in four virtual environment scenes.
as most conservative (guaranteed in all considered directions),

Fig. 7 .
Fig. 7. Limiting angles along which rotated in all considered directions and virtual environment scenes markers are still recognized.

Fig. 9 .
Fig.9.Marker recognition results sorted using two-fold criteria: (4a) and(5) .Note: marker IDs are shown pair wise at the bottom and the top of axis; boundary between A and B marker classes is indicated by dash line.

Fig. 8 .
Fig.8.Marker recognition results sorted using single criteria (4a).Note that marker IDs are shown pair wise at the bottom and the top of axis.

Fig. 10 .
Fig. 10.Marker recognition results sorted using three-fold criteria: (4a), (5) , and (8).Note: marker IDs are shown pair wise at the bottom and the top of axis; relative failure index Fm values are indicated at the right side of axis.

∆
values individual marker's recognition efficiency can be assessed.C.Markers Classification StrategyMarker recognition results according to min m

Fig. 11 .
Fig. 11.Accumulative sum of the relative failure index dependency on marker recognition results sorted using three-fold criteria.Note: marker IDs are shown pair wise at the bottom and the top of axis; the best LMS approximation by two linear functions is indicated by solid lines; boundary between B and C marker classes is indicated by dash line.
worth for sorting factor into account sorting expression(7) can be augmented by: to the considered marker set b AB = 11 (see Fig.9where found boundary is indicated by dash line).Boundary between B and C marker classes is found by the following optimization procedure:

Fig 12 .
Fig 12. Markers classified into three classes according to the recognition efficiency: A class -most reliable; B class -reliable; C class -most risky.

TABLE I THE
TOTAL NUMBER OF CORRECTLY RECOGNIZED MARKERS IN FOUR VIRTUAL ENVIRONMENT SCENES WHILE ROTATING MARKERS ALONG X AND Y AXIS