INTEGRATION OF IMAGE-BASED FOG DETECTION WITH AUTONOMOUS DECISION SYSTEM FOR INTELLIGENT ROAD SIGN

: The paper presents the description of the decision system implemented for Intelligent Road Signs. It focuses on the implementation of the novel air transparency analysis system and its integration with the rule system and the speed control infrastructure. Moreover, there are presented issues of making decisions about the content displayed in the case of autonomous and cooperating signs. To reflect more closely on real-life situations, it is assumed that the content presented by the IRS changes dynamically, depending on the road traffic and weather parameters. The IRS system operation was presented using fog detection as an example


Introduction
The development of technical road infrastructure should keep up with new technologies implemented in vehicles in the scope of autonomous decision making and information exchange possibilities. Therefore, it is necessary to develop systems that, being part of the infrastructure, will have similar characteristics as vehicles, allowing autonomy in decision making, tracking and analyzing dynamic environmental changes, communicating with other elements of infrastructure and vehicles. These assumptions are implemented by the Intelligent Road Sign (IRS) which supports both users and road managers in improving road safety [1].
The developed solution is designed to provide a high level of autonomy of road signs in analyzing the situation and making decisions about the message that is displayed on the IRS or passed directly to the driver.
Autonomy of the IRS is implemented through its inference system which makes decisions based on a set of embedded rules (developed on the basis of expert knowledge) and information from other signs. In order for an intelligent sign to make its own decisions, it must analyze the environmental parameters that include both its static and dynamic features. Situation analysis and information provided to drivers are based not only on the knowledge of a single sign, but also on data from other signs. Therefore, the developed system due to the possibility of information exchange between signs, is characterized by so-called collective intelligence system.
Many papers focus on the application of innovative technologies in the field of traffic management, smart roads or intelligent transport systems from different perspectives. With the advancement of the communication technologies between vehicles (V2V) or vehicle to infrastructure (V2I), the improvement of cooperation between traffic control and driving behaviors can be observed. In [21], the issues and challenges facing traffic signs are discussed. In [18], a cognitive wireless communication system for vehicles and roadside devices to exchange road conditions was proposed. It should be emphasized that to support traffic management an expert system is used. For example, in [16] a multiagent road traffic management system for tackling weather problems was introduced. In turn [2], an expert system (with a set of rules) supporting determining speed limits for road sections was described. In [3], the authors investigate the problem of vehicle route planning in a dynamic environment, where travel times are not known exactly, but bounded from below and from above, i.e., they are given as interval quantities. The proposed system obtains traffic parameters on the basis of image processing. It was integrated into a larger system for traffic management. In [5], a platform for self-organizing reliable wireless connections among road signs equipped with innovative displays and power supplies was discussed. The authors described several research and construction problems which should be addressed, such as: effective and independent of weather conditions traffic monitoring based on simultaneous analysis of several types of data representation, development of a method of calculating gradients and histograms of vehicle speed for various types of road situations or traffic topologies. In [6], an unsupervised learning method used for deploying classification algorithms applied to the automatic annotation of road traffic-related events based on noise analysis was presented. In [7], the authors proposed a material colour layer sensor which can be applied on the road surface. The colour of horizontal road markings is changed as a result of temperature change of pavement and ice occurrence. These road pavements as well as intelligent road signs will warn drivers of the presence of ice patches, and thus, contributing to minimize the occurrence of accidents. In [8], the communication system between an interconnected group of intelligent road signs was described. The LTE technology conveys data from each sign, the current status of each road sign, and in the opposite direction: control commands and information to be displayed and presented to drivers, while LoRa technology interconnects intelligent road signs and introduces longrange communication. It should be mentioned that many recent studies have been aimed at Cooperative Intelligent Transport Systems (C-ITS) [20]. C-ITS focuses on the communication between two or more ITS components (e.g., vehicles, personal mobile devices, infrastructure, or central) to share information about an upcoming traffic situation through ad hoc short-range and wide-area communication technologies [1,4,19]. In [14] the authors presented a prototype of an automated traffic control system using the method of road regulation based on an analysis of the influence of delay points. The logical and physical design was carried out, during which the system use case models were built, the activity was determined and the structure of the earned system was formed. In [15] was presented an urban traffic control system, which is designed based on the real-time traffic flow information. Proposed approach combines traffic control theory, application of single chip computer and ultrasonic technology, the design and research of the traffic control system based on traffic. Article control core of the system is the MCS -51 single chip microcomputer, which achieves real-time monitoring by using ultrasonic sensors for road vehicle. In [12] the authors proposed a traffic light control system enabled by a hierarchical multi-agent modeling framework in a decentralized manner. In the framework, a traffic network is decomposed into regions represented by region agents. Each region consists of intersections, modeled by intersection agents who coordinate with neighboring intersection agents through communication. By employing a reinforcement learning algorithm for each turning movement agent, the intersection controllers are enabled with the capability to make their timing decisions in a complex and dynamic environment. In [13] the authors proposed a group-based signal control approach capable of making decisions based on its understanding of traffic conditions at the intersection level. The control problem is formulated using a framework of stochastic optimal control for multi-agent system in which each signal group is modeled as an intelligent agent. In [11] the authors present a multiagent framework that models traffic control instruments and their interactions with road traffic. A multi-objective Markov decision process is applied to model agent operations, allowing agents to form a decision in the context of multiple policy goals. The problem is reformulated by a constrained Markov decision process (CMDP) to enhance the computational efficiency. In [17] the authors proposed an intelligent traffic control system using the Internet of Vehicles (IoV). The vehicles or nodes present in the IoV can communicate between themselves. This technique helps in determining the traffic intensity and the best route to reach the destination. The city map is separated into many segments of equal size and Ant Colony Algorithm (AOC) is applied to the separated maps to find the optimal route to reach the destination. Further, Support Vector Machine (SVM) is used to calculate the traffic density and to model the heavy traffic. The paper is organized as follows. In Section 2, the operation of the intelligent road sign was presented in detail. A new architecture of the system based on innovative components was proposed. The main feature of the architecture is three types of relationships between the system components: autonomous, cooperational and supervisory. In Sections 3 and 4 , the original rule-based system which uses parameters derived from detectors was described. The rule-based sets can be extended without software modification. Moreover, the methodology and the example of the rules development are presented. In Section 5, the road sign cooperation based on the example of fog detection and air transparency was described. This example includes parameter analysis, inferencing and presentation of information for car drivers. In Section 6, the innovative algorithm for the air transparency analysis on the basis of one camera was described in detail. Moreover, the methodology of research carried out using the images obtained (in the day and night) from the GDDKiA's roads and highways surveillance system was presented.

Operation of Intelligent Road Sign
The IRS rule-based system is presented in fig. 1. The operation of the system will be explained on the example of air transparency analysis. Information on atmospheric conditions is obtained from weather stations, which will form an integral part of the device being built. Due to costs, the standard set of sensors does not include an air transparency detector, so the concept of using the camera (which is part of the built device) and appropriate software was created. Images from the camera are analyzed according to a given time interval using the algorithm described in Section 6. On its basis, the rule-based system determines the safe speed, which results from the model for braking distance of vehicles. The road surface condition (dry, wet) and average speeds of moving vehicles are also considered because the goal is to reduce the speed differences that decrease the risk of overlapping vehicles in the fog.  The developed rules enable considering various road parameters, such as adhesion, road quality, occurrence of arches and traffic parameters, such as vehicle speed, traffic density and road obstacles. The implementation of appropriate rules for the inference engine, describing weather conditions in the spatial-temporal context, enables the proper integration of the air transparency analysis system with the Intelligent Road Sign.

Rules development
In the presented system, the rules are adapted to the place where the sign is installed. The installation place is defined based on the seasonal occurrence of factors influencing the safeness of the road traffic. For example, it can be a road place where there are frequent accidents in the case of occurrence of the specified weather condition, slowing traffic down by a truck in the driveway or congestion on road exits / entrances during peak traffic hours.
The periodic phenomenon closely related to the place of its frequent occurrence is fog. Air transparency determines the distance from which the driver sees an obstacle on the road. Considering his reaction time and braking distance, we can determine the safe speed of the vehicle. A step change of the speed limit is associated with threshold values of visibility at which it should be decreased or increased.
The defined rules are associated with the reaction scenarios aimed at ensuring the safety of traffic participants and improving the efficiency of traffic control.
The intelligent sign is equipped with numerous sensors providing knowledge about the environmental parameters and the behavior of vehicles on the road. Additionally, on the basis of information received from neighboring signs and the System Management Center, it can analyze the current situation. Using a complex rule system, the intelligent sign reacts to:  weather conditions, noise, air pollution,  sudden events causing closure of the road or lane,  changing the traffic characteristics, such as density, throughput, speed, speed differences,  individual behavior of vehicles (V2I). The way in which the system reacts comes down to:  speed management by setting speed limits,  road or lane closure,  informing System Management Center and emergency services with a sudden event,  displaying warning signs,  displaying warning and informative messages,  organizing detours and suggesting alternative routes. Correctness of sensors' operation and the rule system are constantly monitored by System Management Center.
When the conditions for many rules are met, the one with the highest priority is implemented. In terms of speed management, it will be the one that results in the lowest speed limit. The priority of the rules in the case of the implementation of complex scenarios by a group of signs goes as follows: a In the case of fog detection in the image from the cameras, the analysis algorithm detects one or two objects along the road that are not occluded by moving vehicles. The varying distances of these objects allow us to determine the safe speed based on their visibility. When observing two objects, the binary variables Visibility_1, Visibility _2 and safe speeds for visibility Speed_1, Speed_2, Speed_3 (e.g., 90, 70, 50 km/h) were defined.
An example set of rules for segment i and traffic direction 0, displaying two fields (road sign or text information) look as follows: If Speed_1 is the speed limit of the segment, it is not displayed at full visibility. B-33 is a speed limit sign to Speed_x [km/h] and requires appeal (B-34) by IRS [i+1,0] or a further sign. It is possible to notify the fog earlier by IRS [i-1,0] by using the sign A-30 and K-11 "Warning: fog". For direction 1, the indexing sequence of signs is shifted and reversed.

Fig. 2. Sample IRS Rule
Inference engine was implemented in C# language with NRule package. The package supports a fluent API for rules design. Figure 2 shows the sample rule in the NRule language. The original inference algorithm was extended to include the prioritization strategy.

Road sign cooperation
Cooperation manner of intelligent signs is closely related to their topology. In the simplest arrangement, a single sign is considered. It analyzes parameters related to the road, vehicles and surroundings measured in front of (speed) or behind (e.g. visibility) the sign. Based on the set of rules, the action scenario is implemented, concerning the fulfilled rules with the highest priority or, for rules with equal priority, with the lowest speed limit. The scenarios consider the location of the sign, distances characterizing the road infrastructure and periodically occurring threats. In the case of a permanent factor having an impact on road safety, permanent marking is used. Sequential topology refers to placing intelligent signs along the same road. It allows to inform the driver in advance about the threat, including the distance -plate T-1. If the phenomenon occurs on a fixed segment of the road (e.g., congestion), the length of the road section can be given -plate T-2 ( fig. 3). Because each sign in the sequence independently analyzes the traffic situation, information is exchanged between neighbors and the scenario related to the rule with the highest priority is implemented jointly.
Analysis on two (or more) intersecting roads is a more complex topological case. In this case, the most important scenarios are the management of vehicle streams -e.g., organization of detours, indication of alternative routes, information about events on other roads.
Information from each IRS is processed in the Central Management System to detect area-related events -e.g., fog, snow. Additionally, Central Management System is necessary for supervising autonomous IRS. Some events will not be detected by the sensors of the IRS. The source of these events can be Central Management System (e.g., planned repair of roads) or drivers (e.g., reporting an accident). The implementation of such scenarios is initiated by the central dispatcher.

Air transparency analysis
The main objective of the IRS is displaying warnings and suggestions of speed limits. Intelligence of the road sign is based on the information analysis obtained from its own detectors and from the detectors of the neighborhood signs. Volume of road traffic, accidents and weather parameters are critical information for this system.
Information on atmospheric conditions is obtained from weather stations, which will form an integral part of the device being built. The standard set of sensors does not, however, include an air transparency detector, the additional installation of which (in the hardware version) would cause a dramatic increase in device costs. Therefore, the concept of using the camera (which is part of the built device) and appropriate software was created.
In order to implement and test our image analysis algorithms a substantial number of real images is needed. For this purpose, the special algorithm for acquiring data from cameras of GDDKiA from the Małopolska area operated by TraxElektronik [22] was implemented. Thirteen cameras located in seven locations were used for data acquisition. The images with resolution 1280x720 pixels were collected over the next few days at ten minutes intervals (e.g., Rabka -4 days, Chyżne -4 days, Głogoczów -5 days in two series, Rdzawka -2 days).
Several particular situations were detected during image sequence analysis. For example, large lighting changes during the periods of sunrise and sunset lead to a temporary lack of image sharpness caused by camera parameter changes. Therefore, it was necessary to extend the existing software with a module for automatically determining the time of day [23].
The generation of the ground truth was the next stage of work. The obtained ground truth was compared with the results of the image analysis. Each sequence of ground truth contains three, manually generated, sets of data for each image: a. fog/ no fog (values 0/1), b. night/day (values 0/0.5/1/0.7), c. lens clean or splashed with rain (values 0/1). The ground truth generation process needs the analysis of all images in sequence. As mentioned above, if images were taken during sunrise or sunset, then they lose sharpness. This phenomenon is the reason for the false classification of the image. Therefore, the ground truth was initially extended by information: dusk/dawn (value 0.5). In the next experiments, dusk and dawn were distinguished by values 0.5 and 0.7, respectively.

Scene configuration
Configuring the scene (image) is primarily based on the appropriate positioning of the camera. If its main task is the detection of fog, then it should observe an object with clearly outlined edges, located at a medium distance, well-lit at night. This object cannot be occluded by moving vehicles or other objects. After selecting the object, the Region of Interest (ROI), i.e., a rectangle (window) that limits the area of analysis is to be defined. In addition, one can specify a line along which the cross-section through the image will be extracted. Cross-section analysis can provide additional information that will support the recognition algorithm in critical situations. Configuration examples are shown in fig. 4 and  fig. 7.
The configuration is supplemented with the data necessary to run the module which automatically calculates day and night periods. This module was implemented in Matlab (MathWorks, Inc., MATLAB release 2018a) on the basis of software found on the Internet [23]. Input data to the module are camera location (latitude and longitude), time zone and information about annual time changes (summer / wintertime). Based on this data and registration time (coded in the image name), the script qualifies subsequent images as downloaded at night or during the day.  Fig. 5. Image sections grouped into classes. In the second row two fog masks are presented Figure 5 shows all cross-sections of the 'Glogoczow1' sequence grouped into classes. Based on the charts corresponding to the fog (see the first row of fig. 5), two binary masks were generated, presented in the second row. The proposed algorithm imposes subsequent section functions on the appropriate masks and counts the points outside the mask. The results are presented in fig. 6.   Fig. 6. Fog detection results. From above: number of cross-section points outside fog masks, level 10 -graph baseline and detection threshold, detection result (magenta). Below ground truth: fog (red), day (black), rain (blue). Light green -daylight limits calculated with the astronomical script. On the left, the results for the sequence to create the algorithm. On the right -test sequence (the higher threshold and detection result are marked in orange)

Cross-sections analysis algorithm
In the case of the sequence used to create the algorithm (left part of fig. 6), a very low detection threshold was set -at the level of one point. In practice, this means that the crosssection function to be tested must be contained in the mask. Increasing this threshold would generate 'false detections', but they will be located in the sunrise and sunset areas. In the case of the test sequence (right part of fig. 6), the low threshold detection results are slightly worse (magenta line).
Increasing the threshold value improves the results (higher threshold and detection result shown in orange). Another solution would be to supplement the masks with data obtained from the test sequence and recheck the detection results. In the case of small-time intervals of subsequent detections, they can be extended, thus obtaining their merging.

Co-occurrence matrix analysis (GLCM)
The second idea is to use a two-dimensional histogram (2D -Gray Level Cooccurrence Matrix, GLCM) [9,10]. The 2D histogram is a matrix, in which the frequency of the pixel with the intensity value (gray, brightness) i occurs in a specific spatial relation to the pixel with the value j. In the Matlab software, this operation is performed by the graycomatrix function. By default, the spatial relationship is defined as the distance of the pixel under consideration to its right neighbor (horizontal), but other spatial relationships (directions and distances) between two pixels can be specified. Each element p (i, j) of the calculated matrix is the sum of the occurrences of specific spatial relations of the pixel i relative to the pixel j in the image.
GLCM arrays are usually analyzed statistically. In the Matlab software four coefficients based on GLCM (contrast, correlation, energy and homogeneity) were defined. It was found that one of them -contrast -best differentiates hazy and fog-free images: where i, jcoordinates of the GLCM element, p(i, j)value of the GLCM element.
For the presentation of this method, the sequence of images 'Rabka1' was selected. The registration of 624 images lasted over 4 days. Examples of images on which ROI windows have been defined are shown in fig. 7. The use of this method is possible if the camera image has the right number of sharp edges that are clearly visible both during the day and at night. To check this, after processing two images with the Sobel operator, the statistics of the occurrence of edges in eight directions were calculated (see fig. 8). The graphs show that in the images the vertical direction of gradient is dominant. Experiments, during which the GLCM parameters were changed showed that the most accurate detection results are obtained if the entire image is analyzed instead of individual windows. Table 1 summarizes the GLCM parameters used in the subsequent experiments. The results of the experiments in the form of binary detections are shown in fig. 9.
In most experiments, the GLCM matrix was calculated for the distance of pixels D=2. However, the directions in which the matrix was calculated were changed. The 'offset' column shows directions in the form of geographic ones, with the 'symmetry' parameter included.
For a 'symmetry' equal to 'true', the calculations were also carried out in the opposite direction. Examples are Experiments 2 and 3, in which the matrix was calculated for pixels lying at a distance of D = 2 vertically -up and down. For comparison with 4, experiment 5 (D = 1) was carried out, which gave a slightly worse result (see detection diagrams in red and blue in fig. 9). It is also interesting to compare the results of experiments 2 and 3: the number of gray levels of the image was radically reduced in experiment 3 (from 256 to 8), giving a slightly worse result. The last but one detection is too long. The software configuration process should be combined with testing fog detection efficiency. It consists of the following steps: a. Several-day recording of images obtained every few minutes, while at that time atmospheric conditions, especially fog, should occur, b. Defining several image windows with many edges -visible during the day and at night, c. Determining the geographical coordinates of the camera and saving them in the configuration file, d. Reviewing recorded images and generating 'ground truth' data -fog (0/0.5/1), day (0/0.5/1/0.7), lens clean/splashed (0/1), e. Calculation for the recorded sequence section functions and the GLCM contrast coefficient for the whole image and individual windows, f. Summary of diagrams from items 4 and 5 in one drawing -determination of fog detection thresholds. The thresholds calculated in this way should be saved in the configuration file. When acquiring subsequent images, the haze detection program will apply the thresholds to the GLCM coefficient or image section calculated in the given analysis windows, and then generate a detection result.
The process described above should be cyclically repeated, as the acquired images may change depending on the season and construction works carried out. Periodic maintenance works, such as cleaning the lens, are another problem. They can cause a change of camera position and thus acquisition of images different from those used during configuration. In practice, the fog detection is an iterative and multistage process, in which it is important to obtain many images recorded in different atmospheric conditions.

Conclusion
The software configuration process and testing fog detection efficiency for the IRS were presented. It requires recording of images obtained in variable atmospheric conditions (mainly fog), defining analysis windows (edges -visible during the day and at night). As an effect of our subsystem's operation, we get the result of fog detection (according to the given time interval). The lack of fog is interpreted as visibility expressed in meters, which is conditioned by the course of the road and its surroundings (constant parameter). For this value, the rule-based system determines the safe speed using a set of predefined rules.
In the case of fog detection, for the determined visibility the rule system specifies a lower speed than the currently displayed one, and the B-33 sign is displayed with the required speed limit. The preceding IRSs in the sequence display the fog warning and the speed limit. The following IRSs can cancel them if the threat disappears.
The experiments described in sections 6.2 and 6.3 showed that it is possible to detect fog using cameras and appropriate image analysis algorithms. The results of research on the effectiveness of the visual subsystem of fog detection indicate that the autonomous decision system of IRS in speed control in a situation of reduced visibility may be an alternative to other dedicated solutions. The proposed IRS has two valuable properties: continuous detection of changes in road and weather parameters and immediate reaction to the changes due to the embedding in the sign the rules and the inference system.
In the future, the presented system will be extended by modules which allow data exchange between road infrastructure and cars using V2X short-range ad-hoc network. The two-way communication delivers information, such as speed, acceleration, using the ABS system, which can be used by inference engine using extend ruleset. Moreover, the vehicle will directly obtain information about special incidents on the road, such as an emergency braking, a vehicle defect or a slippery road detected.