Infrared Spectroscopic Image Segmentation Based on Neural Immune Network With Growing Immune Field

ZHOU Zi-jie,ZHANG Bao-feng,YU Xiao

1.School of Computer Science and Engineering,Tianjin University of Technology,Tianjin 300384,China 2.School of Electrical and Electronic Engineering,Tianjin University of Technology,Tianjin 300384,China

Abstract Pipeline transportation has great advantages in long-distance transportation of oil and gas,and it is crucial to detect pipeline safety.In order to ensure the effective detection of the pipeline condition at any time,infrared imaging technology is of great significance in the field of pipeline detection because it can reflect the characteristics of the target according to the thermal radiation information of the object and ignore the influence of visible light.However,due to the diversity of the outdoor environment,infrared pipeline images have many problems,such as the uneven distribution of target features,pipeline occlusion and background target interference.These problems increase the difficulty of extracting pipeline targets,which is not conducive to the segmentation and detection of pipelines.The biological immune system exhibits excellent recognition,learning,memory,tolerance and coordination in antigen detection,extraction and elimination.These characteristics are lacking in current complex system optimization strategies.Based on the biological nervous system’s mechanism regulating immune system,a neural immune network is designed to detect and extract infrared pipeline targets in complex background.According to the regulatory mechanism of the biological neural network in the immune system,the neural network for infrared pipeline target location is constructed using the basic pipeline shape feature model.In this paper,three typical infrared pipeline images are selected,and the traditional target detection algorithm is compared with the algorithm based on neural immune network.The true positive rate of the traditional target detection algorithm is 0.405 6,and the neural immune network algorithm is 0.980 5.The Jaccard similarity coefficient of the traditional target detection algorithm is 0.271 8,and the neural immune network algorithm is 0.944 4.The absolute error rate is 0.117 5,and the neural immune network algorithm is 0.011 8.The results show that the true positive rate of the neural immune network algorithm is 0.574 9 higher than that of the traditional algorithm,and the absolute error rate is 0.105 7 lower.It proves that the proposed algorithm can extract the complete infrared pipeline target more accurately than the traditional method in the complex background.This network structure can improve the detection efficiency for pipeline safety.

Keywords Infrared image;Image segmentation;Immune field;Pipeline image

Introduction

Infrared imaging technology has the particularity of reflecting the target characteristics according to the object’s thermal radiation information[1].It can ignore the influence of visible light and detect the state of the object at any time.Therefore,it is used in the field of steam pipeline safety detection.However,due to pipelines are mostly in the outdoor complex environment,the staggered pipelines,the occlusion of a complex environment and the interference of similar targets will affect the detection of pipelines.

More and more attention has been paid to the research of infrared image target extraction algorithm in recent years.Zhang et al.presented an infrared image segmentation based on distance-gray compensation and improved maximum entropy to mitigate the impact of fuzzy boundaries[2].This method considers uniformity within each category on the premise that the difference between categories is ensured.The algorithm can extract small infrared targets from a complex background accurately.R.M.Prakash et al.used K-means,fuzzy C-means (FCM) and Gaussian mixture model-maximum expectation (GMM-EM) to segment infrared breast images[3].They achieved a classification of malignant and benign cancer tissues.Aiming at the weak symmetry problem of infrared pedestrian targets,an improved fuzzy C-means clustering method is proposed by Xiangzhi Bai et al.,which combines the geometric symmetry information introduced by Markov random field theory to extract infrared pedestrian targets[4].Li Lu et al.used the Gaussian mixture model and the background likelihood to extract the target,combined with the shape information and infrared characteristics of the infrared pedestrian target,and presented a foreground estimation method based on kernel density estimation,and verified that it had better adaptability to the blurry contour infrared target[5].He et al.presented an improved search method of optimal parameters to address the parameter estimation problem of pulse coupled neural network (PCNN) in the human infrared segmentation[6].

The infrared image has a good advantage in the performance of target features,but most of the algorithms are still aimed at the infrared target in a simple background,and the target features are higher than the background features.At present,there is no effective algorithm to extract the target accurately from the complex infrared image.

The biological immune system exhibits excellent characteristics of recognition,learning,memory,tolerance and coordination in antigen detection,extraction and elimination.These characteristics are lacking in current complex system optimization strategies[7].In recent years,scholars have combined it with image processing,and many artificial immune algorithms have been proposed.We have also conducted in-depth research in this direction.Dong-Mei Fu et al.proposed a method for hand segmentation based on the “the danger theory” of immunology in 2017[8].According to the template characteristics,the danger signal is defined,and the danger theory is used to segment the image.In the same year,we proposed an infrared target segmentation method based on the growing immune field and clone threshold[9].The algorithm combines the immune field with the region grew to segment the infrared target accurately.In 2018,we constructed an immune segmentation algorithm with a minimum mean distance immune field for target weakened infrared images[10].This method usesa multi-step classification algorithm,immune variation and adaptive immune minimum mean distance recognition to extract the infrared blurred handprint.

Although some achievements have been made in the research of artificial immune algorithm,the artificial immune algorithm only involves part of biological immunity.The characteristics of interaction and inseparability between the nervous system and the immune system in more complex biological immune processes have not yet been involved.In this paper,a target extraction algorithm for infrared pipeline image is proposed based on the mechanism of the biological nervous system regulating the immune system.A neural network is used to provide target feature information for the immune network to express the synergistic effect of the nervous system and immune system in biological immunity.

1 Problem analysis

Figure 1 shows an interlaced infrared pipeline.It can be seen from the figure that the distribution of features inside the pipeline is not uniform,and the gray features of the interlaced parts between the pipelines are very close.This situation increases the difficulty of using the threshold extraction method to extract different pipeline targets effectively.In Figure 1(b),it can be seen that the pipelines cannot be effectively divided.

In addition,there are curved or deformed pipelines.Figure 2(a) shows a pipeline with a curved shape,and the pipeline in Figure 2(b) is deformed due to the limitation of image acquisition location,and the width of both sides of the pipeline is not uniform.These bring difficulties to the location and detection of pipeline targets.

Some infrared pipeline images with the complex background are occluded by trees or buildings.Pipelines are occluded by buildings in Figure 3(a).The gray-scale features of these occluders are generally lower than the average gray-scale features of pipes.Although it is easy to obtain the target region of the pipeline,the extracted region will appear discrete distribution and disconnection due to occlusion,which will affect the integrity of the extracted target.We adopt a simple threshold extraction method and select the appropriate threshold to segment the infrared pipeline image.The result is shown in Figure 3(b).The gray region represents the extracted pipeline,and because of the influence of occlusion,the discrete distribution of the extracted target in this region cannot connect with each other,which affects the integrity of the extracted target.

Fig.1 The infrared image of the interlaced pipeline(a)：Feature analysis；(b)：The result of threshold segmentation

Fig.2 Infrared pipeline images with a special shape(a)：The curved infrared pipeline；(b)：The deformed infrared pipeline

2 Proposed method

The concept of “psychoneuroimmunology” was proposed by Ader and Conhen in 1975.It emphasizes the interaction among nervous,endocrine and immune systems,and explains the relationship between physical and mental stress and human diseases.Studies have shown that emotion and response mechanisms are achieved through a high degree of interdependence of the psycho-immune pathway,and that the immune and endocrine systems are regulated by the central nervous system (CNS)[11].It is well known that immune cells can express a variety of neurotransmitter receptors and hormone receptors.Neurotransmitters and hormones produced by the neuro-endocrine system can act on immune cells and exert a positive and negative regumation on the immune system.It is well known that immune cells can express a variety of neurotransmitter receptors and hormone receptors.Neurotransmitters and hormones produced by the neuro-endocrine system can act on immune cells and exert a positive and negative regulation on the immune system.According to the concept of psychoneuro immunology,this paper designs a kind of neural immune network system.Its network topology is shown in Figure 4.

Fig.4 Neural immune system network topology

For infrared pipeline images,this paper proposes an optimal immune field neural immune network infrared target extraction algorithm based on the mechanism of the nervous system regulating the immune system in biological immunity.The algorithm combines the optimal immune field with region growth,and it includes neural network function,the synergy between neural network and immune network,adaptive immune network function.The proposed algorithm flow is shown in Figure 5.

Fig.5 Flow diagram of the proposed algorithm

2.1 Neural network function

The biological nervous system plays a leading role in the regulation of physiological and functional activities in the body.It is interacted and inseparable with the immune system.As is known to all,the sympathetic nervous system plays a key role in regulating inflammatory conditions,and imbalanced sympathetic activity is usually observed in rheumatoid arthritis[12].The relationship between the nervous system and biological immunity has also been studied by immunologists and neuroscientists.In recent years,researchers have investigated the biological mechanism of communication between the immune system and the nervous system,and the results show that the nervous system can control the immune response[13].In this section,a kind of neural network structure for infrared pipeline image is proposed.The aim is to construct a neural network to locate the pipeline in the infrared image and to form a template for predicting the absolute target region and the undetermined region in the image.

It is difficult to locate the position of the pipeline from complex infrared the pipeline images.In this paper,template matching is used to locate the position of pipeline from infrared images.The flow chart of the algorithm is shown in Figure 6.Firstly,according to the type of pipeline,the algorithm establishes template sets of different sizes,shapes and sizes.These templates are matched with the infrared pipeline image one by one,from which the most appropriate position of the response value is selected to locate the target pipeline.There is an optimal matching point in the response results of each template,and the optimal template is selected according to the judgment criterion of algorithm 1.

Fig.6 Flow diagram of pipeline template selection matching

Algorithm 1The algorithm for pipeline template selection and matching

Input:infrared pipeline image;pipeline template setM(I)

Output:the result of the optimal template matching

Step 1:Select the template,and traversepipeline image to get optimal response results of MatchingN(i) and Corresponding position (xi,yi)

Step 2:Select the optimal template.

Ifi==1

N=N(i) and (x,y)=(xi,yi)

Else

IfN(i)

N=N(i) and (x,y)=(xi,yi)

update optimum response value N and template position (x,y)

Step 3:Matching Completion Conditions

Ifi==1←The matching is completed and the optimal matching template is obtained.

Else

i=i+1←Calculate the next template response.

Step 4:Get an optimal template matching.

2.2 Synergy between neural network and immune network

The nervous system can regulate the immune system by exciting or inhibiting transmitters.And the nervous system and the immune system cooperate in biological immunity[14].In this section,an optimal neural immune growth immune field is proposed to segment the undetermined region,and a synergistic method of neural network and the immune network is designed.The algorithm uses a neural network to determine the target region and the undetermined region of the pipeline.The aim of the algorithm is to achieve regional immune growth by optimal neural immune field.

2.2.1 Selection and matching of pipeline template

The biological immune system protects that normal cells are not affected by pathogens,such as bacteria and viruses.The key is to distinguish between auto-antigens and various external antigens.Combining the theory of fault-tolerant field and the mechanism of biological immune action range,the immune field is defined according to the mechanism of immune factors acting on antigen feature space.The immune field is a characteristic antigen space that can be recognized by biological immune system.For the immune field recognized by an immune factor,if the antigen recognition plane corresponding to the immune field can vertically bisect the line between the antigen in the immune field and an other antigen in the non-immune field.The immune field is called the optimal neural immune field.

In the previous section,the optimal matching template has determined the target region.The key to segment the undetermined region is the correlation between the edge features of the target and the surrounding undetermined region.By calculating the sum of the target edge region features and comparing it with the sum of the features of the undetermined region,we can obtain the optimal neural immune field.

Algorithm 2Computation of optimal neural immune field.

Input:optimal matching template;infrared pipeline image

Output:optimal neural immune field

Step 1:Obtain the undetected region on one side of the optimal matching templateP=(p1,p2,…,pn) and corresponding infrared image regionQ=(q1,q2,…,qn)

Step 2:According to formula (1),calculate the feature value of the undetermined regionT

(1)

Step 3:Obtain the region of the template target edge and a corresponding infrared image.According to formula (1),calculate the feature value of the targetT2

Step 4:According to formula (2),calculate the optimal neural immune fieldI,αrepresent the proportionality coefficient.

I=(T2-T)×α

(2)

Fig.7 Regional immune growth(a)：The optimal matching template；(b)：The original infrared pipeline image；(c)：The black region represents the initial seed region of the target edge,and obtains the features of the region in the original image；(d)：The black region represents the n-layer growth region in the detected region obtained after expansion.The algorithm compares the corresponding features of the black regions in (c) and (d),and judges whether the growth conditions are satisfied by the optimal neural immune field.

2.2.2 Regional immune growth

Region growth is a common algorithm in target extraction.The seed points are set to judge the correlation between the pixels according to the growth criterion,so as to realize the target extraction.The proposed algorithm uses the optimal neural immune field to design the growth criterion.

Because of the high symmetry of the pipeline,the region features on both sides of the target are also correlated to some extent.Therefore,unlike the classical region growth,the edge regions on both sides of the pipeline are taken as the initial seed regions.By expanding the target regions,the edge regions of each layer of pipeline in the area to be detected are obtained and compared with the features of the initial seed regions.If the optimal neural immune field is satisfied,the immune growth of the region is carried out.The schematic steps are shown in Figure 7.The schematic diagram only shows the growth process of the unilateral region.

2.3 Adaptive immune network function

The biological immune system includes the interaction of innate immune factors and adaptive immune factors[15].Although congenital immune factors can quickly identify and kill some pathogens,other pathogens cannot be identified because congenital immune factors rely on specific receptors to identify pathogens.As for these pathogens,congenital immune factors can present and process pathogens to adaptive immune factors,determine the acting director of adaptive immune factors,and achieve adaptive immune effects.In this section,according to the interaction mechanism between innate immune factors and adaptive immune factors,an adaptive immune network is designed.

Because the extraction result of neuro-immunomodulation only related to local targets and their neighborhoods detected by neural networks,it is impossible to segment the global image information reasonably.This section combines the extracted results with the infrared pipeline image and chooses the optimal segmentation threshold through automatic threshold iteration.The detailed steps are shown in algorithm 3.

Algorithm 3Automatic Threshold Iterative Segmentation Algorithms

Input:Infrared pipeline image;immune growth pipeline image

Output:The result of pipeline segmentation

Step 1:Merge image.Multiplication of pixel of immune growth pipeline images and those of infrared pipeline images.

Step 2:Traversing merged image to get the maximumGmaxvalue and the minimumGminvalue of grays.

Step 3:According to formula (3),calculate the initial thresholdT

T=(Gmax+Gmin)/2

(3)

Step 4:The merged images are divided according to the thresholdT,and the average gray valuesgtandgbof the corresponding target and background regions are calculated.

Step 5:According to formula (4),calculate the initial thresholdT

T=(gt+gb)/2

(4)

Step 6:Get the optimal threshold.

IfTn-Tn-1≤alpha

T=Tn-1←Optimal threshold

3 Experimental results and analysis

3.1 Qualitative assessment

In order to verify the effectiveness and accuracy of the proposed extraction method,we have applied it to the dataset of infrared pipeline image.And we compare the proposed algorithm with five image segmentation frameworks,Sobel,Canny,Otsu,maximum entropy algorithmand SVM.According to the characteristics of the infrared steam pipeline,three typical types of infrared pipeline images are selected from the dataset (see Figure 8).The pipeline of the first image is curved and partially occluded;the second pipeline has a simple shape and occluded,and the occluded region is large;the third pipeline is in a complex background.While the shape of the pipeline is bent,the distribution of internal features of the pipeline is not uniform.These three kinds of images contain most of the problems of infrared pipeline image,which can fully reflect the accuracy of the algorithm for pipeline image extraction.

First,we provide the segmentation results qualitatively (see Figure 9).For the pipeline contour extraction of infrared pipeline image,the segmentation by Sobel operator and Canny operator can extract part of the region,but the pipeline region of edge detection is not connected.The segmentation results of Otsu algorithm in the fifth row show that the algorithm can extract the target more completely,but it has the problem of over-extraction.The algorithms represented in the sixth row and the seventh row can ensure the extraction of the pipeline region.Compared with the Otsu algorithm,there are fewer over-extraction regions.However,due to the influence of target occlusion,some occluded regions are lost in the extraction results.In contrast,our method can ensure the integrity of the pipeline segmentation.Promisingly,the extracted region can ignore the influence of a complex environment on pipeline occlusion.

3.2 Quantitative analysis

There is no clear way to verify the accuracy and reliability of the infrared pipeline image extraction algorithm.In this paper,true positive rate,Jaccard similarity coefficient,and absolute error rate are used as the quantitative metric for the evaluation of infrared pipeline extraction results.Quantitative results are used to verify the accuracy of the proposed algorithm for the segmentation of infrared pipeline images.In the following analysis,we assume that the actual segmented image is P and the segmented reference image isT.

Fig.8 Infrared pipeline images in the experiment(a):Original image;(b):Ground truth

Fig.9 Visual comparison of segmentation results using different frameworks(a):Sobel;(b):Canny;(c):Otsu;(d):Maximum entropy method;(e):SVM;(f):Our method

3.2.1 True positive rate

True positive rate (TPR) is an evaluation index of image target extraction.It is expressed as the ratio of correctly labeled pixels to reference target pixels.And correctly labeled pixels refer to the overlap pixels between the actual segmented target and the reference target.The formula of the true positive rate is as flows

(5)

According to the formula,the higher the value of TPR,the more overlapped the actual segmented target and the reference target.The results are shown in Table 1.

Table 1 Comparison of the true positive rate for six methods

As shown in Table 1,the Sobel operator and Canny operator extract fewer pixels in the target region,and the true positive rate is low.The true positive rate of the maximum entropy extraction algorithm and SVM algorithm shows that the algorithms are unstable and cannot effectively extract the target region.The true positive rate of the proposed algorithm and Otsu algorithm is more than 96%,which shows that the target extracted by the two algorithms is the closest to the reference target.

3.2.2 Jaccard similarity coefficient

Jaccard similarity coefficient (JSC) is a probability value used to compare the similarity and difference between sample sets and reflect the degree of coincidence between two segmented images.The higher the cross ratio,the better the image segmentation results.The calculation of the Jaccard similarity coefficient is represented by formula as flows

(6)

From Table 2,it is evident that the Jaccard similarity coefficient of the proposed algorithm is significantly higher than those of other methods.The similarity coefficient of the proposed algorithm is above 91%,which shows that the similarity between the target extraction result and the reference target is high,and the stability of the extraction algorithm is good.

Table 2 Comparison of the Jaccard similarity coefficient for six methods

3.2.3 Absolute error rate

Absolute error rate (AER) is expressed as the ratio of absolute error pixels to total pixelsN.And absolute error pixels refer to the different pixel between the actual segmented target and the reference target.The lower the absolute error rate,the better the segmentation effect.It is defined as

(7)

The results are shown in Table 3.

The absolute error rate of the proposed algorithm is much lower than that of the Otsu algorithm.Although the true positive rate of the Otsu algorithm is similar to that of the proposed algorithm,its absolute error rate indicates that the algorithm cannot extract the target accurately because of over-segmentation.Compared with other algorithms,the proposed algorithm has a lower absolute error rate while ensuring a higher true positive rate,which shows that the target extracted in this paper is closer to the reference target.

In summary,this article compares the evaluation results of the six methods of extraction results,as shown in Table 4.

Table 3 Comparison of the absolute error rate for six methods

Table 4 The extraction results of the six methods

The experimental results demonstrate that for the segmentation of infrared pipeline images,our method is more efficient than the other five methods.In addition,the error of target extraction results is lower than other algorithms,and the accuracy of target extraction is good.

4 Conclusion

In infrared pipeline images,pipelines are affected by complex environments,such as occlusion,pipeline intersection and so on.This paper applies the coordination mechanism between the biological nervous system and immune system to pipeline extraction algorithm,and proposes a target extraction algorithm based on optimal neural immune growth immune field.Experiments show that our method is effective and accurate for infrared pipeline image extraction.