Research on classification diagnosis model of psoriasis based on deep residual network

LI Peng, YI N, DING Chngsong*, LI Sheng, MIN Hui

a. School of Informatics, Hunan University of Chinese Medicine, Changsha, Hunan 410208, China

b. The Third Xiangya Hospital of Central South University, Changsha, Hunan 410013, China

c. Key Laboratory of Medical Information Research, Central South University, College of Hunan Province, Changsha, Hunan 410013, China

d. Software Institute, Hunan College of Information, Changsha, Hunan 410200, China

ABSTRACT

Keywords Psoriasis Deep residual network Data enhancement Cross-entropy Adam algorithm Recall

1 Introduction

Psoriasis is a chronic recurrent disease characterized by excessive proliferation of epidermal cells and immune inflammation[1]. The disease is characterized by a long course, stubborn, common and ugly appearance, and invasion of a variety of organs in the later stages, and is listed as one of the world’s top ten persistent diseases by the World Health Organization[2].

Although several current studies have analyzed and summarized the medication rules of famous traditional Chinese medicine doctors for the treatment of psoriasis[3], the diagnosis of psoriasis is rarely discussed. In fact, according to statistics, there are approximately seven million psoriasis patients in China[4].However, only a small number have been diagnosed,mainly owing to a lack of diagnostic ability of grassroots doctors or hospitals. At the “Popular Science Activities of World Psoriasis Day” held in 2018, experts considered that artificial intelligence (AI) technology combined with big data can provide patients with more intuitive auxiliary diagnosis of psoriasis[5-8].

Psoriasis cannot be completely cured. Basically every patient will relapse and need long-term followup treatment. The diagnosis and treatment of psoriasis, including its four major types, psoriasis vulgaris,joint psoriasis, purulent psoriasis, and erythroderma psoriasis, is very difficult. While diagnosing the disease, we should consider not only the appearance factors, but also the cardiovascular, psychological,gastrointestinal, autoimmune, and other aspects comprehensively. Accurately and quickly diagnosing suspected psoriasis patients and the psoriasis variant they are afflicted with poses a major problem[9]. The diagnosis involves a typical image classification problem, and convolutional neural networks (CNNs) used in deep learning are the primary method to deal with such medical image problems. Deep residual networks (ResNet) are known as one of the most representative CNN models[10]. In the 2015 ImageNet computer vision recognition challenge, ResNet emerged as the champion in all three major challenges: image classification, image location, and image detection.The system error rate of the visual computing group is as low as 3.57%, which can greatly improve computer vision problems. Currently, the group is being widely used in large-scale image data in various applications. This paper proposes a classification diagnosis model of psoriasis based on deep residual network. A ResNet-34 model was trained to classify and diagnose psoriasis, which effectively improved the recognition rate.

2 Classification and diagnosis of psoriasis in ResNet

2.1 ResNet principle

Theoretically, it is generally believed that with greater CNN depth (more parameters), its nonlinear expression ability grows stronger, more complex feature pattern extraction can be performed, and better results can be obtained. However, a large number of studies[10,11]have shown that with an increasing number of layers and a deeper network,the result can worsen because the deeper the network, the lower the accuracy of classification, that is, the performance starts degrading. To solve this problem, HE et al.[10]proposed the famous deep residual network in 2016. ResNet is composed of stacked residual units (as shown in Figure 1). It is easy to optimize and can improve the accuracy by increasing the depth. The internal residual block uses jump connections (short circuit mechanism) to alleviate the gradient disappearance or gradient diffusion caused by increasing depth in deep neural networks.

In the deep residual network shown in Figure 1,relu[12]represents the activation function of the network. The residual unitF(x) can be expressed as:

F(x)=H(x)-x(1)

Here,xrepresents the input value;H(x) represents the feature learned when the input isx. In the two-layer network shown in Figure 1, the optimal output is inputx, therefore, for the network without identity mapping, it needs to be optimized toH(x)=x; however, for the network with identity mapping, that is, a residual block, if the optimal output isx, then only the residual unitF(x) needs to be optimized to 0. The optimization of the latter is simpler and more effective than the former. The principle is as follows.

Equation (3) can be rewritten as follows, that is,we can obtain the learning characteristics from shallowpto deepQ:

According to the principle of back propagation[14],it is assumed that the error is ε and the partial derivative with respect toxpcan be obtained as follows:

2.2 Classification and diagnosis process of psoriasis based on ResNet-34

A flow chart of psoriasis classification diagnosis based on ResNet-34 is shown in Figure 2. Doctors or the suspected patients themselves take photos of the diseased parts and upload them to an application system or APP. The system or APP calls ResNet-34,which has been trained in advance and deployed to classify and diagnose the uploaded images, and output the conclusion (presence or absence of psoriasis) and the type of psoriasis in case of confirmed diagnosis.

3 Technical details

3.1 Psoriasis image preprocessing

Due to the complexity of psoriasis, the location of the disease is also diverse. The pictures captured by the doctors or patients of the diseased parts are easily affected by factors such as illumination, camera equipment, and device pixels. As a result, there is massive noise and inconsistency of image formats in the obtained image data of psoriasis patients, which is not convenient for further processing. In this study,we preprocessed the sample images from three aspects: data enhancement, image size adjustment,and image format coding, to meet the input requirements of ResNet-34 and prepare for its training.

Figure 2 Classification and diagnosis process of psoriasis using ResNet-34

3.1.1 Data enhancement of psoriasis imagesThe acquisition of psoriasis data also involves significant cost. Therefore, if the limited existing data can somehow be enhanced, also called data amplification, it can provide a value equivalent to more data but without a substantial increase in size[13]. It is also an effective way to enlarge the data size. For the ResNet-34 training process, we hope that with a larger scale and higher quality of data, the generalization ability of the trained ResNet-34 can be improved. However, it is mostly difficult to cover all the possible scenarios while collecting data. For example, for illumination conditions, when collecting psoriasis image data, it is difficult for us to control the proportion of light. Therefore, when training the model, we need to add the data pertaining to illumination change and generate various training data dynamically to achieve better outcomes, reduce expenditure, and improve the model robustness.Therefore, it is necessary to enhance the data effectively in the particular case of psoriasis diagnosis. For image data, the commonly used data enhancement methods include rotation, translation,scaling, and edge filling. The core code used for data enhancement of psoriasis images is given below.

Core code for data enhancement:#First import the keras library 1. from keras.preprocessing.image import ImageDataGenerator,2. img_to_array, load_img 3. pic_path = r‘./yinxiebing.jpg’ #create image path 4. augmentation_path = r‘./data_augmentation' #save path#Define the ImageDataGenerator and explain what actions are used to generate new images:5. data_gen = ImageDataGenerator(rotation_range = 30, #rotate width_shift_range = 0.1, #translation height_shift_range = 0.1, //zoom zoom_range = 0.2, #edge fill fill_mode = ‘nearest’)6. img = load_img(pic_path) #the address to load the picture 7. x = img_to_array(img) #convert to array format to ImageDataGenerator 8. x = x.reshape((1,) + x.shape)9. n = 1 10. for batch in data_gen.flow(x, batch_size = 1,save_to_dir = augmentation_path, save_prefix =‘train’,save_format = ‘jpeg’):11. n + = 1 12. if n ＞ 10: #According to the operation defined by ImageDataGenerator, randomly select several types to generate 10 images.13 break

3.1.2 Size adjustment of psoriasis imagesThis process involves uniformly adjusting the size of the pictures, which facilitates the use of ResNet-34 model for deep learning. In this study, considering the new psoriasis pictures generated after data augmentation as an example, the core code to adjust the size of psoriasis pictures is given below.

Image resizing core code:1. from PIL import Image #use PIL library to change image size 2. import os #using os library to read file path 3. img_path = r'./data_augmentation' #read psoriasis pictures

4. resize_path = r'./resize_image' #put the image after resizing into the resize_image folder 5. for i in os.listdir(img_path):6. im = Image.open(os.path.join(img_path,i))7. out = im.resize((224, 224)) #the size after resizing is 224 × 224 8. if not os.path.exists(resize_path):9. os.makedirs(resize_path)10. out.save(os.path.join(resize_path, i))

3.1.3 TFRecord codeAs the ResNet-34 model can only accept numerical data as input, we also need to encode the images, that is, convert them to the TFRecord format. TFRecord is the standard format officially recommended by TensorFlow and helps store image data and tags into binary files, making it convenient to quickly copy, move, read, and store them in TensorFlow[14]. When training ResNet-34, by setting up a queue system, the psoriasis data in TFRecord format can be loaded into the queue in advance. The queue will automatically realize the random or orderly data in and out of the stack, and the independence between the queuing system and model training can accelerate the ResNet-34 reading and training. The following is the core code for converting psoriasis images into the TFRecord format.

Core code for converting image data into TFRecord format:1. import os 2. from PIL import Image 3. import tensorflow as tf 4. cwd = r“./brand_picture/” #image path, two groups of tags are in this directory 5. file_path = r“./” #TFRecord file save path 6. bestnum = 1 000 #number of pictures stored in each TFRecord 7. num = 0 #which picture 8. recordfilenum = 0 #number of TFRecord files 9. classes = [] #put labels into classes 10. for i in os.listdir(cwd):11. classes.append(i)12. ftrecordfilename = (“traindata_63.TFRecords-%.3d”% reco-rdfilenum) #TFRecords format file name 13. writer =tf.python_io.TFRecordWriter(os.path.join(file_path,ftrecordfilename))

14. for index, name in enumerate(classes):15. class_path = os.path.join(cwd, name)16. for img_name in os.listdir(class_path):17. num = num + 1 18. if num ＞ bestnum: #over 1 000, write the next TFRecord 19. num = 1 20. recordfilenum + = 1 21. ftrecordfilename =(“traindata_63.TFRecords-%.3d”% recordfilenum)22. writer =tf.python_io.TFRecordWriter(os.path.join(file_path,ftrecordfilename))23. img_path = os.path.join(class_path,img_name)#address of each picture 24. img = Image.open(img_path, ‘r’)25. img_raw = img.tobytes() #convert pictures to binary format 26. example = tf.train.Example(27. features = tf.train.Features(feature = {‘label’:tf.train.Feature(int64_list =tf.train.Int64List(value = [index])),‘img_raw’: tf.train.Feature(bytes_list =tf.train.BytesList(value = [img_raw])),}))28. writer.write(example.SerializeToString())#serialize to string 29. write.close()

3.2 Construction of ResNet-34

Figure 3 ResNet-34 structure for psoriasis classification diagnosis

3.3 Model training

ResNet-34 solves the problems of information loss in traditional convolution by changing the learning objective, that is, from learning the complete output to only the residual. It protects the integrity of information by passing the input directly to the output. In the ResNet-34 model proposed in this paper, we use cross-entropy as the loss function to evaluate the accuracy of the model, use the adaptive moment estimation (Adam) algorithm as the optimization strategy in the training process, and use the Softmax function to realize the multiclassification diagnosis of psoriasis[15]. We will elaborate in detail below.

3.3.1 Loss functionThe loss function is used to estimate the inconsistency between the predicted and real value of ResNet-34. It is a non-negative real value function. The smaller the loss function, the better the robustness of ResNet-34. In this study, we used cross-entropy as the loss function. Crossentropy can measure the difference between two different probability distributions in the same random variable, which is expressed as the difference between the real probability distribution and the predicted probability distribution of psoriasis. The smaller the cross-entropy, better the prediction effect of ResNet-34. The calculation formula is as follows:

3.3.2 OptimizerIn this study, we used the Adam[16]algorithm to train ResNet-34. The Adam algorithm is an optimization algorithm that combines the Momentum[17]and RMSProp[18]algorithms in deep learning models. In the initial stage of training, we first initialized the cumulant and square cumulant of gradient:

Then, in the t-round training, we calculated the parameter update of momentum algorithm and RMSProp algorithm:

Through Equation (9), we can get the correction value of the parameter gradient cumulant in the first iteration. Next, the weight and bias of the model can be updated according to the combination of Momentum and RMSProp algorithms:

In the Adam algorithm, parameter β1corresponds to β value in the Momentum algorithm, which is generally taken as 0.9; parameter β2corresponds to βvalue in the RMSProp algorithm, which is generally taken as 0.999, while ε is a smooth term, which is generally taken as 10-8, while the learning rateα needs to be slightly adjusted during training. To sum up, the pseudo code of the Adam algorithm can be expressed as follows:

1. Initializevdw=0,vdb=0,sdw=0,sdb=0;

2. In the t-th iteration, calculatedwanddbwith the mini batch gradient descent method;

因此，2017年陽(yáng)光印網(wǎng)推出了“300+1合伙人計(jì)劃”，面向全國(guó)招募供應(yīng)商合伙人。所謂“300+1合伙人計(jì)劃”是指，原本只是與陽(yáng)光印網(wǎng)開(kāi)展接單生產(chǎn)的供應(yīng)廠商，通過(guò)資本合作等方式，成為陽(yáng)光印網(wǎng)控股廠商中的一員，進(jìn)而實(shí)現(xiàn)全面的共生共贏。這些供應(yīng)廠商將在其細(xì)分領(lǐng)域和所覆蓋的地域里，扮演陽(yáng)光印網(wǎng)優(yōu)勢(shì)穩(wěn)定供應(yīng)點(diǎn)的角色。

3. Calculate the weighted average of the Momentum index;

4. Update with RMSProp;

5. Calculate the deviation correction of Momentum and RMSProp;

6. Update the weights.

3.4 Classification diagnosis

After the ResNet-34 training, we used the Softmax function, mainly used in multi-classification processes, to realize the classification diagnosis of psoriasis. It maps the output of multiple neurons to the (0, 1) interval as probability to understand, to realize multi-classification. The output of ResNet-34 model has five values, representing the four common psoriasis types (vulgaris, arthritic, purulent, and erythroderma) and normal condition (no disease).Therefore, for each sample, according to the definition of the Softmax function, the probability that it belongs to the categoryiis as follows:

4 Experiment

4.1 Data sources

From January 2017 to December 2019, we collected data of psoriasis patients from the affiliated hospitals of Hunan University of Chinese Medicine, and used them as the dataset for constructing the classification diagnosis model. The dataset contained the data of patients with four common types of psoriasis. All data types were images, sized 224 × 224. A total of 30 000 data samples were screened.

4.2 Experimental setup

We performed the experiment on an 8-core 16 thread computer (Intel Core i9-9960x @ 3.10GHz CPU, 16G memory), with Ubuntu 16.04 LTS 64-bit operating system. In the ResNet-34 model, relu function was selected as the activation function, and the psoriasis classification and diagnosis model based on deep residual network was implemented using Tensor-Flow and Anaconda platforms. In addition, to avoid model over fitting, we used k-fold cross validation(k = 10 in the paper) on 30 000 psoriasis data samples to evaluate the predictive performance of ResNet-34,and selected the best performance of the super parameters to obtain the final model.

4.3 Evaluation index and comparison object

In this study, the multi-classification problem was transformed into a binary classification problem for experimental evaluation. The transformation method used one vs. the rest method: one class is marked as a positive example and the remaining classes are marked as counter-examples. As the output of ResNet-34 in this study involved five possible types of results (four for psoriasis, one for normal), only five classifiers were constructed to realize the problem conversion. Then, the precision, recall,F1-score, and ROC curve were used to evaluate the performance of psoriasis classification diagnosis based on ResNet-34.Assuming that psoriasis patients represent positive cases and normal people represent counterexamples, the following confusion matrix can be used to measure the performance of ResNet-34.

Here,TPindicates that the positive example is predicted to be a positive example, that is, the real case;FNmeans that the positive example is predicted to be a negative example, that is, a false counter example;FPis a prediction of a counter example as a positive example, that is, a false positive example; andTNis a prediction of a counter example as a counter example, that is, a true counter example. According to Table 1, precision, recall, andF1-score can be defined as:

For the task of psoriasis diagnosis, it is necessary to focus on the recall instead of precision because most cases involve a positive case (no disease) and small number of counter cases (disease). The sample proportion of the two groups is very different. For example, in 100 records, 10 cases of psoriasis were found, out of which six were false positives and four were accurately identified. Although the precision was reduced to 94%, the recall increased from 0 to 100%. Although the disease was misreported occasionally, there was no omission of people with psoriasis.

Table 1 Performance indicators of psoriasis classification based on confusion matrix

The receiver operating characteristic (ROC) curve is used to describe the tradeoff between true positive and false positive rates. The true positive and false positive rates are defined as follows:

In addition, we compared the performance of ResNet-34 and VGG19 in psoriasis diagnosis to evaluate the superiority of this model. The implementation of VGG19 is described in reference.

4.4 Result analysis

Figure 4 shows the comparative recall rates of ResNet-34 and VGG19 for classification diagnosis on the psoriasis dataset. As evident from Figure 4, the recall rates of both the methods increase by varying degrees with an increase in the psoriasis dataset size.However, on the whole, the recall rate of this method is always higher than that of VGG19. On average, the recall rate of this method is approximately 9.5%higher than that of VGG19. The reasons are as follows: (1) compared with VGG19, ResNet-34 has greater depth and can extract better and richer features of psoriasis; (2) ResNet-34 can effectively solve the performance degradation problem caused by the increasing network depth by introducing the concept of residual blocks and adding identity mapping connection into the network structure.

Figure 4 Comparison of recall (ResNet-34 vs. VGG19)

Figure 5 shows the comparativeF1-scores of the two methods for the classification diagnosis of psoriasis. It is visible that theF1-scores of the two methods show a rising trend with an increase in the scale of psoriasis dataset. However, the performance of the proposed method is always better than that of VGG19. The reasons are as follows: (1) we used a variety of techniques, such as data enhancement and TFRecord encoding to clean the original psoriasis images, which minimized the impact of noise data on the diagnosis model; (2) we used the Adam algorithm to train the model, which reduced the training time and further ensured the accuracy of diagnosis.

Finally, to comprehensively evaluate the specificity and sensitivity of ResNet-34 and VGG19 in the classification and diagnosis of psoriasis, their ROC curves were drawn and compared, as shown in Figure 6. For each test sample, ResNet-34 and VGG19 received a “score” value for each classification, which indicated the likelihood of the sample to belong to a positive (or negative) case. To draw the ROC curve,we required a series of values pertaining to true positive and false positive rates. In this study, we achieved this objective by performing the following steps:

Figure 5 Comparison of F1-scores (ResNet-34 vs.VGG19)

(1) Sort the “score” value from high to low and use it as the threshold;

(2) For each threshold, the test samples, whose“score” value is greater than or equal to this threshold, are considered as positive cases, while others are negative examples. This step helped form a set of forecast data;

(3) The ROC curve can be obtained by connecting the observed data values.

Figure 6 ROC-AUC comparison (ResNet-34 vs.VGG19)

In Figure 6, the area under the ROC curve is called AUC. The classifier with a larger AUC value (area) has better performance. In Figure 5, the AUC values below the red and black lines represent the classification performances of VGG19 and ResNet-34, respectively. The AUC value of the latter is clearly higher than that of the former, which shows that ResNet-34 performs better than VGG19, and can be applied to psoriasis classification diagnosis task in real environment.

5 Conclusion

Psoriasis is a type of skin disease and is very difficult to cure. Owing to the various causes of the disease,accurately classifying and diagnosing psoriasis is difficult. In this paper, a psoriasis classification diagnosis model based on deep residual network is proposed. A 34-layer residual network was designed to achieve an accurate diagnosis of psoriasis. The final experimental results also verify the effectiveness of the proposed model. In the next step, we will continue to analyze the symptoms, syndrome types,and medication rules of psoriasis, build the knowledge map of the integrated diagnosis,treatment, and medication of psoriasis, and further propose a psoriasis medication recommendation model based on graph convolution neural network,to provide better a decision support system for doctors' diagnosis and treatment.

Acknowledgements

We thank for the funding support from the Key Research and Development Plan of China(No. 2017YFC1703306), Youth Project of Natural Science Foundation of Hunan Province (No.2019JJ50453), Project of Hunan Health Commission(No. 202112072217), Open Fund Project of Hunan University of Traditional Chinese Medicine(No. 2018JK02), and General Project of Education Department of Hunan Province (No. 19C1318).

Competing interests

The authors declare no conflict of interest.