Mining intrinsic information of convalescent patients after suffering coronavirus disease 2019 in Wuhan

YAN Shixing,Lü Yi,LIU Ziqing,REN Meng,HE Haiyang,XIAO Li,GUO Feng,PENG Miao,LI Xiaoxia,WANG Yong,XU Xi,YANG Tao,SHAO Zuoyu,HUANG Jingjing,XIAO Mingzhong

YAN Shixing,HE Haiyang,GUO Feng,LI Xiaoxia,Department of TCM Data Intelligence,Shanghai Daosh Medical Technology Co.,Ltd.,Shanghai 201200,China.

Lü Yi,REN Meng,PENG Miao,XU Xi,SHAO Zuoyu,HUANG Jingjing,XIAO Mingzhong,Hepatic Disease Institute,Hubei Key Laboratory of Theoretical and Applied Research of Liver and Kidney in Traditional Chinese Medicine,Hubei Provincial Hospital of Traditional Chinese Medicine,Wuhan 430061,China;Affiliated Hospital of Hubei University of Chinese Medicine,Wuhan 430061,China;Hubei Academy of Traditional Chinese Medicine,Wuhan 430074,China.

LIU Ziqing,Electronical Medical Records and Information Management Center,Yueyang Hospital of Integrated Traditional Chinese and Western Medicine,Shanghai 200437,China.

WANG Yong,Chinese Medicine Development research Center,Shanghai Literature Institute of TCM,Shanghai 200025,China.

XIAO Li,College of Acupuncture and Massage,Shanghai University of Traditional Chinese Medicine,Shanghai 201203,China.

YANG Tao,College of Artificial Intelligence and Information Technology,Institute of Information and Technology Nanjing University of Chinese Medicine,Nanjing 210023,China.

Abstract OBJECTIVE:To summarize the potential characteristics of convalescent patients with coronavirus disease 2019(COVID-19) in China based on emerging clinical tongue data and guide the treatment and recovery of COVID-19 patients from the perspective of Traditional Chinese Medicine tongue diagnosis.METHODS:In this study,we developed and validated radiomics-based and lab-based methods as a novel approach to provide individualized pretreatment evaluation by analyzing different features to mine the orderliness behind tongue data of convalescent patients.In addition,this study analyzed the tongue features of convalescent patients from clinical tongue qualitative values,including thick and thin,fur,peeling,fat and lean,tooth marks and cracked,and greasy and putrid fur.RESULTS:We included 2164 tongue images in total (34%from day 0,35.4% from day 14 and 30.6% from day 28)from convalescent patients.The significance results are shown as follows.Firstly,as the recovery time prolongs,the L average values of tongue and coat decrease from 60.21 to 57.18 and from 60.06 to 57.03 respectively.Secondly,the decrease of abnormality rate of tongue coat,included greasy tongue fur,putrid fur,teeth-mark,thick-thin fur,are of significant statistical difference (P <0.05).Thirdly,the average value of gray-level cooccurrence matrices increases from 0.173 to 0.194,the average value of entropy increases from 0.606 to 0.665,the average value of inverse difference normalized decrease from 0.981 to 0.979,and the average value of dissimilarity decrease from 0.1576 to 0.1828.The details of other radiomics features are describe in results section.CONCLUSIONS:Our experiment shows that patients in different recovery periods have a relationship with quantitative values of tongue images,including L color space of the tongue and coat radiomics features analysis.This relationship can help clinical doctors master the recovery and health of patients as soon as possible and improve their understanding of the potential mechanisms underlying the dynamic changes and mechanisms underlying COVID-19.

Keywords:COVID-19;tongue inspection;quantitative values;dynamic change mechanisms;radiomics features analysis

1.INTRODUCTION

In December 2019,the first acute respiratory disease case was caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2),recently officially named coronavirus disease 2019 (COVID-19) by the World Health Organization (WHO).1Person-to-person transmission has been identified through respiratory droplets or close contact.2Generally,the median incubation period for COVID-19 is three days (range:0-24 d).Fever,cough,and fatigue are the most common symptoms.Approximately 20%-30% of cases develop severe illness,and some require further intervention in the intensive care unit.Organ dysfunction,including acute respiratory distress syndrome,shock,acute cardiac injury,and acute renal injury,can occur in severe cases of COVID-19.It has been reported that critically ill patients are more likely to be older,have underlying diseases,and have symptoms of dyspnea.Oxygen therapy,mechanical ventilation,intravenous antibiotics,and antiviral therapy are usually applied in clinical management,but presently,there are no effective drugs for improving the clinical outcome of COVID-19,especially in severe cases.It was initially considered that the lung is the organ most commonly damaged by SARSCoV-2 infection since human airway epithelia express angiotensin-converting enzyme 2 (ACE2),a host cell receptor for SARS-CoV-2 infection.However,increasing numbers of clinical cases have shown cardiac,renal,and even digestive organ damage in patients with COVID-19,which is consistent with the findings that the kidney,colon,and other tissues also express ACE2 in addition to airway epithelia.The above clinical phenomena and basic research suggest a more complicated pathogenesis of COVID-19.

Meanwhile,analyzing the clinical tongue characteristics of cases with COVID-19 is urgently,since in Traditional Chinese Medicine (TCM) domain,the observation of tongue is regarded as important.3Few studies have focused on high-quality clinical tongue studies.In this paper,we focused on comprehensive analysis using novel analytical methods to fill the research gap in clinical tongue medical research by analyzing the different features and orderliness that hide behind the tongue data of convalescent patients.This study aimed to summarized the potential characteristics of convalescent patients with COVID-19 in China based on emerging clinical tongue data and guide the treatment and recovery of COVID-19 patients from the perspective of TCM tongue diagnosis.

2.MATERIALS AND METHODS

The framework of our study is shown in Figure 1.Many feature families require a region of interest (ROI) to be isolated from the surrounding voxels.An ROI intensity mask was used to extract the image volume to be studied.The excluded voxels are commonly replaced by a placeholder value.4-7This placeholder value may then be used to exclude these voxels from the calculations.The voxels included in the ROI mask retained their original intensity.

Segmentation is the most critical component of radiomics because the subsequent feature data are generated from the segmented volumes.In our study,instance segmentation-based maskRcnn method is approached to the tongue segment parts.8-10MaskRcnn is a general framework for instance segmentation,which can efficiently detect objects in a tongue image while simultaneously generating a high-quality segmentation mask for each instance.MaskRcnn extends Faster RCNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition.MaskRcnn can evade other interference information,including the face around the tongue and the tooth under the tongue.Empirically,eliminating interference information using the above methods can achieve the correct results,even though many tongues have indistinct borders.

After tongue segment part,then we use a re-segmentation method (i.e.,tongue coating segmentation) in multispaces to extract ROI part (tongue and coat),which is based on Otsu model applied to the image after a preprocessing (MaskRcnn method).11,12This model is used on RGB space when extracting thin coating,while for thick coating it is used on HSV space.In addition,the qualitative analysis of observation features is carried out.The specific experimental process and results are presented in the Results section.

2.1.Tongue qualitative values

In Traditional Chinese Medicine,observation of the tongue is regarded as important.7In particular,tongue coating reflects the nature and location of pathogenic factors as well as the condition of the stomach.The conventional way to examine the tongue for diagnosis is to have the tongue looked at directly by a doctor.However,it lacks a unique standard and shareable record.In recent years,digital cameras have become powerful tools for obtaining images.However,original pictures are difficult to compare and analyze.As a result,the extraction of tongue coating from digital photos is essential for an automatic tongue diagnostic system.13Most segmentation methods cannot effectively extract tongue images,especially tongue coatings,from the original photographs.In this study,we use a novel tongue coating segmentation method in multiple spaces.It can extract both thin and thick coatings of different colors.As shown in Figure 1,the tongue coat segmentation method is a useful tool to achieve the correct results.

Tongue diagnosis is expected to be used for preventative medicine after tongue coating segmentation because it can comprehensively check the health condition.14,15Automatic tongue detection is needed for computer-assisted tongue diagnosis.In this study,we employed deep learning-based tongue qualitative feature detection because the deep learningbased method can detect the tongue quite accurately by providing appropriate training datasets.In our study,qualitative values of the tongue include thick and thin fur,peeling,fat and lean,tooth marks and cracked,and greasy and putrid fur.

2.2.Radiomics features

The extraction of quantitative information from medical images (radiomics) holds great potential for cancer image prediction and the monitoring of therapy response.5,6These radiomic tools are promising for adding a quantitative component to existing qualitative measures and advancing personalized medicine in oncology.In our experiment,we used these tools to analyze tongue images.Thirty-one radiomic features were extracted as follows:12 from first order,seven from gray-level co-occurrence matrices (GLCM),four from gray-level run-length matrices (GLRM),three from graylevel size zone matrices (GLSZM),and five from neighborhood gray-tone difference matrix (NGTDM).The terms and features are presented in Table 1.Firstorder features were calculated from the intensity-based volume histograms.Second-order features based on the GLCM were initially developed by Haralick.16Tongue radiomic features were analyzed in this study.Note that GLCM,GLRM,GLSZM,and NGTDM were extracted from the radiation oncologist segmented ROI and abbreviations for gray-level co-occurrence matrices,gray-level run-length matrices,gray-level size zone matrices,and neighborhood gray-tone difference matrices,respectively.

Table 1 Tongue radiomic features analyzed in this study

By converting medical images into high-dimensional,mineable,and quantitative imaging featuresviahighthroughput extraction of data-characterization algorithms,radiomics methods provide an unprecedented opportunity to improve decision support in oncology at low cost and noninvasively.Previous studies have shown that biomarkers based on quantitative radiomics features are associated with clinical prognosis and underlying genomic patterns across a range of cancer types.11,17Recently,the most widely used imaging modality in radiomics research has been computed tomography,which can quantify tissue density.To our knowledge,no published study has determined whether tongue images can be evaluated using a radiomic approach.Thus,in this study,we developed and validated radiomic-based methods as a novel approach for providing an individualized pretreatment evaluation for analyzing the different features and the orderliness that hides behind the tongue data of convalescent patients.4-7,18

GLCM is one of the earliest techniques used for image texture analysis,in which texture is an important characteristic used in identifying the ROI in an image.In this study,GLCM is a distinguished arithmetical procedure for attribute removal.The GLCM is considered a dissimilar amalgamation of pixel gray levels in an image.The foremost objective of the method is to allocate an unidentified model image to a group of texture features that is distinct from its distinctive possessions.Additionally,they distinguish the textural possessions of images as spatial arrangement,contrast,roughness,orientation,etc.,which contain convincing associations through the preferred output.7

GLRM reflects comprehensive information about the direction,adjacent interval,and change range of theimage gray-level.The gray run-length matrix is one of the bases for analyzing local patterns and arrangement rules of images.The gray run matrix can be used to count the number of consecutive gray values in the same direction in an image.In an image,continuous pixels in a certain direction have the same gray value,and the gray run matrix is used to obtain the texture features through the statistics of the distribution of these pixels.6,18

GLSZM is the primary form of the Thibault matrix,which is an advanced statistical matrix of texture features and a powerful tool for medical image analysis.For graylevel images,the gray area size matrix is used to describe the bivariate conditional distribution probability density of the image brightness.The calculation method is similar to the stroke length matrix:the value in the matrix is equal to the number of areas corresponding to the graylevel.The number of rows is equal to the gray-level,and the number of columns is equal to the size of the maximum area.4-7

2.3.Statistical analysis

Mann-Whitney tests and χ2tests were applied to normal distribution variables,non-normally distributed continuous variables,and categorical variables,respectively.Pearson correlation analysis was performed to show the correlation between the two variables.Statistical analysis above were performed using R software.19All statistical tests were two-sided,andPvalues of less than 0.05 were considered significant.Box plots are used to describe the distribution characteristics of measurement tongue variables and express five characteristic values of measurement variables:(a) the lower tentacle of the box represents the minimum value without abnormal small value;(b) the upper tentacle of the box represents the maximum value without abnormal large value;(c) the bottom of the box represents the lower quartile;(d) the top of the box represents the upper quartile;(e) the thick line in the middle of the box represents the median.20

3. RESULTS

Our experimental population from the WVCSE rehabilitation station and Hubei University rehabilitation station was recovered from confirmed COVID-19.They all had confirmed COVID-19,met the discharge criteria and were in quarantine (rehabilitation period) after discharge.Day 0 was defined as the day of discharge and entering the rehabilitation station.Medical staffs from Hubei Hospital of Traditional Chinese Medicine collected the tongue pictures of the participants on days 0,14,and 28.Eliminating food,drug and other interference on the tongue,the participants were asked to take the sitting position,make the light directly on the tongue surface,extend the tongue naturally,relax the tongue body,flat the tongue surface,keep the tip of the tongue is naturally droop,expose the tongue body fully,do not overexert the tongue,extend the tongue time should not be too long.The medical staff used the same model of mobile phones to take pictures of the participants' tongue and uploaded the tongue pictures by using the APP of Chinese Medicine Huilian owned by Shanghai Daosh Medical Technology Co.,Ltd.(Shanghai,China).

The experimental contents on days 0,14,and 28 were as follows:collection between qualitative tongue values and recovery days,tongue lab mean-value features of patients with COVID-19,collection between tongue radiomic quantitative values,and recovery days.

We obtained the following results:according to statistical knowledge,there is a significant relationship between quantitative tongue values and recovery days;patients with different recovery periods have an obvious relationship with L color space and coat radiomic feature analysis.

3.1.Collection between tongue qualitative values and recovery days

Deep learning methods are adopted to obtain the qualitative values,including thick and thin tongue fur,peeling,fat and lean tongue,teeth-marks and cracked tongue,greasy and putrid tongue fur,as is shown in Table 2.13With a slow recovery time,the most affected quantitative value of the tongue was greasy tongue fur[decrease from 48.17% (355/737) to 36.87% (243/659)],putrid fur [decrease from 6.78% (50/737) to 2.58%(17/659)],teeth-mark [decrease from 25.37% (187/737)to 13.81% (91/659)],and thick-thin fur [decrease from 77.34% (570/737) to 71.47% (471/659)].From the hypothesis test results,in addition to point fat-lean,cracks,theP-value of other tongues quantitative was less than 0.05.According to statistical analysis,there is asignificant relationship between quantitative tongue values and recovery days.

Table 2 Univariate clinical tongue features analysis of patients with COVID-19

3.2.Collection between tongue lab quantitative values and recovery days

The lab model was established under the international standard for measuring the color sought by the CIE in 1931.21-23It was improved in 1976 and was named as a color pattern.The lab color model compensates for the deficiencies of the RGB and CMYK color modes.It is a device-independent color model and color model based on physiological characteristics.The lab color model consists of three features,one of which is brightness (L),and 'a' and 'b' as two-color channels.'a' includes colors from dark green (low brightness value) to gray (medium brightness value) to bright pink (high brightness value)and b from bright blue (low brightness value) to gray(medium brightness value) to yellow (high brightness value).As a result,this color blend produces a bright color.

As shown in Figure 1,we adopted a deep learning approach to obtain the tongue and coat ROI parts,respectively,and compute the L,a,and b average values of all ROI parts.As shown in Table 3 and Figure 2,there were differences in the lab average values.From the data,with the longer recovery period,the new L mean was smaller in patients with coronary disease (L average value decreased from 60.21 to 57.18,from 60.06 to 57.03,for tongue and coat,respectively).We analyzed the reasons behind this;the greater the L,the higher the brightness.In other words,the L average value decreased gradually with increasing convalescence.Table 4 gives the paired hypothesis test of 0,14,and 28 d using the Mann-Whitney and Wilcoxon rank-sum tests.24From the hypothesis test,we can also see that the P-values of the hypothesis tests at 14,28,0 d,and 28 d of recovery are all less than 0.05.According to the statistical knowledge,patients with different recovery periods have an obvious relationship with the L color space.

Table 3 Clinical tongue lab mean-value features of patients with coronavirus disease 2019

Figure 1 Framework of how to get the tongue information

3.3.Collection between tongue radiomic quantitative values and recovery days

Depth research of some radiomic features on medical images has been the subject of recent studies that have proposed corrected algorithms for NGTDM-coarseness,GLRM-GLNU,and GLRM-RLNU.4-7The tongue radiomic features analyzed in this study are listed in Table 1.The results of the models are listed in Tables 5 and 6.From the results,we observe that there are large collections between tongue radiomics quantitative values and recovery days,including contrast,entropy,inverse difference normalized (Idn),dissimilarity for GLCM,long-run emphasis,short-run emphasis for GLRM,small area emphasis,zone percent for GLSZM,contract for NGTDM,10 percentiles,maximum feature,and range feature V for first-order features.We discuss these collections later as follows:

In our clinical tongue experiment,the average values of radiomics quantitative variables associated with COVID-19 progression are shown in Table 5 and Figure 3,the recovery days correlated with most radiomics features,including contrast (increase from 0.173 to 0.194),entropy (increase from 0.606 to 0.665),Idn(decrease from 0.981 to 0.979),dissimilarity (decrease from 0.158 to 0.183) for GLCM,long-run emphasis(decrease from 346.10 to 129.58),short-run emphasis(increase from 0.289 to 0.312) for GLRM,small area emphasis (increase from 0.315 to 0.331),zone percent(increase from 0.0140 to 0.0169) for GLSZM,contract for NGTDM (increase from 0.006 to 0.007),10 percentile (decrease from 148.73 to 143.30),maximum feature (decrease from 151.22 to 146.41),range feature V (decrease from 150.45 to 145.33) for first-order feature.Meanwhile,there was no difference between the average values of the remaining features and recovery days.

As shown in Table 6,the paired hypothesis tests of 0,14,and 28 d were performed using the Wilcoxon rank-sum test.From the hypothesis test,we can also see that there are much more differences in theP-values of the hypothesis tests of 14 and 28 d,0,and 28 d of recovery;theseP-values are all less than 0.05.According to statistical knowledge,patients with different recovery periods have an obvious relationship with coat radiomics feature analysis.

Table 4 Clinical tongue color features a P-value analysis of patients with coronavirus disease 2019

Table 6 P-value of Wilcoxon test of recovery period patients between 0 and 14 d,14 and 28 d,and 0 and 28 d

Figure 3 shows the box plot of the radiomics quantitative variable values obtained at 0,14,and 28 d of patients with COVID-19 using box plots.Figure 3 illustrates the results that as the recovery days grow,radiomics quantitative variables such as CoatGLCM Autocorrelation,CoatGLCMContrast,and CoatGLRM ShortRun emphasis have already clearly appeared with some regularities.COVID-19.Undoubtedly,it is entirely meaningful and significant because we go in-depth into the potential imaging biomarkers,which enriches the physical mechanism of the tongue medical images research and enlightens a horizon who expect to touch the effectiveness of another kind of quantitative scientific information,especially specialized diseases.

Figure 2 Box plots of the lab values by 0,14,and 28 d of patients with coronavirus disease 2019

Figure 3 Box plots of the radiomics quantitative variables values obtained at 0,14,and 28 d of patients with coronavirus disease 2019

4.DISCUSSION

Many guidelines have been developed in tongue research.3,11,26,27Till date,no comprehensive tongue research by analytical methods for COVID-19 on quantitative information from tongue images has been discovered.Tongue research is far more enough in medical image research for COVID-19,let alone the analysis of the quantitative features.

Considering the situation of the global COVID-19 outbreak,methods on tongue quantitative features should be considered.1,27,28This study analysed the depth of tongue quantitative features in convalescent patients.The main motivation of our work was to discover the relationship between tongue features and recovery days.We analysed the potential critical links between radiomics features and clinical tongue information of the convalescent patients with COVID-19.To the best of our knowledge,this work is first attempting to construct quantitative information from tongue images for Several limitations need to be addressed in our paper.Also,a number of points need to be solved in our future work.Given medical images,as described by Zhanget al,6there are significant associations between radiomic features and clinical data.Theoretically,radiomics can complement other omics techniques,such as proteomics and genomics.29,30Unfortunately,we do not collect data about omics and biochemical,such as inflammatory factors,in the blood or urine,since all patients need is a nucleic acid test and collecting proteomics and genomics is not medically necessary for convalescent patients.In our future work,we will attempt to collect determine the relationship between radiomic features and natural biological mechanisms of the tongue.It can be anticipated that,in the future,the combination of several omics approaches would be the best scheme for tongue diagnosis and research.Furthermore,proteomics studies of proteins in the issue of the tongue need to be conducted to find the changes in protein structure andfunction in the diseases,which also needs to be proven by clinical experiments in our future work.

Shuet al31,32reconfirmed several well-recognized risk factors and TCM-specific phenotype markers.Also their results showed the potential effectiveness of TCM herbal prescriptions and Chinese medicine could be considered as an add-on regimen for trial use.In our experiment,we did not touch the links between effectiveness of TCM herbal prescriptions and tongue quantitative features.This area will be further studied.

In addition,he links between other kinds of constraints of Western Medicine characteristics with radiomics features of the tongue for convalescent patients is also needed to be explored in depth and investigated in future work.In conclusion,we summarize the current understanding of COVID-19 from patients’ clinical tongue information to patients’ clinical representation,focusing on the interaction between COVID-19 and the quantitative features of convalescent patients.Through an in-depth analysis of the quantitative features of the tongue images of convalescent patients,the potential relationship between them is revealed,which provides a reference for further research.

Our experiment shows that patients with different recovery periods have an obvious relationship with quantitative values of tongue images,L color space of the tongue images,and coat radiomics features analysis,which can help clinical doctors master the recovery and health of patients as quickly as possible.In addition,this study helps doctors to choose the best rehabilitation treatment according to patients’ situation.

By combining the tongue features of convalescent patients with clinical findings,we can improve our understanding of the potential tongue dynamic change mechanisms underlying COVID-19,paving the way toward the development of preventative and therapeutic solutions.We also enlighten a horizon that we expect can assist the effectiveness of other kinds of scientific quantitative tongue information,especially in specialized diseases.

5.ACKNOWLEDGMENT

The authors express their great appreciation to all coordinators,patient advisers,therapists and evaluators for their efforts.