Application of artificial intelligence in tongue diagnosis of traditional Chinese medicine: A review

Zhao Chen, Xiaoyu Zhang, Ruijin Qiu, Yang Sun, Rui Zheng, Haie Pan, Yin Jiang, Changming Zhong, Chen Zhao, Guihua Tian, Hongcai Shang,＊

1 Key Laboratory of Chinese Internal Medicine of Ministry of Education, Dongzhimen Hospital, Beijing University of Chinese Medicine, Beijing 100700, China;

2 Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing 100700, China.

Abstract Tongue diagnosis is an important process to non-invasively assess the condition of a patient’s internal organs in traditional Chinese medicine (TCM) and each part of the tongue is related to corresponding internal organs.Due to continuing computer technological advances, especially the artificial intelligence (AI) methods have achieved significant success in tackling tongue image acquisition, processing, and classification, novel AI methods are being introduced in traditional Chinese medicine tongue diagnosis medical practices.Traditional tongue diagnose depends on observations of tongue characteristics, such as color, shape, texture, moisture, etc.by traditional Chinese medicine physicians.The appearance of the tongue color, texture and coating reflects the improvement or deterioration of patient’s conditions.Moreover, AI can now distinguish patient’s condition through tongue images, texture or coating, which is all possible increasingly with help from traditional Chinese medicine physicians under the traditional Chinese medicine tongue theory.AI has enabled humans to do what was previously unimagined: traditional Chinese medicine tongue diagnosis with feeding a large amount of tongue image and tongue texture/coating data to train the AI modes.This review focuses on the research advances of AI in TCM tongue diagnosis thus far to identify the major scientific methods and prospects.In this article, we tried to review the AI application in resolving the tongue diagnosis of traditional Chinese medicine on color correction, tongue image extraction, tongue texture/coating segmentation.

Keywords: Artificial intelligence, Traditional Chinese medicine, Tongue diagnosis, Machine learning, Deep learning, Color model, Tongue segmentation, Tongue image extraction

Background

TCM has made many brilliant achievements in a long history, including in the treatment of new coronavirus pneumonia (COVID-19), therefore it has been accepted by more and more people in worldwide [1-3].Tongue diagnosis[4-8] is very valuable, non-invasive and widely diagnostic method using inspection of the appearance of human tongue in TCM, which mainly depends on the observation of patient tongue.Traditional Chinese medicine believes that the tongue is connected with internal organs (Zang Fu organs in TCM terms) through meridians, so the tongue reflects the state of Zang Fu organs, Qi and blood, body fluid, and reflects the stage or degree of disease[6, 9].Tongue diagnosis (TD) is also one of the most significant basic criteria for the diagnosis of TCM and it has played indispensable role in TCM for thousands of years.Clinical studies of TCM have proved that the tongue changes rapidly and brightly during the development of the disease[10].

TCM diagnosis is based on the information obtained from the four diagnostic processes, namely Inspection, Listening & Smelling, Inquiring and Palpation[11].The most common tasks are pulse check and tongue examination.The tongue is the main taste organ that recognizes the taste of TCM, and can convey a lot of diagnostic information about the health of the human body in TCM.Tongue body reflects the most sensitive indicators of physiological function and pathological changes, which has important application value in the process of TCM diagnosis and treatment [12-15].With the popularization of TCM, people have a deeper and more comprehensive understanding of tongue diagnosis[15].Generally, tip of the tongue is related to lungs, heart, chest and neck.The central part is related to liver, spleen, stomach and pancreas.see Figure 1.Therefore, in order to detect disease, the system must inspect specific areas of the tongue [16].

Chinese medical practitioners always diagnosed the health status of a patient based on observation of these tongue characteristics.The morphology, texture, color and other characteristics of tongue coating can reflect the physiological and clinical pathological conditions of the body (such as viscera, Qi and blood, heat, cold, body fluid, etc.) and the severity or progress of the disease.The tongue features, such as tongue shape, coat, texture and color, can reflect the physiological and clinicopathological condition of the body (e.g., organs, Qi, blood, heat, cold, body fluids) and the severity or progression of the diseases[17].

Figure 1.Subdivision of tongue into areas corresponding to different internal organs.

TCM tongue diagnosis plays an important role in TCM diagnosis.But the complexity and elusiveness of diagnostic methods limit their development and popularization[18].However, the diagnosis of human diseases based on tongue condition is a standard technology of TCM, but its practicability is limited by the following factors: (1) Tongue diagnosis means that its approach is subjective, qualitative, and difficult to diagnose automatically[19]; (2) Tongue diagnosis is usually based on the ability of eyes to make detailed discrimination, and requires many years of experience and training of practitioners to obtain competence; (3) The correctness of tongue diagnosis relies on the doctor's experience [20]; (4) traditional tongue diagnosis is always dedicated to identify the syndromes other than diseases; (5) TCM tongue diagnosis was affected by the external environment (such as light and temperature); (6) The results of traditional tongue examinations cannot be described scientifically and quantitatively.Therefore, there is an urgent need to establish a modern medical system for TCM tongue diagnosis , which ought to be in the direction of the leading modern standardization of science and technology, objectivity, quantification, automation and exhibition[21].

With the improvement of digital medical imaging equipment and artificial intelligence (AI) methods, computer-aided tongue diagnosis has great potential to play an important role in clinical medicine by providing more accurate, consistent and objective clinical diagnosis[22, 23].

The history of tongue diagnosis in China

The earliest medical classics in China,Huang Di Ne i Jing, has already outlined many important aspects of tongue theory.The tongue was mentioned in 6 places inShang Han Za Bing Lun, which recorded the research on external sentimental diseases and was by Zhang Zhongjing in the late Eastern Han Dynasty.Zhu Bing Yuan Ho u L unwas a monograph on the origin of diseases, which was written by Chao Yuanfang in Sui Dynasty.When explaining the relationship between the local symptoms of the tongue and the internal body, it discussed the relationship between the tongue, meridians and viscera.In the content of "tongue theory" of Wang Tao'sWai Tai Mi Yaoin the Tang Dynasty, the anatomical content of the tongue was described, and tongue disease was regarded as a kind of oral and stomatology diseases.

Shang Han Ming Li Th eory(A.D.1156) was a book that distinguished syndromes by analyzing the main symptoms compiled by a famous physician Cheng Wuji in Song and Jin Dynasties.This book discussed the characteristics of tongue coating and its significance in syndrome differentiation.Historically, the observation of the tongue was a complete diagnosis methodology first appeared in China around the Jin (1115-1234) and Yuan Dynasties with the development and introduction of the first such manual in the world, theAo Shi Sh ang Han J in Ji ng Lu.However, the total representation of the tongue and the theoretical contents introduced up until the Yuan Dynasty (1271-1368) were so extremely limited that they can only be perceived as an introduction to tongue diagnosis.Xue Ji, a physician in Ming Dynasty (1529), first put forward the view that the disease of tongue was related to five zang organs in his workKou Chi Lei Yao.Furthermore, Wu Zhiwang added another organ, the kidney, to the tongue and meridians in his workJi Yin Gang M uin 1625.Shen Yan Yi Zo ng J in J ingwas written by Wang Jinghan, a physician in the late Ming and early Qing Dynasty.This work was the first proposed to establish the viscera division of the tongue.Most of the 120 tongue pictures recorded inShang Han She J ian, written by Zhang Deng in 1728, were discussed according to the pattern of six classics.The division of meridians and collaterals in the tongue was first proposed inShang Han Di Yi Sh u, which was written by Shen Yueguang in the Qing Dynasty (1780).Wu Kunan recorded two types of tongue organs inShang Han Z hi Zh ang(self-prologue in 1796).The lung in the tongue was further described inYi Shu(self preface in 1826).Fu Songyuan'sShe Tai To ng Zh i(1874) is a monograph on the diagnosis of internal injuries.He successfully established the tongue diagnosis method of internal injury, the key of which is that the author changed the previous method of taking tongue coating diagnosis as the key point of tongue diagnosis, and put forward the concept of taking tongue color diagnosis as the key point.Liu Hengrui'sCha She Bian Zhe ng Fawas the first one to introduce the anatomy and physiology of western medicine into the tongue diagnosis of traditional Chinese medicine.After Liu Hengrui, He Lianchen revised the section of "syndrome differentiation by observing tongue" in Wu Kun'an'sShang Han Zhi Zhang, which was renamed asGan Zheng Bao Jian, and then was published in 1912.Cao Bingzhang had compiled a guide to tongue discrimination (Bian S he Zhi Nan),which comprehensively collated and combined the knowledge of tongue diagnosis in traditional Chinese medicine and western medicine in 1920.In 1933, Qiu Junsheng and Qin Bowei edited, revised and published the book of tongue diagnosis of national medicine (Guo Yi She Zhen Xue).The first section of the first part is the definition of glossology.Zhong Yi She Zhenwas first published by the Beijing University of Chinese Medicine at the People's Medical Publishing House in 1960.This book systematically integrates the knowledge of diagnosing rare tongue coatings based on the theoretical system of the motherland medicine.Diagnostic and Therapeutic Atlas of Moden Chinese Medicine (Chinese and English version), mainly edited by Professor Xu Jiatuo in 2009, was divided into three parts: the principle and method of tongue diagnosis, the content of tongue diagnosis and the clinical application of tongue diagnosis.The characteristics, typical images and clinical significance of common tongue images are introduced in detail.For more than 70 years in New China, TCM tongue diagnosis has made great progress in objectification and standardization under the promotion of clinical diagnosis practice and scientific research of research institutions and experts and scholars of TCM, see Figure 2.

Figure 2.The historical development of tongue diagnosis in traditional Chinese medicine

Why AI with tongue diagnosis?

In its essence,tongue image analysis is interdisciplinary in many aspects.In recent years, for the purpose of TCM tongue diagnosis academic research, significant AI technologies, including, image classification, tongue image extraction, tongue texture/coating segmentation, have been widely used in tongue diagnosis to improve its diagnostic robustness and accuracy.Widespread application of AI in health care has been anticipated for half a century [24].The importance and rapid development of AI has established an independent and rapidly developing branch in the field of TCM, and also led to the urgent and extensive demand for tongue diagnosis technology.

In clinical practice of TCM, practitioners observed the characteristics of tongue, such as color, shape, coating, texture and saliva quantity, and then deduce the main diseases of the patients.The TCM tongue diagnosis usually depends on medical practitioners’ experience and knowledge, this can't cause the diagnosis of subjectivity and difficult to repeatability.Traditional tongue diagnosis is mainly to observe the tongue color, shape, coating, etc.However, the way of acquiring tongue feature information is often biased by subjective judgment, originating from human vision, color perception or interpretation.In order to reduce the error of subjective judgment, many researchers have proposed artificial intelligence methods based on tongue texture, coating, color and other features through image processing and pattern recognition [16].

AI has become a fundamental part of the revolution that has learned the experience of tongue diagnosis from famous and talent TCM physicians at an everincreasing pace over the past decade in TCM [2].Image classification, tongue image extraction and tongue texture / coating segmentation were widely used in tongue diagnosis to improve the validity and accuracy of tongue diagnosis[24].AI was realized by a system that combines representation learning with sophisticated ratiocination [25].Chuang-Chien Chiu [26] proposed a novel method of TCM tongue diagnosis based on color and spatial texture features and the overall accuracy of tongue diagnosis was more than 86%.Mingjun Gui, et al.[27] used label distribution for multi label learning of tongue color, and described tongue image through the description degree of color labels.This approach can also be regarded as a summary of the diagnosis results of the same sample by different doctors, which is in line with the reality of TCM diagnosis.

There are no precise or quantifiable standards existing in TCM tongue diagnosis and the examination outcome in traditional tongue diagnosis could not be described scientifically and quantitatively.Hence, it was an inevitable trend to electronize the tongue image and make the AI learn from the clinical practitioner's subjective data how to classify a patient’s health status by extracting meaningful features from tongue images [28].However, in the process of tongue image electronization, the specific device-dependent color space is difficult to compute the color difference that can be useful for the tongue texture/coating segmentation and diagnosis [28].Due to the strong subjectivity and difficulties in reproducing results of the traditional tongue inspection, so it was difficult to guarantee the diagnosis quality of the relationship between disease and tongue image.Therefore, it was urgent to use AI technology to modernize TCM and realize the quantitative and standardized processing of tongue images.

Deep learning method has been widely used in medical image processing and provided technical support for auxiliary diagnosis and treatment.The deep learning algorithm applied to tongue image processing was mainly based on convolutional neural network (CNN).Many studies have been conducted on tongue image using CNN.Houet al.used CNN to study the details and characteristics of tongues [21].In the twoclass experiment, the CNN model has achieved better results in the classification of tongue image syndrome compared with traditional machine learning methods such as SVM, multilayer perceptron network (MLP) and random forest (RF)[15].

Tongue color and color correction

Tongue color

In tongue diagnosis, the color information of tongue kept valuable information about disease status and its relationship with internal organs[29].Tongue color could also provide beneficial information on blood congestion, water imbalance, and psychological problems.The tongue color recognition with high accuracy will contribute to the efficiency of TCM diagnosis [21].

In recent years, the research of tongue color classification based on the combination of TCM and computer science has been widely emphasized and implemented[21].Tongue color is one of the most obvious features in TCM tongue diagnosis, which is used by physicians to diagnose diseases or potential diseases[7, 30].However, the whole tongue image analysis needs to be processed in a color format and it is particularly important to choose a suitable color model to build an intelligent tongue diagnosis model.Generally, there are currently 10 commonly used color models, namely RGB (Red, Green, Blue), HSV (Hue, Saturation, Value), HSI (Hue, Saturation, Intensity), HSL (Hue, Saturation, Lightness), CIELAB, CIEYxy, CIELCh, CIEXYZ, CMYK, CIELUV.The color gamut of a tongue image in different color spaces for tongue diagnosis are described what kind of light needs to be emitted to produce a given color.

RGB and HSV are different representations of color space.RGB color model is an additive color model and a kind of color space for equipment, which red, green, and blue light are added together in various ways to reproduce a broad array of colors (https://en.wikipedia.org/wiki/RGB_color_model), see Figure 3.RGB is usually the basic color space of most applications, and is often used in most computer applications, because there is no need to convert the information displayed on the screen.HSV (hue, saturation, value) is an optional representation of RGB color model, which can be obtained by RGB conversion.HSV, the color model for visual perception, makes use of three basic properties of color to represents color, where H represents hue, S for saturation, V represents Brightness [31, 32].

Hue is the main acceptable color for the observer, and it is an attribute to describe the pure color; Saturation is a measure of how much pure color is diluted with white light; Value is a subjective description, which embodies the concept of colorless intensity and is the key parameter to describe color perception[33].

HSI color model uses three components to represent each color: hue (H), saturation (s) and intensity (I).Figure 4 illustrates how the HSI color space represents colors.HSI (hue, saturation, intensity) color model is also an ideal tool to develop image processing algorithm based on color description that are natural and intuitive to humans [13].Due to the particularity of the application of tongue image extraction, the HSI color model can achieve better segmentation results than other methods in the clinical diagnosis of tongue image in TCM[34, 35].

The CIE (the International Commission on Illumination) system is based on the description of color as luminance component Y and two additional components X and Z[36].The magnitudes of the XYZ components are proportional to physical energy, i.e., any color is represented by a positive set of values [20].CIE XYZ color space (also known as CIE 1931 color space) was created by the International Commission on lighting in 1931.CIEXYZ color space is usually used as a reference color space and as an intermediate color space independent of the device.In fact, it's usually convenient to talk about "pure" colors without brightness [18].While CIE XYZ is used to report color from measuring instruments, it is not so useful for humans to describe color.CIELAB color space (also known as CIE L*a*b* or sometimes abbreviated as simply "Lab" color space) is a color space defined by CIE in 1976 (http://conceptmap.cfapps.io/ wikipage?lang=en&name=CIELAB).CIE L*a*b* is one of the color spaces independent of the device and it represents the color as three values: L * represents the brightness from black (0) to white (100), a * represents the brightness from green (-) to red (+), b * to blue (-) to yellow (+).The CIELab lightness L* remains unchanged.HCL (Hue-Chroma-Luminance) or Lch refers to any of the many cylindrical color space models that are designed to accord with human perception of color with the three parameters.The CMYK color model (also known as process color, or four-color) is a subtractive color model based on CMY color model, which is used for color printing and also to describe the printing process itself.CMYK refers to the four ink plates used in some valence color space, which is the update of CIE1964 (U*, V*, W*) color space (CIEUVW) (https://en.wikipedia.org/wiki/CIELUV).Since both of the L channels in CIELUV and CIELAB represent the lightness perception in human visual system, it is only used once in calculating color features.

Figure 4.HSI color model

Figure 3.RGB color model

CIEXYZ and RGB systems are far from consistent in perception.Therefore, CIE standardize two systems based on CIEXYZ, CIELUV and CIELAB, whose main goal was to provide a perceptual equal space.This means that Euclidean distance between two colors in CIELUV/CIELAB color space was closely related to human visual perception.Both of the L channels in CIELUV and CIELAB indicate the sensation of the lightness in human vision system.Bo Pang et al.[4] used the remaining four color spaces (RGB, cieyxy, CIEluv and CIELAB) to extract quantitative features of color, and the experimental results are encouraging.The summary of tongue acquisition in color models was in Table 1.

Color Correction and AI

Color correction is to use a certain mapping relationship to change the color value of the target image to "correct" any deviation from the standard image, also known as color transfer.It can be used to eliminate the color difference between images, or transfer the color style of the reference image to the target image[28, 37].Color correction could be used to cover errors made with camera settings as well as to pull more information from flat-profiles.If the color distortion in the tongue picture is serious, it will have a considerable negative impact on the diagnosis of TCM, and thus might result in incorrect diagnosis and prescription, which thus even does harm to the health of patients [38].The colors in images captured or rendered on one device may be different from those in images captured or rendered on different devices, which makes it difficult to reliably and meaningfully exchange or compare images [1,15].

Therefore, in order to render the color image in high quality, reduce the error caused by the device, better reflect their actual colors, and obtain the same colors as those perceived by our visual system directly, color correction is necessary for accurate image acquisition and minimizing the impact of color distortion on the diagnosis results [39].Color correction method to ensure the color consistency between multi-device views and make the color closer to the perception of human vision [38].It is usually calculated by means of the so-called "perceptual uniform color space", such as CIE L * b *, which is one of the color spaces independent of the device [15].To effectively reduce the difference, a large number of color correction algorithms based on AI for digital cameras have been introduced [39].

Table 1.Color models in tongue color acquisition

Color correction can make a color similar to the color observed by people, and then AI can diagnose human diseases or symptoms by imitating the identification of human tongue diagnosis from the perspective of human after color correction for device views.Since the color gamut of the human tongue images is much narrower than the whole visible color area, these existing algorithms cannot be applied to AI tongue image analysis directly, but need to be further optimized and improved.Some scholars divide color correction into global color correction algorithm and local color correction algorithm.The global color correction algorithm uses the same mapping function for all pixels in the image, and the local color correction algorithm uses different mapping functions for different regions of the image [28].Qi Wang, et al[38] proposed SIFT keypoint matching algorithm — OF-SIFT to avoid the problem that the global correction technique had poor correction effect in the local areas of an image.

The correction algorithm is generated by comparing several popular correction algorithms, i.e., ridge regression, polynomial-based regression (variableexponent regularization polynomial regression), SVM, and neural network mapping algorithms[18, 40].Ping Dai and Jieyue Yu[41] proposed to use local polynomial fitting to modify the features of the file to achieve the purpose of color correction.This research chose 9 items as local polynomial to achieve good color correction effect.A more reasonable mapping algorithm based on variable-exponent polynomial regression was proposed to evaluate the mapping matrix coefficients between RGB and XYZ color spaces [39].In order to circumvent the subjective and qualitative problems of traditional tongue diagnosis, Zhang et al.presented a novel computer aided tongue diagnosis system (CATDS).Experimental results showed that the proposed tongue image acquisition device and the SVR based color correcting algorithm were competence for computer aided tongue diagnosis system (CATDS) [45].

With the development of deep learning technology, more and more tongue color classification studies have introduced this technology [24] and the neural network was always used as a post-processing optimization step for color correction.Yuanyuan Fan et al.[28] proposed a framework of integrating depth neural network into stereo image color correction algorithm.The algorithm was effective, which could speed up the optimization speed and obtain better optimization effect.Xiaoguang Li, et al.[42] proposed a Two-phase Deep Color Correction Network (TDCCN) to eliminate the color distortion between the acquired tongue image and human visual perception image.

Mapping the tongue color to ZHENG and diseases with AI

The tongue could objectively reflect the states of a disease or TCM symptom, which could help differentiate syndromes, establish treatment methods, prescribe herbs and determine prognosis of disease [43].AI for medical images has been a powerful tool in the diagnosis or assessment of diseases [44], see Figure 5.ZHENG, TCM syndrome, is an integral and essential part of TCM theory.TCM practitioners usually observed the tongue color and coating to determine ZHENG (such as Cold or Hot ZHENG) and to diagnose different stomach disorders including gastritis.There is a close relationship between traditional tongue diagnosis and syndrome recognition, but the understanding of tongue diagnosis in western medicine and modern biomedicine is not very clear[5].Tongue appearance and characteristics were important tools for TCM diagnosis of patients' syndromes.Most patients with gastritis were classified as Hot or Cold ZHENG and were identified with a color label (yellow or white) based on the observed color of the coating on their tongue, as determined by the TCM clinical practitioners [11].Ratchadaporn,et al.[14]used machine learning technology to extract color features from 311 tongue images, consisting of tongue images from 263 gastritis patients and a control group of 48 healthy volunteers and established the relationship between color features extracted from tongue images and ZHENG.This research has applied three different supervised learning algorithms (SVM, MLP, RF) to build classification models for training and evaluating the proposed ZHENG diagnosis system.

Figure 5.The scheme of mapping the color image to ZHENG and diseases

As tongue diagnosis has played a prominent role in the diagnosis and subsequent treatment of disease, it has received more and more attention, both in clinical medicine and in biomedicine [4, 5].The clinic study of TCM has proved that the change of the tongue is rapid and brilliant in the developing of the diseases when mapping from tongue signs to Western medicine defined diseases [4, 5].So it is necessary to build the AI models to predict patients’ illness caused by physical and mental disharmony with the subjective and quantitative precise tongue color quantification.The research showed that there was a certain relationship between tongue color and the state of the human body, which can be used in medical applications to detect various illnesses with a new painless and efficient way [29].SVM [11] and Bayesian network [5, 13, 45] were employed to build the mapping relationships between color features and diseases, respectively[18].The estimate prediction accuracy of the Multi-class SVM classification is 86.6%, which outperforms the Joint Bayesian network classification [18].These results showed that SVM could be efficie-ntly build the mapping relationships between extracted tongue pathological features and diseases.J.Zhang [11] also used SVM to establish the classification model for diabetes based on standardized tongue image, the accuracy rate of diabetes predication was increased from 77.83% to 78.77%.It indicates the feasibility of using the information science method to carry out TCM diagnosis.Nur Diyana Kamarudinet al.[46]presented a two-stage tongue’s multicolor classification based on a SVM whose support vectors were reduced by our proposed k-means clustering identifiers and red color range for precise tongue color diagnosis to overcome this ambiguity in the judgement of tongue.Zhang et al.[45] proposed the BN classifiers based on quantitative chromatic and textural measurements to classified six groups: healthy, pulmonary heart disease, appendicitis, gastritis, pancreatitis and bronchitis.This method had got diagnosis accuracy over 75%.Huang et al.[13] proposed a quantization vector oriented scheme for tongue color analysis, and employed BN to model the relationship between these quantization vectors and tongue color categories.The effectiveness of this method was tested on 418 tongue images, and the classification results were given.The detailed summary of different AI algorithms used for tongue color in Supplementary Table 1.

AI segmentation of tongue coating, body and shape

Over the past thousands years, tongue diagnosis has been proved to be one of the most valuable and the most extensively applied TCM diagnostic approaches in clinical practice.Tongue is an organ of the human body for tasting sense, which carries abound of information of the health status [20].Healthy conditions can be known from observation of the surface tongue by an expert [12].The color, moisture, shape, size and texture of tongue reveal the overall health condition and dysfunctions of specific organs [52].Because tongue image segmentation is an important procedure in the tongue characterization, its accuracy affects the following automatic process directly [4].

Tongue diagnosis is one of the essential methods in TCM diagnosis.The accuracy of tongue diagnosis could be improved by the objectification [43].A suitable method was to combine the clinical experience of TCM experts with modern information technology to achieve quantitative, objective and standard tongue examination.Among the characteristics of the tongue, the color and coating of the tongue body and its distribution were the main contents of the tongue examination[40].Many researchers have devoted themselves to the field of tongue image analysis and have put forward many mature and stable methods and solutions.These include the study of the texture features [22, 23], color features [53, 54] and segmentation of tongue images [15], shape features [30, 55,56], tongue coating [57, 58], tongue surface [16, 59] and other part [50, 60, 61].

Automated tongue segmentation

Automatic tongue area segmentation is crucial for AI aided tongue diagnosis, and it has received much attention in medical image analysis and TCM tongue diagnose.In recent years, the development of automatic tongue segmentation technology has aroused extensive research interest.Due to the complexity of the pathological tongue, the change of tongue shape and the interference of the lips, this technology is difficult to achieve [19].Among these methods, the representative ones are active contour method, level sets method, region growing and merging, random walk method, and so forth [51].However, the complexity of pathological tongue, variance of tongue shape, and interference of the lips make automated tongue segmentation very challenging [19].For any tongue segmentation algorithm, not only the tongue itself but also the complex background should be considered.Usually, the background of the captured environment includes face, beard, lips, teeth and so on, which makes tongue segmentation difficult [62].

Active contour method (ACM), also called snakes, as one of the most important classes of deformable shape models, are a sophisticated method for contour extraction and image interpretation.ACM was widely used in applications like image segmentation, object tracking, shape recognition, edge detection and stereo matching [51].ACM was a model-based image segmentation method developed in the late 1980s.ACM could be divided into two types: parametric active contour model and geometric active contour model.The geometric method was the intrinsic model[63].The classical geodesic active contour (GAC) model has only local segmentation property [64].

ACM algorithm and has been widely used in many clinical imaging systems Snakes or ACM [2] have shown their great performances are the key methods for image segmentation, which is an effective and accurate method to segment out the regions of interests (ROIs) [65].Generally, tongue body segmentation usually involves two major steps: edge enhancement and detection of the tongue body contour [66].Jifeng Ning et al.[67] has presented a region merging-based automatic tongue segmentation method of watershed algorithm and snake algorithm combination for traditional Chinese tongue diagnosis.Zhenchao Cui et al.[66] proposed a novel automatic tongue segmentation method, which can well address the problem of edge enhancement and initialization of tongue contour.An edge enhancement detector based on Gabor amplitude was proposed for edge enhancement.Gradient vector flow (GVF) snake was used for tongue body segmentation, and augmented Lagrange method (ALM) was adopted for for fast GVF calculation.Junghoon Lee et al.[68] proposed a semiautomatic segmentation method based on dynamic MRI for three-dimensional motion analysis of tongue.This method required a small amount of userinteractions only at initial stages to guide the algorithm.Random walk (RW) was used to segment time superimposed volume in all time frames, and then 3D super-resolution volume segmentation was performed, which could realize accurate and robust automatic segmentation of time-varying structure.Wenshu Li et al.[69] compared the Canny, Snake and threshold (Otsu's thresholding algorithm) methods for edge segmentation, the threshold method using Otsu's thresholding algorithm and filtering process can achieve an easy, fast and effective segmentation the tongue coating and tongue body.Hongzhi Zhang [19] proposed a novel automated tongue segmentation method, which combined the polar edge detector, edge filtering, adaptive edge binarization, and an ACM to automatically segment tongue body from an image.The experimental results demonstrated that the proposed method could segment the tongue body accurately and effectively.

Yuke Wei et al.[70] used self-organizing map (SOM) network to simulate the process of medical diagnosis in order to realize the automatic classification of tongue images.Meanwhile, the SOM network proposed in this paper integrates the prior knowledge of tongue clinical probability, which not only avoids the situation that other weight initialization methods will be easily clustered to the same network, but also improves the convergence speed.

Tongue segmentation and AI

A computer tongue coating detection system (CTEs) based on chromatic and textural algorithm was developed to quantify the tongue coating since the color feature alone was not enough to display a large amount of information on the tongue coating[26, 71].The main information that must be acquired for a diagnosis would be about the tongue (tongue body and tongue coating) and these must be classified for tongue region analysis [20].The accurate separation of tongue body from tongue image is the premise of recognition and diagnosis.The color of the tongue coating and tongue body contain much information about physiological and pathological status within human body from the viewpoint of TCM [69].

As the first step of tongue image analysis, tongue segmentation plays a significant role in this field [72].Tongue segmentation refers to the segmentation of the tongue into different regions, which are homogeneous or similar in some features, but inhomogeneous in some features, such as intensity, coating or texture[62].Tongue image segmentation was the premise of computer tongue image diagnosis, many tongue image segmentation algorithms have been proposed.The segmentation process starts with the selection of the region of interest, and the background was removed from the original image[73].The simple segmentation method is limited by the weak edges of the tongue and the details of the tongue surface, so a unique segmentation method based on the combination of watershed transformation and ACM is proposed [74].

Many different image segmentation techniques were presented, such as thresholding, edge based segmentation, region based segmentation, Neutral Network and also covered finding threshold value for image segmentation [62].In particular, the region based image segmentation technique could provide closed region boundaries and is robust to random noise due to its statistical basis Specifically, region-based image segmentation techniques could provide closed region boundaries and were robust to random noise due to their statistical basis [75].Edge-based methods could accurately locate part of the edge, and the detected boundaries were likely to be an open curve.In addition, edge-based methods were very sensitive to noise and often produce false edges.However, these two commonly used segmentation methods (edge based or region based) had their own limitations, especially for medical applications, it is often difficult to obtain satisfactory results.Kebin Wu and David Zhang proposed a robust tongue segmentation method by fusing region-based and edge-based approaches to avoid the limitations of the common segmentation methods (edge-based or region-based) [76].Zhi Liu et al.proposed a novel tongue segmentation method that adopted hyperspectral images and the SVM to address the problem of traditional intensity-based segmentation methods that make use of monochromatic images cannot provide accurate and robust results [73].Mingfeng Zhu et al.[51] proposed a novel method to apply multiobjective greedy rules and makes fusion of color and space information in order to extract tongue image accurately.Pengzhao Sun et al.[12] had proposed a fast tongue image segmentation algorithm based on convolution neural network to address the robustness problem of tongue image segmentation algorithm.The network was inspired by shufflenet, which provided an efficient classification and detection network.The average segmentation accuracy is about 90.5%, which has great real-time application potential[12].To ensure the feasibility and reliability of tongue diagnosis, a robust and accurate tongue segmentation method is a prerequisite [76].In recent years, various kinds of image segmentation methods have been applied to the application of tongue image extraction.

Tongue shape segmentation and AI

The tongue shape could be used to determine a patients’ illness [3, 4, 20].Tongue shapes collected from different people (different diseases) were completely different, so it is impossible to describe them correctly with a predefined deformable template[76, 77].Chao et al.[78] proposed a tongue segmentation method based on the combination of tongue shape features and snakes correction model.Firstly, a rough tongue contour was obtained by using the features of tongue image in HSI color model.Then, a preliminary tongue contour was corrected by using the tongue shape features.Finally, the result was applied to snake model to get the final result.Tayo Obafemi-Ajayi et al.[22] proposed a novel set of features, based on shape geometry and polynomial equations, for automated recognition and classification of the shape of a tongue image using supervised machine learning techniques.The feature set proposed is robust and thus can be extended to other shape recognition and classification applications [22].The authors in [73] proposed an approach to automatically recognize tongue shapes based on geometry features.Experimental results conducted on 362 tongue images exhibited an accuracy of 90.3% for shape classification.Bo Huang et al.presented a fuzzy fusion framework classification approach that combines seven Analytic Hierarchy Process (AHP) modules for automatically recognizing and analyzing tongue shapes based on geometric features.Experimental results show that this shape correction reduces the deflection of tongue shapes [30].

Li et al.proposed a method to segment the tongue using multiple color spaces with revealing the chromatic feature of image based on hue channel in HSV space and using the luminance features in L*a*b color space.The method was applicable to most tongue images with various tongue colors and shapes and over 70 percent images yielded satisfactory segmentation results [72].Nevertheless, there was little quantitative analysis between tongue shape and the relationship to its current health state.Bob Zhang and Han Zhang adopted a decision tree to discriminate healthy and disease tongue images based on 5 tongue shapes (rectangle, acute and obtuse triangles, square, and circle).The Experimental results showed that the extracted geometric features were effective in tongue shape classification (coarse level) with a large dataset consisting of 672 images comprising of 130 healthy and 542 disease examples (labeled according to Western medical practices) [30, 52, 73].

Tongue coating/surface and other part

For thousands of years, Chinese medical practitioners have diagnosed the health status of a patient's internal organs by inspecting the tongue, especially patterns on the tongue's surface [13, 14].In addition, tongue coating was of important meaning in TCM clinical tongue diagnoses and to extract tongue image with coating was of a certain difficulty [51].Tongue image with coating has an important clinical diagnostic significance, but the traditional tongue image extraction method is not capable of extracting the thickly coated tongue image.The chromatic algorithm, which was based on the hue, saturation, luminance (HSL) color model, served to identify tongue color and tongue coating thickness, while texture algorithm is used to detect dirty tongue coating[71].Du Jian-qiang et al.[74] proposed the fuzzy C-means (FCM) to separate tongue body and tongue coating from color tongue images.By analyzing and processing the hue histogram of tongue image, the number of clusters and the initial cluster center were automatically determined.

In state-of-the-art computerized tongue image analysis, color and texture features were the most prevalent [52].The tongue diagnosis comprises recognizing the symptoms/disease-related colors on the tongue surface and analyzing the texture.There are many pathological details on the surface of the tongue, which have a large influence on edge extraction [77].Zhi Liu et al.proposed a novel automatic tongue surface classification method which makes use of hyperspectral medical images analysis and the SVM classifier [20].Further, Zhi Liu et al.proposed a method (HSI + SVM) combining HSI and SVMs, which used the spectral variability of different tissue types and hyperspectral images to distinguish the tongue and surrounding tissues.The experimental results and the corresponding quantitative evaluation showed that the proposed hybrid HIS-SVM method was more effective and robust than the traditional method [73].B.Saritha presented the improved level set curve evolution for tongue contour extraction based on an automatic initialization of contour by the feature of tongue in the HSV color space [43].

The Deep Neural Network (DNN) architectures in general can automatically extract features avoiding feature selection and reduce manual steps, which are key elements to enable translation of such systems in to the clinical practice.Chen et al.used ResNet34 CNN architecture to extract features and perform classifications for the recognition of tooth-marked tongue.The overall accuracy of the models was over 90% [79].Dan Meng et al.has proposed deep CNN model for deep tongue image feature analysis to learn high-level features in the training process and provide more classification information, so as to improve the prediction accuracy of test samples.The experimental results showed that the proposed model can divide the given tongue image into health and disease states, and the average accuracy is 91.49%, which demonstrates the relationship between human body's state and its deep tongue image features [23].

The tongue body and tongue coating separation effect conforms to the view point of TCM better than that by threshold value segmentation method.Siu Cheung Hui et al.[71] has applied different machine learning algorithms for tongue diagnosis.A total of 21 different tongue features and attributes, 24 categories, were identified for the establishment of classification model.Five different machine learning algorithms, including ID3, j48, Na?ve Bayes, BayesNet and SMO were applied to 457 tongue datasets.Therefore, SMO has achieved very good performance results in terms of accuracy and AUC[71].

Most of the existing methods classify the tongue image by pixel without considering the area information, which is very important for Chinese medicine doctors to extract the color distribution of the tongue material and coating when diagnosing the tongue.Yong-Gang Wang et al.has presented a colortexture segmentation algorithm to obtain a number of homogenous regions; then these regions were classified into different categories of colors of substances and coatings based on Earth Mover’s distance (EMD) [80].The detailed summary of different AI algorithms used for tongue texture segmentation in Supplementary Table 2.

Discussion and Conclusion

Currently, there were some classical TCM AI publications have been published.The TCM brain introduced in the book "AI Qihuang TCM Brain Medical Records" was the deep integration of clinical big data and contemporary AI accumulated in the TCM field for thousands of years.The connection between the main evidence and the TCM pairs were vividly remembered.With its assistance, human beings could save ten years of hard work in learning TCM and get twice the result with half the effort in clinical practice.This book described more than a dozen cases, all of which used AI to interpret classical prescriptions, grasp the main syndromes, establish classical prescriptions database.

Advances in biological and medical technologies have been providing us explosive volumes of biological and physiological data, such as medical images, electroencephalography, genomic and protein sequences.Learning from these data facilitates the understanding of human health and disease.

TCM tongue diagnosis is based on the basic theories of traditional Chinese medicine and reflects the process of making judgments on human illnesses through tongue diagnosis in ancient China.TCM Tongue diagnosis contains a diagnostic method refined and sublimated by ancient doctors through long-term clinical practices and summary of changes in the tongue (tongue image, tongue coating and tongue texture, etc) before and after human illness, or the difference between normal people and patients in terms of tongue image, tongue coating and tongue texture.However, each TCM clinical expert, due to different visual sensitivities, does not fully agree on the judgment of the same tongue (tongue texture, color, coating, etc) which leads to inconsistent diagnostic results for the same patient.

Through this review, we need to give some suggestions to researchers in the field of AI in Chinese medicine.

1).HSI (hue, saturation, intensity) was closer to human vision, and achieved better segmentation results in [26-28, 51].

2).When performing multi-modal information fusion of TCM tongue diagnosis, using CIELUV/CIELAB, CIE L*a*b* could ensure the color stability obtained by different devices.

3).RGB was often combined with CIEYxy, CIELUV, and CIELAB as the comprehensive color model to construct a tongue image diagnostic prediction model, and achieved good prediction results.

4).In order to reduce this visual discrepancy, we need to electronicize the tongue image information to ensure consistent information on the color, size, length of the same tongue, the thickness of the tongue coating.The machine learning methods could learn from the experience of experts in the field on tongue diagnosis, from the purpose of achieving fast and accurate prediction of the patient's disease.It was also essential to realize the standardization of tongue diagnosis, and there are many issues needed to be resolved, such as, separating the tongue coating from tongue nature, analyzing the colors of tongue and its coating, the thickness and moisture of tongue coating, and quantify the cracks of the tongue in TCM tongue diagnosis with modern AI technologies.Hence, Chinese medical practitioners rely on AI-driven tongue image analysis to assist them make more accurate, consistent clinical diagnosis.

Supplementary materials

Supplementary Table 1 and Table 2.

Supplementary Table 1.Performance of different ai algorithms used for tongue color

?

Supplementary tables 2.Performance of Different AI Algorithms Used for Tongue texture segmentation

?

?

?

?

?