A BPR-CNN Based Hand Motion Classifier Using Electric Field Sensors

Hunmin Lee,Inseop Na,Kamoliddin Bultakov and Youngchul Kim,

1Department of Computer Science,Georgia State University,Atlanta,30302,USA

2Nat’l Program of Excellence in Software Centre,Chosun University,Gwangju,61452,Korea

3Department of Computer Inf.&Communication Eng.,Chonnam Nat’l Univ.,Gwangju,61186,Korea

Abstract: In this paper, we propose a BPR-CNN (Biometric Pattern Recognition-Convolution Neural Network) classifier for hand motion classification as well as a dynamic threshold algorithm for motion signal detection and extraction by EF (Electric Field) sensors.Currently, an EF sensor or EPS (Electric Potential Sensor) system is attracting attention as a next-generation motion sensing technology due to low computation and price,high sensitivity and recognition speed compared to other sensor systems.However,it remains as a challenging problem to accurately detect and locate the authentic motion signal frame automatically in real-time when sensing body-motions such as hand motion,due to the variance of the electric-charge state by heterogeneous surroundings and operational conditions.This hinders the further utilization of the EF sensing; thus, it is critical to design the robust and credible methodology for detecting and extracting signals derived from the motion movement in order to make use and apply the EF sensor technology to electric consumer products such as mobile devices.In this study,we propose a motion detection algorithm using a dynamic offset-threshold method to overcome uncertainty in the initial electrostatic charge state of the sensor affected by a user and the surrounding environment of the subject.This method is designed to detect hand motions and extract its genuine motion signal frame successfully with high accuracy.After setting motion frames,we normalize the signals and then apply them to our proposed BPR-CNN motion classifier to recognize their motion types.Conducted experiment and analysis show that our proposed dynamic threshold method combined with a BPR-CNN classifier can detect the hand motions and extract the actual frames effectively with 97.1%accuracy,99.25%detection rate,98.4%motion frame matching rate and 97.7%detection&extraction success rate.

Keywords: BPR-CNN; dynamic offset-threshold method; electric potential sensor; electric field sensor; multiple convolution neural network; motion classification

1 Introduction

The EF(Electric Field)sensor extracts information by sensing the variations of the electric charge near the surface of the sensor and converts the signal into voltage-level output.Today, EF sensors are mainly categorized by two modes, contact mode and non-contact mode.The contact mode has widely been applied in the area of healthcare and medical applications by sensing bioelectric signals such as electrocardiogram,electromyogram and electroencephalogram[1–5].Meanwhile,non-contact type measures the electric potential signal on the surface of EF sensors induced by the disturbance of the surrounding electric field which is caused by movement of dielectric substances such as human bodies or hands due to coupling effect [6].A few of non-contact EF sensor systems have been applied in commercial products, while several studies in academic institutions have been reported in application areas of proximity sensing, placement identification, etc [7–10].As EF proximity sensing systems are gaining attention,recent studies have been published regarding the area of hand or body motion detection and recognition [11–21].Our past studies [11–14] were focused on noncontact EF sensing,extracting and processing the signals through EF sensors.Moreover,by utilizing deep learning algorithms such as LSTM and CNN,multiple hand gesture classification mechanisms were proposed after a series of signal processing steps.Wimmer et al.[22]introduced the‘Thracker’device that utilized capacitive sensing,which encouraged the possibility of interaction between human and computer systems through non-contact capacitive sensing.Singh et al.[15] suggested a gesture recognition system called Inviz for paralysis patients that implemented textile-built capacitive sensors,measuring the capacitance interaction between the patient’s body and the sensor.Aezinia et al.[16]designed a three-dimensional finger tracking system through a capacitive sensor,which was functional within 10 cm range from the sensor.

In this paper, we present a real-time hand motion detection and classification system adopting Biometric Pattern Recognition-Convolution Neural Network(BPR-CNN)classifiers combined with a dynamic threshold method for automatic motion detection and motion-frame extraction(Fig.1)in EF signals.Our proposed system is fully automated with real-time motion detections,extracting the true frame and classifying motion types occurring at the range of up to 30 cm near the EF sensors.Accuracies of detecting hand motions and extracting signal frames were quantitatively rated through our suggested metrics.Furthermore, we suggest the simulation results of our CNN architectures;Multiple CNN(MCNN)and BPR-CNN with other classification algorithms and empirically evaluate the hand gesture classification performance.

Figure 1:A proposed hand motion extraction and classification process

This paper is organized as follows.Section 2 describes our suggested methods of computing the dynamic threshold for motion detection and frame extraction by analyzing the intrinsic features of the EF signals.We also explain the two CNN-based motion classifiers that were designed and thus applied into our system.In Section 3,we present the results of four hand gesture classifications through multiple experiments.In conclusion,we conclude our study and suggest the future works.

2 Discussion

In this section, we explain our dynamic threshold method for motion detection and frame extraction in the EF sensor signals.After setting the optimal signal frame,we conduct normalization followed by transforming the dimensions of normalized data in order to be trained into our proposed CNN model.We implement the MCNN and BPR-CNN to effectively train the features of the transformed signals thus classifying the inputs into corresponding gestures.Note that we handle the signals that have been already processed through Low-Pass-Filter(LPF)and Simple Moving Average(SMA).Since natural frequency from the human hand or arm is known to have 5～10 Hz[23],which is a domain of extremely low frequency,thus we use the 10 Hz as a cut-off frequency in the LPF.The readers can refer to our previous studies[11–14]for more information regarding the implementation of LPF to filter out unnecessary noises and conduct the Moving Average to smoothen the filtered gesture signal from the sensor.

2.1 Dynamic Offset and Threshold

One of the challenging problems in dynamic thresholding in order to detect the signal and to locate the “genuine”signal frame is to compute an offset voltage for each Electric Potential Sensor(EPS)and adjust the threshold values periodically before detecting the target hand motion.As most hand motions and gestures are being done within a short period of time,we set the update cycle time unit to be a second.We implemented two EPS(Sensor A,B,sensor type PS25401)[14,23],and each sensor started with unidentical initial offset due to the various electric charging and discharging states near the sensors according to diverse environmental conditions in the moment of time.

Through our empirical past simulations,the initial voltagevinit∈R(unit:V)ranges from-0.2 ≤vinit≤0.02 withμ(?vinit)= -0.08 andσ(?vinit)= 0.03 whereμdenotes average andσis standard deviation,when implementing EPS to measure the capacitance changes when the subject is proximally located.The statistics were acquired from four distinct hand gesture types(Tab.1),each conducting 600 trials;n(?vinit)=600,and the distribution visualization ofvinitis shown in Fig.2a.LetS={vn|1 ≤n≤T·1000,n∈N} be a set of time series raw sensor (voltage) datavn, where the sampling ratewhichn(S)=1000 whenT=0.001sec.

Table 1: Motion types and their images

Likewise,due to heterogeneous property ofS,dynamictracks the discrete variantand its upper and lower bound, which maintains the robustness regardless of time-variant offset value.As for the time-varying signal pattern which is the electric field disturbances due to hand motion,charging and discharging the electrics triggers the voltages to display waveform.Figs.2b and 2c shows the typical output signal that charges and discharges the sensor plate which soars up(otherwise falls down),reaching its peak(bottom),then descends to bottom point(vice versa)and finally returns to an initial level when there was a hand gesture near the sensor plate.

Figure 2:(a)distribution of vinit in 800 trials.(b,c)the produced EF signals,each with sensor A,B and A–B,where is(b)after the LPF and(c)after LPF&SMA

Due to these features,dynamicare considered to be effective in order to detect our targets which limits the possible starting point of a next hand motion,and we empirically show its detection rate in Section 3.

2.2 Motion Detection and Extraction

Each individual hand motion generates unidentical signal phases depending on various conditions such as hand movement speed,distance and direction.In order to detect and extract the corresponding motion frames,we considered not only the duration,but also the time frame that could be divided into the stages of motion.The detection is composed of 4 steps where the left and right term enclosed with braces each indicate the two contrasting cases(case 1,2).The detection steps are indicated as follows;

In Fig.3,a signal of hand gesture moved from left to right(LR)is shown.As we make another hand gesture,it detects the motion and locates the following frame continuously.Fig.3 contains two motion signals;each signal obtained from two different sensors(A-red and B-blue),where the dotted lines intersected with the signals show the starting point of the frame(magenta dot)and ending point of the frame(black dot).Each red and blue dot indicates the intersection point where=vi.Through these steps we could successfully distinguish the hand motions and compute the significant frame that encompasses the authentic motion signals.

2.3 Normalization of an Extracted Motion Frame

When hand motions are identified by the sensor,the time period of the extracted frame is diverse even if the motions were the same types,due to the speed or distance range of the motion.Likewise,the amplitude of the motion signal also tends to change on every new motion since the subject’s potential electrostatic state varies through numerous conditions such as textile of the cloth,location,or nearby machines,etc[4,9,11,12].Thus,it is imperative to conduct normalization in order to be properly trained in deep learning models as normalizing input data is known to be a productive measure to enhance the performance.In our signal,the time(X-axis)and voltage(Y-axis)are the two axes that are to be normalized.For Y-axis,standardization was applied and letwhered=(n′′-|n′′-n′|·0.1)-(n′+|n′′-n′|·0.1),For X-axis,we normalizedn(into 1000, deletingd- 1000 data ink-periodical sequence, whereThis leads to= 1000, computingwhere 0 ≤m ＜(d-1000).The normalized signal is shown in Fig.4, where the dotted line indicates the extracted frame of clockwise(CW)hand gesture,meanwhile solid line is the result after the normalization.Note that even if the identical subject performs the same motion type in a homogeneous environment such as time and location,the phase of the signal is distinctive due to the constantly varying charging state.

Figure 3:Motion frame extraction visualization

Figure 4:Visualization of original clockwise hand motion frame signal and normalized frame signal

2.4 Signal Dimension Transformation

After applying the normalized motion frame, the signal frame must be transformed from 1-dimension voltage signal to 2-dimension image in order to train the CNN model.Based onwheren(=1000,we primarily reducen()into 900,deleting theandin order to compute the 30×30×1 image(900 = 30·30).Fig.5 shows the schematization of transforming the motion frame into the 30×30×1 image format.

Figure 5: Dimensional transformation of a motion frame: (a) the transformation schematization reshaped into(30×30×1)grayscale image,(b)the transformed images for 4 hand gestures

2.5 Motion Classification Through CNN Models

In this section,we define the structure of our two CNN-based classifiers;MCNN and BPR-CNN that were implemented to effectively categorize the types of input hand motion signal images.

2.5.1 Multiple CNN Classifiers with Voting Logics

The first classifier is composed of five pre-trained CNN models, which are operated in parallel with unidentical filter size in their convolution layer.Fig.6 shows a five-layer CNN structure that was applied in our MCNN.

This model was trained through the extracted feature patterns from convolution layers learning local features to global features.After the fully-connected layer andsoftmaxfunction,it outputs the class probabilityp(f(x)|x)(soft label).Once the CNN classifiers have been trained,each CNN predicts the input into a single category.Fig.6 represent that an example of implemented CNN structure.The kernel size of each Chin?is as follows;δ(C1,??)= (5×5),δ(C2,??)= (7×7),δ(C3,??)= (3×3),δ(C4,1)=(5×5),δ(C4,2)=(3×3),δ(C5,1)=(7×7)andδ(C5,2)=(3×3).

Figure 6:An example of implemented CNN structure

The outputs from the five CNNs are considered in order to make the final classification prediction through majority voting as shown in Fig.7.Each input data consists of three 30×30×1 images,which represents three channels for sensor A,B and A-B signals.These three motion signal images are extracted in real-time and inserted into pre-trained CNN classifiers in Fig.6.

Figure 7:Parallel multiple CNN with majority voting classifier

Our ultimate goal is to successfully classify the four types of hand motions (Tab.1) with high accuracy.Recall that the five CNN contains different kernel sizes,(refer to Fig.6 for detailed kernel sizes in each layer) and MCNN classifier conducts majority voting (Fig.7) between soft labels.LetH= {Ch|1 ≤h≤5,h∈N} and letδ(Ch,?)a kernel size on convolution layerCh??= {1,2},whereChdenotes a CNN model,andhis an index of the model.Ch(y,f(x),L(w,b,x))=Ph(x),whereydenotes the true class,f(x)is a prediction class and loss functionL(·)based on set of weightw,biasband inputx.Ch(·)calculates a label prediction probabilityPhand eventually computing hard labelH(x)=argVoting classifier aggregates theH?h(x), and final classification ?y=arg max(φ(H?h(x)))whereφ(Hh(x))= {γi|1 ≤i≤4,i∈N,(Hh(x)=g(i))→γi+1}where initialγ?i=0.

2.5.2 BPR-CNN Classifier

Biomimetic Pattern Recognition (BPR) [16,23–25] utilizes high-level topology features from biomimetic signals to discover certain patterns, which focuses on the concept of cognizing feature topology.Combining BPR with CNN (BPR-CNN) triggers higher performance as features are extracted from the CNN model, and BPR computes the topological manifold properties in given Euclidean parameter space based on Complex Geometry Coverage (CGC) [26] as shown in Fig.8.Manipulating the prediction probabilityP(x),it computes theηnumber of distance-based clusters?η,?η?P(xi)where 1 ≤i ≤n(x),i ∈N,η=n(classes)=n(P(x)).Since the set of trained w and b itself are not permutationally invariant,we cannot implement the distance-based geometry coverage based on the w and b.

Figure 8:Visualization of the BPR-CNN mechanism

However,P(x)would indicate the proximity between the classes and the output ofx, which guarantees the closest single class in the Euclidean space.Based on the proximity of class-wise topological space,it cognizes the matter using the high-level features.The processed input image set is abstruse to distinguish the classes or interpret the meanings of the indicated number of the pixels in the human eye,thus high-level robust features are preferred to accurately compute the decision boundary instead of using low-level features.To elaborate,input images that clearly show an object for CNN to classify the target such as cat or dog,their intrinsic features are distinct such as its eye,nose,or other parts of the subject,whereas our case doesn’t.

Our case specifically requires the robust features in high-level feature space where the trend of each hand motion signal image could be found.Implementing the BPR-CNN,we could derive better input signal classification performance compared to conventional CNN,and we validate this through experiments in Section 3.In BPR classifier,pre-trained CNN model=P(x)andφ(?P(xi))=?η,whereφ(·)indicates pairwise distance-based clustering such as K-means(K=η)[27].In Euclidean space Rη,where Rη??η?P(xi),K(xnew)=argmin1≤k≤η,k∈NdL2(μ(?k),P(xnew)),which allocatesxnewto?Kwherexnew/xi.The layer structure of the CNN was set with Conv-MaxPooling-Conv-MaxPooling- Conv-MaxPooling-Dense-Dense.The kernel size of each convolution and maxpooling layers were set with 5×5 and 2×2 respectively.

3 Experimental Results

3.1 Experiment Setting

Through the empirical experiments,we evaluate the performance of our designed methodology.Utilizing the EPS sensor[14,28],four hand motion types indicated in Tab.1 were extracted from each of six subjects, 100 gestures for each motion, collecting a total of 2400 motion samples.Among the dataset, we randomly split the 2160 samples for training and 240 for test data.The extracted raw signal was processed through consecutive signal processing methods starting from the LPF and SMA,followed by automatically detecting hand motions and setting signal frame by dynamic threshold,and normalizing the signal.Next,we transform the signal into an image and a pre-trained classifier determines its label.Note thatη=4,since our objective is to classify the four motion types(Tab.1).All this process(Fig.1)is operated in real-time and test dataset were generated and classified(Less than second when computing through CPU i7–7500U RAM 8GB).The performance has been measured through our metrics of Correct Detection Rate(CDR),Motion Frame Matching Rate(MFMR)and Detection & Extraction Success Rate (DESR).CDR shows the degree of correspondence between the signal and the actual motion, and the MFMR quantitatively assesses the matching rate of the computed motion frame.Finally,DESR is obtained by CDR multiplied by MFMR to indicate their combined accuracy level.The training epoch was set with 20 and learning rate of 0.01, withreluactivation function in each convolution layer.

3.2 Experiment Result

Following Tab.2 shows the result of our three designated metrics, which validates that the proposed method of our study works with high accuracy of around 98%on average.

Table 2: Performance of the selected metrics

Following Tab.3 displays the experiment results of the four classifier algorithms.Their performances were evaluated with classification accuracy based on identical test data of four specific hand motions in Tab.1.We denote the average of four motion accuracy as Classification Correction Rate(CCR), which is computed in Tab.3.From each motion in Tab.1, three distinct output signals are produced;sensor A value,sensor B value and subtracted value(A-B).Performance of the two CNN classifiers (MCNN and BPR-CNN) were also compared with other algorithms such as HMM and SVM.Our experiment results show that the suggested motion detection and frame extraction based on the two threshold works with high CDR and MFMR,and also the classification accuracy of BPRCNN classifier outperformed other competitive models.

Table 3: Classification correction rate of the BPR-CNN model

4 Conclusion and Future Works

In this paper,we proposed the dynamic thresholding and framing algorithms in order to set the accurate motion EF signal frame in real-time, and evaluated its performance using the following metrics;99.4%in CDR,98.4%in MFMR,97.8%in DESR.Moreover,we implemented the MCNN and BPR-CNN motion classifiers and compared the accuracy with other algorithms.Based on the extracted features of the 3 channel (sensor A, B, A-B) input signal images, BPR-CNN had shown the highest performance of 97.1% in CCR.Utilizing EF sensing is regarded as a prospective research domain and accommodates practical usage in industry due to diverse advantages such as low computation & price, high sensitivity & recognition speed.Our future work is to adopt the introduced methods to mobile devices and apply the algorithms to control the interface through noncontact hand motions.Training and classifying the diverse and detailed gestures in order to gain algorithmic robustness and versatility is a part of our future work.Combining our studies into interface technologies such as Human Computer Interaction(HCI)or Natural User Interface(NUI),we expect the further utilizations of controlling the various applications through user-friendly interfaces based on EF sensing.

Funding Statement:This work was supported by the NRF of Korea grant funded by the Korea government(MIST)(No.2019 R1F1A1062829).

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present study.