Poisoning attack detection scheme based on data integrity sampling audit algorithm in neural network

Zhao Ningning Jiang Rui

(School of Cyber Science and Engineering,Southeast University,Nanjing 210096,China)

Abstract：To address the issue that most existing detection and defense methods can only detect known poisoning attacks but cannot defend against other types of poisoning attacks,a poisoning attack detecting scheme with data recovery (PAD-DR) is proposed to effectively detect the poisoning attack and recover the poisoned data in a neural network.First,the PAD-DR scheme can detect all types of poisoning attacks.The data sampling detection algorithm is combined with a real-time data detection method for input layer nodes using a neural network so that the system can ensure the integrity and availability of the training data to avoid being changed or corrupted.Second,the PAD-DR scheme can recover corrupted or poisoned training data from poisoning attacks.Cauchy Reed-Solomon (CRS) code technology can encode training data and store them separately.Once the poisoning attack is detected,the original training data is recovered,and the system may get data from any k nodes from all n stores to recover the original training data.Finally,the security objectives of the PAD-DR scheme to withstand poisoning attacks,resist forgery and tampering attacks,and recover the data accurately are formally proved.

Key words：poisoning attack; neural network; deep learning; data integrity sampling audit

Deep learning (DL)[1]is the foundation for several modern artificial intelligence applications.It has rapidly matured and entered safety-critical applications such as self-driving cars[2-3],drones,and robots[4-5].Deep neural network (DNN)[6]is a machine learning technique that tries to imitate the neurons of the human brain to transmit information and interpret data.

Unfortunately,the defense methods against poisoning attacks are not systematic enough.Zhang et al.[14]designed a strong defense scheme called DUTI to deal with the label-flipping attack.Given a small portion of the trusted items,the DUTI scheme could learn the difference between the distribution of trusted items and training samples to find the potentially corrupted labels and then get the corrupted labels to a domain expert for further examination.To defend against tool command language (TCL) attacks[12],Peri et al.[15]proposed a scheme named Deep-kNN to remove poisoning samples.In Ref.[15],the authors compared the class labels of each testing sample with itskneighbors.If most neighbor samples differed from the testing sample,this testing sample should be removed.Using the cooperative deep learning system,Shen et al.[16]proposed a defense method called AUROR to defend against poisoning attacks.The AUROR scheme could automatically identify and display the process of abnormal distribution features and then detect malicious users in the system according to the abnormal features.Based on a prior observation that poisoned samples may exploit spare capacity in the neural network,Liu et al.[17]proposed a fine-pruning technology as a defense method to enhance the security of deep neural networks by eliminating some dormant neurons to turn off poisoning behaviors.

Considering that the previous schemes[14-17]can only defend against specific attacks,Diakonikolas et al.[18]proposed a robust algorithm named Server that could defend against poisoning attacks in classification and regression models.Chen et al.[19]proposed a generic and attack-agnostic defense approach called De-Pois.The key idea of De-Pois was to train a mimic model to imitate the behavior of the target model with clean samples.

In this study,we proposed a poisoning attack-detecting scheme with data recovery (PAD-DR).First,we designed a data sampling audit algorithm and combined it with a real-time data detection method to detect all kinds of poisoning attacks.Second,we applied Cauchy Reed-Solomon (CRS) code technology to encode training data and store them on multiple servers to recover the corrupted training data.Finally,we formally proved the security goals of our PAD-DR scheme to withstand poisoning attacks and to recover the data accurately.

1 Preliminaries

1.1 Bilinear mapping

Definition1LetG1andG2be multiplicative cyclic groups of a large prime orderp; a pairing is a bilinear mape:G1×G1→G2.It satisfies the following properties:

1) Bilinear.e(ua,vb)=e(u,v)ab,?u,v∈G1;a,b∈Zp.

2) Non-degeneracy.?u,v∈G1,thuse(u,v)≠1∈G1.

3) Computability.?u,v∈G1; there is a polynomial time algorithm to calculatee(u,v).

4) Safety.It is difficult to calculate the discrete logarithm problem inG1andG2.

1.2 Discrete logarithm problem (DLP)

1.3 Computational Diffie-Hellman problem

1.4 Co-computational bilinear Diffie-Hellman problem (CO-CDH)

2 Proposed PAD-DR Scheme

2.1 System model

In our PAD-DR scheme,as shown in Fig.1,the system consists of five types of entities: system administrator (SA),third party auditor (TPA),coded data stores (CDSi),training data store (TDS),and nodes of the neural network for input layer (NNs).SA is responsible for processing the original training data.TPA is responsible for generating challenges to detect whether poisoning attacks are launched.

Fig.1 System model for Our PAD-DR scheme

CDSiare cloud spaces for storing regenerating backup data,where the regenerating backup data are distributed onnmultiple CDSi.TDS stores the original training data with tags and sends them to the NNs.NNs receive the training data and feed them to the neural network.Besides,NNs are responsible for detecting the poisoning attacks in real-time with the data tags.

2.2 Threat model

Our PAD-DR scheme defines the threat model in terms of the SA,TPA CDSi,TDS,NNs,and network attackers (NAs).

The SA is credible.The SA encodes the training data,generates tags for the original data,and collects regenerating data blocks to help recover the poisoned data.

The TPA is semitrusted.The TPA may run the algorithm and protocol in the system.However,the TPA is curious about the contents of training data and tries to obtain the contents of training data in the verification process.

The TDS is semitrusted.The TDS may faithfully run the algorithm and protocol in the system.However,to maintain the reputation,they may conceal the truth when poisoning samples attack the training data.At that time,TDS may attempt to forge proofs to cheat TPA for passing the auditing process.CDSiare credible and may store encoded regenerating data for backup.NNs are credible and may run the algorithm and protocol in the system and transmit the training data to the nodes of the next layer.

NAs are malicious.NAs attempt to launch poisoning sample attacks.Furthermore,to pass the audit by TPA and NNs,NAs may attempt to launch tempering and forgery attacks by forging data and signature proofs.

3 Construction of PAD-DR Scheme

We proposed the detailed construction of our PAD-DR scheme.The scheme includes three phases: the setup,detection and recovery phases.Details of each phase are as follows.

3.1 Setup phase

The setup phase includes setup,encoding and SigGen algorithms.Among them,the setup algorithm generates system parameters,the encoding algorithm applies CRS code technology to encode the original training data for backup,and the SigGen algorithm generates tags for the original training data.Three algorithms are described in detail as follows.

3.1.1 Setup algorithm

3.1.2 Encoding algorithm

Supposem,k,w∈Z+,a Galois field GF(2w),for data fileF={m1,m2,…,mk},SA utilizes CRS[22]technology to generaten=m+kencoded data blocks by constructing a code matrixC=ΨM,where the encoding matrixΨisn×k,and the message matrixMisk×1.

Then,the encoding matrixΨis designed as follows:

Hence,thei-th row ofΨis defined as the encoding vectorΨi(i∈{1,2,…,n}).

For example,letk=6,m=4,andn=10,the Cauchy matrixGis with dimensions 4×6 on GF(28).LetX={0,1,2,3,4,5},Y={6,7,8,9},the encoding process is shown as follows:

C=Encode(M)=ΨM=

3.1.3 SigGen

3.2 Detection phase

The detection phase consists of TPA detection and node detection algorithms.With our proposed sampling audit algorithm,the TPA detection algorithm can detect poisoning attacks on the training data stored in TDS.In real-time,a node detection algorithm can detect poisoning attacks on the training data at the input layer for NNs.

3.2.1 TPA detection

Having received the response proof from the TDS,the TPA verifies the proof as follows.TPA checks the verification equation as

(1)

If Eq.(1) holds,the training data stored on TDS are securely protected.Otherwise,the training data stored on TDS should be changed or corrupted by the poisoning attacks.When the training data is detected to be poisoned,the TPA immediately makes an alarm for the poisoning attack and sends feedback {error} to the SA.Then,the SA performs the recovery operation to recover the poisoned data on TDS.

3.2.2 Node detection

After receiving training data with signatures from TDS,NNs run a node detection algorithm to detect poisoning attacks as follows.First,NNs verify the correctness of the signatureStsk(F‖μ′‖σ′) by TDS’s public key according to the RSA signature algorithm.If the verification fails,the NNs abort the message and send {error} to SA.Otherwise,we have to design the real-time data detection method for the NNs to check the verification equation as follows:

(2)

If Eq.(2) holds,the training data sent by TDS are complete and correct.Otherwise,the poisoning attacks could change or corrupt the training data.NNs may send error feedback to inform SA that the training data could be attacked and request SA to recover the training data.

3.3 Recovery phase

In the recovery phase,there is a data recovery algorithm.Once SA receives error feedback from TPA or NNs,implying that the training data has been poisoned and attacked,SA may execute the data recovery algorithm to recover the original training data.

With the data recovery algorithm,the original data fileF={m1,m2,…,mk} can be recovered by collecting anyknodes of CDSifor itsciandΨi.Thus,the original data fileFcan be recovered through linear operations on the entries of anykrows of the code matrixC.

Next,based on the definition in the preliminaries,we provided a theorem indicating that the entire process of the PAD-DR scheme is correct.

Theorem1Our PAD-DR scheme can correctly run to detect the poisoning attack and recover the original training data accurately.

4 Security Analysis

In this section,we formally proved that our PAD-DR scheme can resist tampering and forgery attacks.Also,we proved that the poisoned data can be correctly recovered.

4.1 Related theorems

Some theorems formally showed that our PAD-DR scheme can resist tampering and forgery attacks.Furthermore,we showed that the poisoned data can be correctly recovered.

Theorem2In our PAD-DR scheme,it is computationally infeasible for the TDS and NAs to forge proof for passing the TPA’s auditing.Suppose there is a (t,?)-algorithm to forge a proof.Then,if a (t′,?′) adversary can solve the Co-CDH problem witht′≤t+qcG1and ?′=?/e(1+qs),whereqcG1denotes one exponentiation time inG1,andqsdenotes the number of requests.

Theorem3In our PAD-DR scheme,NNs can detect poisoned samples in real-time training data received from TDS.

Theorem4In our PAD-DR scheme,the original data fileFcan be recovered by collectingciandΨifrom anyknodes of CDSi.

4.2 Security goals comparison

In this section,the security goals of our PAD-DR scheme are compared with that of DUTI[14],Deep-kNN[15],Server[18]and De-Pois[19].The comparison includes specific poisoning attack detection,arbitrary poisoning attack detection,and poisoned data recovery.In Tab.1,“√” depicts that the security problem has been solved,“×” indicates that the problem has not been solved.

Tab.1 Security goals comparison

According to Tab.1,our PAD-DR scheme can realize all the security goals mentioned above,while other schemes cannot.

5 Experiment and Performance Analysis

This section begins by describing the experimental setup and the accuracy under different attacks.Then,we analyzed the communication overhead and computation costs in detecting poisoning attacks of training data and evaluated the performance of our PAD-DR scheme.

5.1 Experiment setup

5.1.1 Datasets

The training data used in our scheme are from MNIST,CIFAR-10,and house pricing.MNIST is a database of handwritten digits containing 60 000 training sample sets and 10 000 test sample sets stored in binary form.The width and height of each sample image are 28×28.The CIFAR-10 database contains 32×32 color images in 10 different classes,with 50 000 training and 10 000 testing images.The 10 different classes include cars,dogs,frogs,airplanes,cars,ships,horses,birds,and trucks.The house pricing dataset utilizes predictor variables such as the number of bedrooms and lot square footage.It contains 1 460 houses and 81 features.In the experiment,we randomly split this dataset into training and testing datasets with 70% and 30% of the data,respectively.

5.1.2 Experimental environment and parameter setting

We evaluated the process of detecting poisoning samples on NVIDIA 2080Ti GPU and applied Ali Cloud to store training data.The operating system is Windows 10.The RAM size is 8 GB.Matrix multiplication algorithms are implemented with the open-source library Jerasure Version (1.2).Auditing algorithms are implemented onCwith a pairing-based cryptography library.The curve utilized in the experiment is an MNT curve.We set the length ofG1to 175.In our experiment,we adopted a common neural network that applies three full connection layers with ReLU activation and one full connection layer with sigmoid activation,and we applied the cross-entropy loss function to calculate its loss.

5.1.3 Generation of poisoning training data

To verify the performance of our PAD-DR scheme,we randomly extracted 10%,15%,20%,25%,and 30% of the training data to generate poisoning samples.To compare the detection rate of poisoned samples for our PAD-DR scheme,TCL[12],LF[8],and R[13]attacks are used to generate poisoning data.For MNIST,the number of iterations is 400,the learning rate is set to 0.1,and the initial value is set to 0.01.For CIFAR-10,the number of iterations is 6 000,the learning rate is 0.01,and the initial value is 0.01.

5.2 Detection rate for poisoning attacks

In this section,we evaluated the detection rate of poisoned samples for our PAD-DR scheme compared with that of TRIM[13],DUTI[14],Deep-kNN[15],Server[18],and De-Pois[19].The results are presented in Figs.2,3,and 4,respectively.

As shown in Fig.2,for TCL attacks,the detection rate of our PAD-DR scheme is always 100%,which is higher than that of Refs.[15,19].There are two reasons for the relatively low detection rate of Ref.[15].One is that the authors only compared the samples with the surroundingksamples,which was not sufficient or accurate.The other reason is that the authors should set an artificial setting threshold to distinguish poisoned samples from clean samples.The main reason for the relatively low detection rate of Ref.[19]is that the authors adopted conditional GAN technology to expand a small part of a trusted dataset as the whole training data,which could not fully reflect the features of the real training data.In our PAD-DR scheme,we could only detect any changed or poisoned samples in the training data by verifying Eqs.(1) and (2).Hence,the detection rate of our PAD-DR scheme is always 100%.

Fig.2 Detection rate under TCL attacks

As illustrated in Fig.3,for TF attacks,the detection rate of our PAD-DR scheme is always 100%,which is higher than that of Refs.[14,18-19].In Ref.[14],the main reason for the relatively low detection rate is that the authors required a small piece of completely reliable data,which could not be ensured in the system to detect the TF attack.In Ref.[18],the detection rate decreased rapidly with the increase of the poisoning rate.The main reason to detect the TF attack is that the authors should set an artificial setting threshold,which could not accurately distinguish poisoned samples from clean samples.In Ref.[19],the main reason for the relatively low detection rate is that the authors should train the mimic model,which was ineffective in obtaining accurate prediction results.

Fig.3 Detection rate under LF attacks

As depicted in Fig.4,for R attacks,the detection rate of our PAD-DR scheme is always 100%,which is higher than that of Refs.[13-14,19].With the increase of the poisoning rate,the detection rate of Refs.[13-14,19]decreased rapidly.In Ref.[13],to detect R attacks,the authors ignored the influence of poisoned samples in the lowest residual set with iteration,which may cause the failure of R-attack detection.In Ref.[14],the authors required a small piece of completely reliable data to detect an R attack,which could not be ensured in the system.In Ref.[19],the authors should train the mimic model to detect R attacks,which was not effective enough to obtain accurate prediction results.In our PAD-DR scheme,we could only detect any changed or poisoned samples in the training data using verifying Eqs.(1) and (2).Hence,the detection rate of our PAD-DR scheme for R attacks is always 100%.

Fig.4 Detection rate under R attacks

5.3 Computation cost

We evaluated the computation cost for our PAD-DR scheme,whereLdenotes the length of each encoded data block,EGdenotes one exponentiation inG1andG2,EZdenotes one exponentiation inZp,MGindicates a multiplication operation on groupsG1andG2,MZindicates a multiplication operation on the number fieldZp,MFrepresents a matrix multiplication in a finite fieldF(2n),MIrepresents the cost of computing the inverse of the matrixM,AZindicates an addition operation on the number fieldZp,Pedenotes the computation cost of one pairing operatione,andHdenotes the computation cost of an operation of calculating the hash value for a number.Furthermore,|I|represents the number of challenged data blocks.All the statistical results are the averages of 20 trials.

Tab.2 depicts the computation cost of our PAD-DR scheme in the setup,detection,and recovery phases.

Tab.2 Computation cost of our PAD-DR scheme

In the detection phase,the computation cost refers to two parts: TPA detection and node detection.The computation cost of TPA detection is|I|(MZ+H+EG)+(|I|-1)(AZ+MG)+Pe,where|I|(MZ+H)+(|I|-1)AZand|I|EG+(|I|-1)MGare the computation cost of TDS to generate a linear combination of data blocks and an aggregated tag,respectively,wherePeis the computation cost of TPA to verify the proof.The computation cost of node detection is|I|(MZ+H+EG)+(|I|-1)(AZ+MG)+Pe,where|I|(MZ+H)+(|I|-1)AZand|I|EG+(|I|-1)MGare the computation cost of TDS to generate a linear combination of data blocks and an aggregated tag,respectively,Peis the computation cost of NNs to verify the proof.Hence,the computation cost in the detection phase is 2|I|(MZ+H+EG)+2(|I|-1)(AZ+MG)+2P.

5.4 Communication overhead

We assessed the communication overhead in the setup,detection,and recovery phases.In this section,|Zp|represents the size ofZp,|G|represents the size of groupG,and|GF|represents the size ofGF(2w).Tab.3 presents the communication overhead for the three phases.

Tab.3 Communication overhead for our PAD-DR scheme

In the setup phase,communication overhead mainly includes two parts.The one is SA uploads data and tags {F,Φ} to TDS.The other is SA sends the encoded data {ci}i=1,2,…,nto CDSi.Hence,the communication overhead in the setup phase is sizeof(F)+k|G|+n|GF|.

In the detection phase,the communication overhead mainly refers to two parts: TPA detection and node detection.The communication overhead for TPA detection includes two parts.One is that TPA sends a challenge chal={(i,vi)} to TDS.The other is that TDS replies as proofP={μ,σ}.Hence,the communication overhead for TPA detection is 2|I||Zp|and|G|+|Zp|.The communication overhead for node detection is sizeof(F)+|G|+(k+1)|Zp|which arises from TDS sending {F,μ′,σ′,Sssk(F‖μ′‖σ′),vi}i∈Ito NNs.Thus,the total communication overhead in the detection phase is sizeof(F)+2|G|+(k+2+2|I|)|Zp|.

In the recovery phase,the communication overhead is decided by the data size of {ci,Ψi},which arises from theknodes of CDSi.Hence,the communication overhead of our PAD-DR scheme is 2k|GF|.

6 Conclusions

1) An algorithm combining data sampling audit and real-time data detection is designed,and experiments show that the algorithm can accurately detect toxic data contained in the data.

2) A data recovery algorithm based on CRS encoding is designed,and experiments show that it can efficiently restore poisoned data to clean data.

3) The current algorithm cannot guarantee the security of parameters during transmission.We will conduct further research on improving the integrity and confidentiality of parameters in the future.