Precise Agriculture:Effective Deep Learning Strategies to Detect Pest Insects

Luca Butera,Alberto Ferrante,Mauro Jermini,Mauro Prevostini,and Cesare Alippi,

Abstract—Pest insect monitoring and control is crucial to ensure a safe and profitable crop growth in all plantation types,as well as guarantee food quality and limited use of pesticides.We aim at extending traditional monitoring by means of traps,by involving the general public in reporting the presence of insects by using smartphones.This includes the largely unexplored problem of detecting insects in images that are taken in noncontrolled conditions.Furthermore,pest insects are,in many cases,extremely similar to other species that are harmless.Therefore,computer vision algorithms must not be fooled by these similar insects,not to raise unmotivated alarms.In this work,we study the capabilities of state-of-the-art (SoA) object detection models based on convolutional neural networks (CNN)for the task of detecting beetle-like pest insects on nonhomogeneous images taken outdoors by different sources.Moreover,we focus on disambiguating a pest insect from similar harmless species.We consider not only detection performance of different models,but also required computational resources.This study aims at providing a baseline model for this kind of tasks.Our results show the suitability of current SoA models for this application,highlighting how FasterRCNN with a MobileNetV3 backbone is a particularly good starting point for accuracy and inference execution latency.This combination provided a mean average precision score of 92.66% that can be considered qualitatively at least as good as the score obtained by other authors that adopted more specific models.

I.INTRODUCTION

THIS work is in the field of precise agriculture and aims at exploring different deep learning (DL) models for detecting insects in images.More in detail,this paper discusses the use of DL-based object detection models for the recognition of pest insects outdoors.In particular,we focus on the comparison of known architectures in the context of a novel dataset.Considering the field of interest,we also evaluate computational resource requirements of the considered models.

Early pest insect identification is a crucial task to ensure healthy crop growth;this reduces the chance of yield loss and enables the adoption of precise treatments that provide an environmental and economic advantage [1].Currently,this task is mainly carried out by experts,who have to monitor field traps to assess the presence of species which cause significant economic loss.This procedure is prone to delays in the identification and requires a great deal of human effort.In fact,human operators need to travel to the different locations where traps are installed to check the presence of the target insects and,in some cases,to manually count the number of captured insects.The current trend is to introduce machine learning techniques to help decision makers in the choice of suitable control strategies for pest insects.Even though this approach is going to automate some time-consuming tasks,it leaves monitoring of pest insects limited to traps.For some pest insects that spread outside crop fields,it can be beneficial to have additional information coming from other locations.This is the case of Popillia japonica,a dangerous phytophage registered as a quarantine organism at European level [2].Early detection of the presence of this insect in specific regions is extremely important,as it permits to track its spread.In this exercise,the general public may be of great help,as they can provide valuable input by means of mobile phones.As shown in Fig.1,our monitoring system for pest insects will integrate inputs from traps and from smartphones,with the purpose of creating a map of the presence of the pest insects.Machine learning techniques become necessary when reports come from a large pool of users that take images with different cameras,lighting conditions,framing and backgrounds.Furthermore,pest insects are,in many cases,extremely similar to other species that are harmless (e.g.,Popillia japonica is very similar to two harmless species,Cetonia aurata and Phyllopertha horticola).Therefore,computer vision algorithms must be able to distinguish between them,even in presence of high similarities.This is especially important when the general public is involved in the process,to avoid non-motivated alarms and insect eradication actions that are useless and may damage the environment.

Throughout the years,many Computer Vision approaches have been proposed to assess the presence of pest insects in crop fields.Early solutions (e.g.,[3]) were based on image classification by means of handcrafted features.These approaches suffered from lack of generality,as most features were specific to particular species and,moreover,required great domain knowledge and design effort.

Fig.1.Sketch of a multimodal detection system.

With the advent of DL and convolutional neural networks(CNN),the problem has started being tackled with Data Driven approaches that leveraged the abstraction power of deep architectures both to extract meaningful features and perform the recognition task.The majority of these studies focused on Image Classification [4],[5] which assigns a class label to each image.CNNs solved this problem with the help of Transfer Learning from models pre-trained on generic datasets,such as ImageNet [6].

Only in recent years,as highlighted in [7],some studies have tackled the task of both recognizing and localizing pest insects in images.Remarkably,PestNet [8] compared an adhoc network to state-of-the-art (SoA) ones,for the task of detecting insects in images taken in highly controlled environments,such as in traps.

However,not much work has been done on detecting insects in the open,despite the fact that exploiting the pictures any citizen could take with a smartphone,can greatly improve the control of pests over the territory.In the context of a multimodal pest monitoring system like the one shown in Fig.1,it is important to have a reliable detection algorithm that is accurate with highly diverse backgrounds and robust to the presence of harmless insects that may be mistaken for dangerous ones.In fact,in such a system,observations would come from a large number of locations and at much higher rates if compared to trap-only monitoring systems.In a large scale system,false alarms become a critical problem that requires a specific study.

In this paper,we assess the capabilities of SoA Generic Object Detection models for the detection of very similar insects in non-controlled environments.We do so as we believe that current architectures are well suited for this task and there is no need for new ones;the main problem is understanding how they compare and which ones are better choices.

We built a dataset of images taken from the Internet,with high variability in their characteristics,such as background,perspective,and resolution.The dataset contains 3 classes:the first one contains images of a pest insect,Popillia japonica;the other two classes contain images of insects commonly mistaken for our target one.Special care was put in minimizing the presence of highly similar samples.This task poses many challenges that SoA methods may have trouble with,such as presence,in the images,of small or partially occluded insects,bad lighting conditions or insects disguised in the background.Most importantly,without any prior knowledge,the object detectors may not be able to learn the general features necessary to disambiguate two similar insects.

We then used our dataset to study the performance of three SoA object detection models across four different backbones.The goal was to:

1) Assess the adaptability of existing models to detection of Popillia japonica.

2) Identify the best candidate for task-specific improvements and modifications.

3) Study thecomputational cost/accuracytrade-off of different backbones.

With respect to existing literature,the novelty of our work can be summed up as:

1) We adopt advanced filtering techniques to provide enhanced datasets.The impact on performance is well evident both in terms of accuracy and reliability.Notably,we avoid near-duplicate samples that may hinder the model evaluation procedure correctness in case of similar couples being split in the training and test sets.

2) We experimentally verify that designing object detectors targeted to beetle-like insects is neither requested nor advisable,as general purpose models can be successfully used after proper tuning.Furthermore,we compare these general models both from the stand-point of performance and from that of computational resources by employing common metrics.

3) For the first time,we focus on the detection of extremely similar insects.In our application only one insect is actually harmful and must not be confused with others closely resembling it.

4) We consider images that are very different in their characteristics and,in general,not shot in controlled environments nor with the same equipment.This,together with the previous point,sets up a highly complex detection environment that has not been previously addressed in similar studies on insects.

5) We propose a study on the impact of output thresholding on false positive count,showing that it is possible to find a good trade-off between false alarms w.r.t.the detection of the dangerous insects also for our specific application scenario.

6) We show the applicability of Guided Backpropagationbased visualization techniques that can help experts gather insights on insect features considered by the models.This is of particular importance since it allows validation of the algorithm’s decisions from a more qualitative standpoint.These visualizations highlight the key features of the image used for the prediction:a good detector is expected to rely on visually meaningful parts of the image.

The results we obtained show that known SoA Object Detection models are suitable for beetle-like insect recognition in the open,even in presence of similar species.Additionally,backbone choice can affect performance,in particular for SSD models.Transfer Learning is a necessary step but pre-training on a generic insect dataset provided no improvements.In general,FasterRCNN with MobileNetV3 backbone provides the best trade-off between accuracy and inference speed.

The paper is organized as follows:In Section II,we go over the state of the art for both Generic Object Detection and Pest Insects Detection.In Section III,we describe our dataset,how we created it and how we ensured absence of duplicates/nearduplicates.In Section IV,we briefly introduce the different models that we have compared.In Section V,we explain how we designed our experiments.In Section VI,we present and comment on our experimental results.

II.STATE OF THE ART

In this section,we describe the state of the art for our application.First we spend a few words on Generic Object Detection,then we talk specifically about the work that has been done specifically on insect recognition.

A.Generic Object Detection

Generic Object Detection is the task of assigning,for each relevant object instance in an image,a Bounding Box and a Label.The state of the art about this topic grew considerably in recent years as shown in [9],which represent an up-to-date and complete survey about Generic Object Detection.

We can divide Generic Object Detection models based on CNNs into two broad categories:Two-stage and One-stage detectors.Two-stage detectors are generally slower but more accurate.They first compute regions of interest (RoI) for each image and then compute the actual predictions for each RoI.One-stage detectors,instead,are faster at the cost of some accuracy,as they compute the predictions directly.Both detector types are composed of two parts:a feature extraction section,called backbone,and a detection head.The backbone is generally an Image Classification model stripped of the classification layer.Common examples are VGG [10] and ResNet [11].

We chose to select one representative model for each category to compare the trade-offs of the two approaches.FasterRCNN [12],for Two-Stage Detectors,is a well known architecture that uses a region proposal network (RPN) to extract the RoIs.For One-Stage Detectors we picked SSD[13],as it has shown,in the available literature,the highest inference speed for its class,while maintaining a respectable accuracy.SSD uses a set of Default Boxes which span the whole image across different scales,instead of an RoI pooling layer.To the two aforementioned models we have added the third competitor:RetinaNet [14],which is sort of a mixedbreed.It is a One-Stage Detector but its speed and accuracy are similar to the ones of common Two-Stage Detectors.Its peculiar characteristic is that it neither uses an RoI pooling layer nor Default Boxes,but it exploits a Focal Loss function which weights the foreground samples as of higher importance with respect to the background samples.

B.Pest Insects Detection

For Pest Insects Detection the state of the art is quite narrow.The majority of the studies focus on Insect Classification,usually with an approach similar to the one introduced in [4],where known CNN-based classifiers are trained on multi-class insect datasets like IP102 [15].This dataset is one of the few openly available of its kind and it comprises over 75 000 images of insects from 102 different species;additionally,about 19 000 images of this dataset are annotated with bounding boxes.IP102,though,presents a very long tailed distribution,which is not ideal for learning.

Another popular branch,commonly associated with Pest Insect Detection is the recognition of Plant Diseases.For example,[16]–[18] namely relate to the identification of pest insects,but they actually detect their presence indirectly,by considering damages on plants.

For actual Pest Insect Detection,different studies,like [8],[19]–[23],use common DL architectures.In these examples,however,images are collected in controlled environments,by adopting standardized equipment and locations.In [21]images are taken in an open field on a wide,bright red,trap.In [8],[19],[20],closed traps with a white background are used,while [22] and [23] consider images taken in grain containers.The only study closely related to ours,both in terms of approach and final goal,is [24].In this study,images of 24 different species have been collected from the internet,and a network similar to FasterRCNN [12] with the VGG [10]backbone has been trained to perform detection.In comparison to the aforementioned studies our work has a major difference:our images are not taken in controlled environments nor shot with the same equipment.Differently from the other works,and in particular from [24],in our work we consider insect species that exhibit high similarity.Unlike other works we have extensively studied the capabilities of SoA models in order to find the best starting point for finer model design.Moreover,we show a duplicate safe data acquisition procedure,while similar studies have not given relevance to this aspect,which,in our opinion,is quite important.

III.DATASET

The images for our dataset have been gathered through the internet,by scraping the results from two popular Search Engines,Google and Bing,as well as from Flickr,a known photo sharing community.We have collected images for three classes of insects:Popillia japonica(PJ,Fig.2(a)),Cetonia aurata(CA,Fig.2(b)) andPhyllopertha horticola(PH,Fig.2(c)).The first one is a rapidly spreading and dangerous pest insect;the other two,even though similar to P.japonica,are harmless.P.horticola,in particular,for its morphological characteristics,is easily mistaken for P.japonica [25].In a real-world scenario,it is really important to detect those insects correctly to avoid false alarms.For each class of insects we have collected the first 4000 results from each website,corresponding to a total of 36 000 pictures.These samples were then subject to two phases of filtering,with the purpose of obtaining a high-quality dataset.The first phase involved some automatic methods,but also manual filtering;the second phase,instead,was fully automatic.

Fig.2.The species selected for the dataset.

A.Initial Filtering

In the initial filtering phase,we have applied 3 steps:

1) Duplicate removal based on file size and on the hash of the first 1024 bytes.

2) Unrelated removal based on a weak classifier that distinguishes images containing insects from images that do not.

3) Manual inspection.

After the filtering we had approximately 1200 images remaining for each class.

B.Removal of Near Duplicates

A problem that persisted through the first filtering phase was the presence of near duplicates:two or more non-identical but extremely similar images.These images,of which an example is shown in Fig.3,are particularly harmful in assessing performance of models.This problem is highlighted and tackled in [26] for the CIFAR Dataset [27].We have decided to use a similar approach,with a ResNet50 [11]Image Classification Model,pre-trained on ImageNet [6],instead of a model specifically trained on our dataset,as we found the results to be sufficiently good for our application.Once the classification layer is removed,the model,given an RGB image as input,outputs a 2048 dimensional vector,which can be interpreted as an embedding of the input image.These embeddings are known to be useful,in image retrieval applications,to assess image similarity by means of common distance metrics,as stated by [28].We have used theL2-distanceand an empirically chosen distance threshold of 90,below which we have considered two images to be near duplicates.In the end,we built our sanitized dataset incrementally,by adding iteratively all the images that were distant enough from the ones already present.We chose our threshold conservatively;thus,this procedure may have ruled out some “good” samples.Nonetheless,we preferred it to having near duplicates slip through.This step removed another 5% to 7% of the samples,depending on the class.Finally,we split the dataset into training and test sets,with the usual 80%–20% proportions.The actual number of samples for each class is shown in Table I.Splits are balanced with respect to the set of insect classes that appear in the image:this approach produces subsets with reasonably similar distributions.

Fig.3.An example of near duplicates.

TABLE IDETECTION DATASET SAMPLE COUNTS NUMBERS REFER TO TOTAL OBjECTS

IV.MODELS

In this section,we provide a brief explanation of the main characteristics of the considered architectures.Since all detection models work on high-level features extracted by the backbone,we first focus on describing the actual detection networks.Backbones are described later in this section.

A.Detection Models

A Detection Model tries to assign a Bounding Box and a Class Label to each relevant object in an input image.The former is generally represented by a four-value vector;the latter is a scalar that encodes the actual class name.The Object Detectors we consider are based on CNN [29] which are implemented as a computational graph that comprises layers of 2D convolutions,nonlinear activations and maxpooling operations.The weights of the convolution filters act as parameters of the network.These models are generally composed of two conceptual sub-networks:the backbone and the detector.The backbone is generally derived from SoA CNN Image Classifiers and outputs a set of high level features extracted from the input image.The detector takes these features as input and,through a series of convolution and linear operations,outputs the desired predictions.Each CNN is a parametric function,whose parameters can be optimized iteratively by means of common gradient descent algorithms,such as Stochastic Gradient Descent and Adam.In general the loss function compares the predicted output with the expected one and yields lower values when predictions are close to ground truths.Since these networks work with images,they must be robust to changes in the input that are not semantically meaningful.This is partially achieved during training with augmentation techniques that enhance data variability and improve shift,scale,and rotation invariance of the learned model.Scale invariance,usually,is also boosted by specific design choices that allow features to be implicitly computed at different scales.

We have considered the following models:FasterRCNN,SSD,and RetinaNet.We have chosen these models as they represent the state of the art when it comes to general performance and reliability in real world scenarios.

1) FasterRCNN:FasterRCNN [12] is a well known Two-Stage detector.In the first stage,the region proposal network(RPN),shown in Fig.4,extracts proposals from high level feature maps with respect to a set of predefined Anchor Boxes.These proposals take the form of:

i) an objectness score:The probability that an anchor in a specific location contains an object.

ii) a box regression:A 4-value vector that represents the displacement of the anchor that best matches the object position.

Fig.4.FasterRCNN’s RPN architecture from [12].

Features are then extracted from the best proposals,by means of an RoI Pooling layer.These proposed-specific features are then passed to a regression head and a classification head;the former computes the final displacements to best fit the proposed box onto the object;the latter assigns the predicted class label.This model uses a loss function with two terms:one accounting for the classification error and one for the box regression one.The former takes the shape of a log loss over a binary classification problem,the latter is the smooth L1 loss for the box displacement regression.

2) SSD:SSD [13] belongs to the family of One-Stage detectors.To improve inference speed,even though sacrificing accuracy,features are directly computed on top of the backbone,without the aid of RoIs.Each layer extracts features at a different scale,and uses them to calculate predictions;this helps achieving scale invariance.Predictions have the same form of FasterRCNN’s ones.Box Regression values are computed with respect to a set of Default Boxes,which are similar to the Anchor Boxes used in FasterRCNN.This model uses a loss similar to that of FasterRCNN,with a softmax log loss accounting for the classification error and a smooth L1loss taking care of the box regression one.

3) RetinaNet:RetinaNet’s [14] architecture is straightforward:features coming from the backbone are fed into a regression head and a classification head,which predicts the bounding box regressions with respect to a set of anchors and the class label.

The uniqueness of this network is not in the architecture,but in the loss function that it uses,calledFocal Loss.This loss assigns higher importance to hard-to-classify samples and,in turn,it reduces the problem of the gradient being overwhelmed by the vast majority of easily classifiable samples belonging to the background class.

B.Backbones

The backbone is the section of a detection model that performs feature extraction.Usually,a backbone is composed of an Image Classification network stripped of its classification head.Here we briefly introduce the backbones considered in our study that are:VGG,ResNet,DenseNet,and MobileNet.As previously stated for the detectors,these models have been chosen for their general reliability in real world scenarios.

1) VGG:VGG [10] is one of the first very deep Image Classification networks.Its design is rather surpassed by fully convolutional approaches,but it still represents a relevant baseline.Its architecture borrows from AlexNet [30],with Convolutional Layers followed by Max Pooling and activation function.The final fully connected layers are stripped,since the model needs only to perform feature extraction.

2) ResNet:ResNet [11] is considered to be the current gold standard both as image classifier and as backbone.ResNet first introduced the concept of Residual Blocks,shown in Fig.5.This is a classic convolutional block with a skip connection that allows the gradient to flow backwards freely and minimizes the problem of Vanishing Gradient.Letxbe the block input,andy=F(x) its output;then the residual block with skip connection is defined asy=x+F(x);this corresponds to the computational flow in Fig.5.

3) DenseNet:DenseNet [31] is considered a good Image Classifier,but it is not popular as a backbone.DenseNet takes the idea of skip connections a step further,with the DenseBlock:a series of feature extraction blocks,each one connected to the following ones,allowing both backward gradient flow and forward information flow.This is particularly suitable when learning from small datasets.In comparison to ResNet,features from different depth levels are explicitly combined inside the Dense Block.

Fig.5.ResNet’s residual block from [11].

4) MobileNet:MobileNet is an effort to optimize CNNbased Image Classifiers to make them run on mobile devices while minimizing loss of accuracy.To achieve this,a number of strategies used to reduce the parameter count have been employed.The convolution operations have been decomposed into a Deptwise Separable Convolution,where the operation against a CxNxN filter gets split into a convolution against a 1xNxN followed by a convolution against a Cx1x1 one.On top of that,MobileNet uses an Inverted Residual Block that has the same purpose as the one shown in Fig.5,but it is more efficient from the standpoint of computational resources.

5) Feature Pyramid Network:A feature pyramid network(FPN) [32] is a building block designed to enhance the output of a feature extraction backbone with features that span the input across multiple scales.Features are computed across different scales by means of convolution operations,then they are merged,as shown in Fig.6,in order to obtain a set of rich,multi-scale,feature maps.

Fig.6.FPN architecture from [32].

The FPN is very important for Object Detection networks,as multi-scale features improve the performances when recognizing objects of different sizes in the same image.

V.ExPERIMENT DESIGN

In this section,we describe the experiments used to compare the different models and backbones discussed in Section IV.Additionally,we describe some experiments performed to assess the impact of pre-training on model performances.

A.Performance Evaluation of Models and Backbones

Each detection model has been combined with each of the four selected backbones to assess how this choice impacts accuracy and inference time.Backbones were all pre-trained on ImageNet and the experiments have been carried out on an Nvidia Titan V GPU.

Note that FasterRCNN and RetinaNet use FPNs,while SSD works directly on the backbone’s output constructing multiscale features.

Each model was trained on the same data;parameters were optimized with stochastic gradient descent (SGD) with Momentum,using Cosine Annealing with Linear Warmup as learning rate schedule policy.This policy,shown in Fig.7,stabilizes training in the initial epochs and allows for fine optimization in the last ones.

Fig.7.Example of Cosine Annealing with Linear Warmup learning rate scheduling.

Table II shows the different hyperparameters considered for each Backbone/Detector pair.The number of epochs has been selected so that each model was trained to convergence.SSD was trained with a larger batch size as its smaller memory footprint allowed to do so.

We employ McNemar’s test,as suggested in [33],to assess statistic difference among pairs of models performance.This test is applied to a 2×2contingency table,of the form shown in Table III,where,given a binary classification test,aanddcount the samples on which the two models agree,whilebandccount those on which they do not.In the Object Detection case,each model can predict a bounding box for a particular ground truth or not,hence:

●ais the number of ground truths predicted by both models.

●dis the number of ground truths missed by both models.

●bis the number of ground truths predicted by the first model but missed by the second one.

●cis the number of ground truths missed by the first model but predicted by the second one.

McNemar’s test statistic makes use of the formula shown in(1)

The null hypothesis is thatH0:pb=pc;the alternative is thatH1:pb≠pc,wherepsymboldenotes the theoretical probability of occurrences in the corresponding cell.In case of rejection of the null hypothesis it is possible to affirm that the two models are significantly different.

Applying McNemar’s test to an Object Detection algorithm is not straightforward,as each input image may contain multiple objects and each of these objects may or may not be recognized correctly.In our case we considered an object recognized if the model predicted a box with the followingcharacteristics:intersection over union (IoU) with the object’s ground truth box greater than 0.5;confidence score greater than 0.5;the correct label for the object.Each prediction can match only one object.IoU for two boxesAandB,as defined in (2),is the ratio between the area of the intersection of the two boxes and the area of their union.Thus,its value is 1 when the two boxes perfectly overlap but shrinks to 0 as the two boxes either move away from each other or one becomes contained by the other.

TABLE IIMODELS AND HYPERPARAMETERS USED FOR TRAINING

TABLE IIIExAMPLE OF CONTINGENCY TABLE

B.Pre-Training Impact

A secondary aspect we wanted to test is the impact of pretraining on performance for the following two different scenarios:

1) Starting from a model completely pre-trained on the common objects in context (COCO) [34] dataset.

2) Using a backbone pre-trained on a generic insect dataset instead of ImageNet.

We restricted this experiment toResNet50-FPNFasterRCNNas pre-trained COCO weights were already available.COCO is a widely known dataset,annotated for different computer vision tasks,one of which is Object Detection.To train the insect-specific backbone,we have used a custom dataset generated by combining the images from our 3-class dataset with a filtered version of IP102 [15],from which we removed insects with too low sample count,duplicates,and misplaced images.To this we have added some background images that contained only plants or flowers without any insect.All the steps to ensure dataset fairness,described in Section III,have been applied to this data as well.To account for the high unbalance of this dataset,we have used a Weighted Cross Entropy loss function,assigning higher importance to less frequent classes.For these tests we have used the same hyperparameters ofResNet101-FPNFasterRCNN,which are listed in Table II.

VI.RESULTS

In this section,we present the results of the experiments discussed in Section IV.Furthermore,we compare our results to the ones obtained in other studies available in the scientific literature.

A.Comparison Across Models

Table IV shows the performance of each tested modelbackbone combination.The capabilities of the model to correctly predict the bounding boxes and to assign the labels have been evaluated by means of mean average precision(mAP).In Object Detection,the calculation of this parameter,though,may introduce some ambiguity.Therefore,we have used theAPIoU=.50as specified in the COCO challenge [34].Another relevant aspect that we have evaluated is the computational requirements,as we target smart agriculture,where power IoT devices are deployed in the field and edge computation may be preferable to avoid frequent energy expensive data transfers over the network.To evaluate the computational requirements,we have measured the inference speed in Frames per Second,both on GPU (an Nvidia GeForce RTX 2 080) and on CPU (an Intel Xeon Silver 4 116).These measurements do not provide a direct evaluation of the performance on IoT nodes,but rather a relative ordering of the model-backbone combinations in terms of computational requirements.These tests have been performed averaging the inference time of 100 iterations without batching;the inference operation is comprehensive of any preprocessing needed by the model,which usually is normalization and resizing of the input images.The input images were randomly generated,with a size of 1280×720 pixels.

As shown in Table IV,FasterRCNN and RetinaNet perform better than SSD with small changes in mAP among different backbones.SSD struggles with less powerful backbones likeMobileNet and VGG16,but it obtains higher scores when paired with more demanding backbones.There is no clear winner in terms of raw mAP,as scores of the best performing model-backbone combinations are very similar.However,these architectures do not support interpretability of results as it is not possible to open the black box and draw cause-effect indications.This is particularly true when we keep the detector unchanged and we only switch the backbone.For these reasons,the optimal architecture must be found empirically through a trial and error approach.

TABLE IVPERFORMANCE OF THE COMPARED MODELS IN TERMS OF MEAN AVERAGE PRECISION ON THE TEST SET AND FRAMES PER SECOND AT INFERENCE TIME.FALSE POSITIVES COUNT FOR CLASS POPILLIA JAPONICA IS ALSO SHOWN

Regarding computation speed,SSD is the fastest on CPU;this does not come as a surprise,as this model is designed to be fast and it is especially suitable for applications where the available hardware has no GPU acceleration.On GPU,SSD is still fast,but,surprisingly,FasterRCNN is computationally lighter with a noticeably higher FPS than all the others.This follows from the MobileNetV3 model being specifically optimized to reduce execution time;this is proportionally more relevant for FasterRCNN than for other detectors.Whether this depends on how the computation is allocated on GPU or to some specific backbone-detector interaction is difficult to say and should be the object of a further investigation task,outside of the scope of this work.

Table IV also shows the number of false positives,produced on the test set,for the class P.Japonica.FasterRCNN is the model showing less false positives;in particular,when this model is associated with the MobileNetV3 backbone,the number of false positives is the lowest among all models.If we consider these numbers with respect to the overall number of non-PJ insects present in our dataset,which is 530,we see that the best configuration has a false positive ratio of 2.26%.Mind that this is not a formally precise false positive ratio as the concept of negative sample is ill defined for the problem of Object Detection.

Fig.8 shows the influence of a threshold on the confidence score of predictions on the false positives count and the mAP for each model.Every prediction below the threshold is automatically ignored.For the best performers,mAP is not greatly reduced,even at higher threshold values;while using small thresholds can still have great impact on the number of false positives for models that produce many of them.This type of plot is really helpful in reducing false alarms while preserving detection performance.

Fig.8.False Positives for PJ class and mAP variation for different prediction confidence thresholds.

Fig.9 shows the trend of the training loss for our three models;we can notice how it smoothly converges in all cases but FasterRCNN with MobileNetV3 backbone.The loss increase,however,seems to marginally affect the mAP score during validation,as shown in Fig.10(a).This tells us that a thoroughly calibrated training procedure may lead the loss of this model,which already is one of the best,to a plateau as well,possibly with better performance.

Fig.9.Train loss for the 3 detection architectures with different backbones.

Fig.10.Validation mAP for MobileNetV3 and DenseNet169.

Fig.10 shows validation mAP score for our models with the DenseNet169 and MobileNetV3 backbones.As shown in Fig.10(b),DenseNet169 provides a smooth increase in mAP for all architectures,with minimal variance among training epochs and detectors,whilst MobileNetV3 plot is noisier and the results among the detectors are very different.The equivalent plot for ResNet101 is closer to DenseNet169,while the one for VGG16 is closer to MobileNetV3;this suggests that deeper and more demanding backbones have more consistent behavior independently of the specific detection model,while less demanding ones have troubles with light detectors like SSD.

Fig.11 shows the McNemar’s Test p-values for each model pair.Considering the conventional significance level of α=0.05,we can see that the majority of the models are significantly different one another,meaning that,even with similar performances,they have different weaknesses and strengths.This makes the choice of the best model a nontrivial decision.Furthermore,for each detector aside from SSD,the mAP score remains similar across the different backbones,suggesting that backbone choice is not that critical.

Given the performance results together with McNemar’s test outcome we can confidently say that a FasterRCNN detector with a MobileNetV3 backbone is the best choice for our case study,as it yields top mAP coupled with the highest speed on GPU.Moreover,the false positives are the lowest among competitors,this result is particularly relevant in this application.

When computational power is a strong constraint and no hardware acceleration is available,SSD with the MobileNetV3 backbone should be preferred instead,as it is reasonably fast without a significant drop in mAP.We comment that,even though an SSD with VGG16 backbone is almost 3 times faster on CPU,the loss in detection accuracy does not nicely trade off with the improvement in latency.Yet,this combination can be used,accepting the reduced detection performance,in systems that are highly constrained in computational resources and/or energy.

B.Effects of Pre-Training

Table V shows the performance of FasterRCNN with the ResNet50 backbone in the 3 cases described in Section V-B.For this evaluation,we only consider Mean Average Precision since the architecture is constant.Thus,the FPS scores of the different solutions are constant too.

The highest mAP is reached when the model is fully pretrained on COCO;pre-training the backbone on the insect dataset presented in Section V-B results,instead,in the lowest score.This is consistent with the loss trends of Fig.12,withcocobeing the lowest curve andinsectsthe highest.Results suggest that using an ad-hoc dataset to pre-train the backbone harms the overall Transfer Learning process instead of improving it.Whether this is due to the relatively small size of the dataset used or inherent in the usage of less diverse data,needs to be investigated further.

Fig.11.McNemar’s Test on compared models.Each cell contains the p-value for the specific pair.Significance level is α=0.05.

TABLE VPERFORMANCE OF FASTERRCNN-RESNET50, WITH DIFFERENT PRE-TRAINING DATASETS, ExPRESSED IN TERMS OF MEAN AVERAGE PRECISION ON THE TEST SET

As shown in Fig.13,no statistical difference between the model pre-trained on COCO and the one pre-trained on ImageNet is present,suggesting that pre-training the detection head has no significant benefits,at least in this case.The model pre-trained on generic insect images,however,is significantly different from the others.This further supports the thesis that pre-training on specific data significantly affects the learning,possibly for the worst.Hence,sticking to common Transfer Learning procedures seems to be the safer choice.

C.Visualization of Importance

In applications where the solution is implemented by means of very deep models,it is also important to assess the prediction quality in a human-readable form.Gradientweighted class activation map (Guided GradCAM) [35] is an approach that combines the results from Guided Backpropagation [36] and GradCAM [35] to form a visually understandable image of the areas of major importance for the prediction.

Guided Backpropagation relies on the intuition that the gradient of the relevant output (e.g.,the class score) w.r.t.the input image is a good indicator of which areas of the image are more relevant for the prediction.It is calledguidedbecause only positive gradients,corresponding to positive activations,are back-propagated.Instead,in GradCAM,the activation of the deepest convolutional layer of a CNN,properly weighted by the gradient of the relevant output w.r.t.said layer’s parameters,is interpreted as a heatmap of the relevant regions of the input.Guided GradCAM is the product of these two pieces of information.

In the case of insect recognition,these visualizations are particularly useful if an expert wants to evaluate the model based on the relevance of the highlighted insect features.In Fig.14 we can see how the regions of the highest importance are located on the head of the Popillia japonica and around its white hairs,a distinctive feature of this species.

D.Comparison with Other Studies

Our results are qualitatively similar to those of other works in the scientific literature,such as [8] and [24].The former reached 75.46% average mAP with a peak of 90.48% on the best predicted class;the latter topped at 89.22% mAP.We have reached a maximum mAP of 93.3%.

However,we point out that,even though the two mentioned studies consider some of the models that we have also included in our work,this comparison can only be qualitative for the following reasons:

1) The considered datasets are different;in particular for [8],whose data come from traps.To be more precise,we believe that our dataset is inherently more challenging.For this reason,we can consider our results to be at least as good as the ones reported by the cited works.

2) The two mentioned studies have not disclosed their approach to mAP calculation.Therefore,the reported numbers may not have the same exact meaning as in our paper.

3) In [24],the adopted backbone and the procedure used to train SSD and FasterRCNN are not specified.

4) In [8],when ResNet101 and FasterRCNN are considered,reported results are significantly worse than ours,with a top mAP of 71.62%,as opposed to our 92.14%.This strengthens the belief that adopted procedures are inherently different.

VII.CONCLUSIONS AND FUTURE WORK

Fig.12.Train loss (left) and Validation mAP (right) for FasterRCNN ResNet50 with different pre-training.

Fig.13.McNemar’s Test on FasterRCNN ResNet50 models with different pre-training datasets.The first name is the dataset used to pre-train the whole model,the second is the one used for the backbone.Significance level is α=0.05.

In this paper,we have evaluated different combinations of models and backbones for detecting a pest insect in images that are not obtained in controlled environments.Our results demonstrate that,at least for insects similar to Popillia japonica,this task can be performed with high accuracy,even by using general-purpose models.Not only detection performances have been estimated,but also inference speed,which can provide information on which model is less computational resource hungry among the tested ones.The best detection performance has been reached by the combination RetinaNet-ResNet101 (mAP=93.3%),but,on average,FasterRCNN was the best performer.The model with the best throughput on GPU is FasterRCNN paired with a MobileNetV3 backbone (FPS=60.92).The best throughput on CPU was obtained by the combination SSD-VGG16(FPS=4.27).Given the statistical similarity between some of the models we think that the critical part is the choice of the overall detection architecture rather that the specific backbone.SSD is an exception to this,as the backbone choice demonstrated to play a significant role in the final results.

Fig.14.Example of visualizing image importance through Guided GradCAM for a FasterRCNN model with ResNet50 backbone.

Additionally,our experiments show that pre-training on ImageNet is a suitable Transfer Learning setup for the insect recognition tasks and pre-training on small task-related datasets seemingly has no benefits.Overall,we consider FasterRCNN with MobileNetV3 backbone a strong baseline for insect detection given both the good performance and high inference speed on CPU and GPU;moreover this model produced the lowest number of false positives for the pest insect class and this is of particular importance for this type of application.

We conclude that widely adopted generic object detection architectures are well suited for the recognition of beetle-like insects and,realistically,for insects in general.The real advance in general insect recognition would probably come from the construction of bigger datasets,with a big number of species and images,rather than from the search for particular architectures which may,in the end,be too much task specific.

Future work should investigate the development of optimized models that take the found optimum as a baseline and make task-specific improvements to the architecture.A few examples are:addressing hardware and embedded system resource constraints to port the solution on mobile devices,leveraging known characteristics of the target pest insect(inductive bias),working on methods aimed at improving detection of small insects as well as dealing with bad lighting conditions and harsh environments.In the spirit of deep learning we would also envisage generation of a huge image dataset,possibly containing different species.This might be achieved by considering collaborative methods where citizens contribute in taking pictures and delivering them to a cloud validating process before enriching the database.