張彥斐,劉茗洋,宮金良,蘭玉彬
·農(nóng)業(yè)信息與電氣技術(shù)·
基于兩級(jí)分割與區(qū)域標(biāo)記梯度Hough圓變換的蘋果識(shí)別
張彥斐1,劉茗洋1,宮金良2※,蘭玉彬1
(1. 山東理工大學(xué)農(nóng)業(yè)工程與食品科學(xué)學(xué)院,淄博 255049;2. 山東理工大學(xué)機(jī)械工程學(xué)院,淄博 255049)
自然環(huán)境下果實(shí)的準(zhǔn)確分割與快速識(shí)別是采摘機(jī)器人作業(yè)面臨的難題之一。針對(duì)自然環(huán)境中的成熟蘋果,該研究提出一種基于Otsu與分水嶺相結(jié)合的兩級(jí)分割算法與區(qū)域標(biāo)記梯度Hough圓變換的蘋果識(shí)別方法。首先,使用亮度自適應(yīng)校正算法對(duì)表面亮度分布不均的蘋果圖像進(jìn)行校正,增強(qiáng)圖像的細(xì)節(jié)信息。結(jié)合果實(shí)顏色特征,提取YCbCr顏色空間的Cr分量圖像作為預(yù)處理樣本。然后,采用改進(jìn)后的Otsu算法進(jìn)行初次分割,得到蘋果目標(biāo)的二值圖像,該算法通過引入形態(tài)學(xué)開-閉重建濾波去除大量背景噪聲,通過縮減灰度級(jí)遍歷范圍提高分割速率。采用基于距離變換的分水嶺算法進(jìn)行二次分割,分離粘連果實(shí)區(qū)域,提取目標(biāo)蘋果的外部輪廓。最后,在輪廓外設(shè)置最小外接矩形標(biāo)記有效區(qū)域,在標(biāo)記區(qū)域內(nèi)進(jìn)行梯度Hough圓變換實(shí)現(xiàn)蘋果目標(biāo)的自動(dòng)識(shí)別。對(duì)自然環(huán)境中采集的200幅蘋果圖像進(jìn)行測試,并與傳統(tǒng)梯度Hough圓變換方法進(jìn)行對(duì)比,該文方法在順、逆光下的識(shí)別準(zhǔn)確率為90.75%和89.79%,比傳統(tǒng)方法提高了15.03和16.41個(gè)百分點(diǎn),平均識(shí)別時(shí)間為0.665和0.693 s,比傳統(tǒng)方法縮短了0.664和0.643 s。所提的兩級(jí)分割算法不僅可以從復(fù)雜環(huán)境中準(zhǔn)確分割果實(shí)目標(biāo)區(qū)域,而且可以從粘連果實(shí)區(qū)域中提取單個(gè)果實(shí)邊界。利用區(qū)域標(biāo)記的梯度Hough圓變換方法能夠快速準(zhǔn)確地對(duì)果實(shí)進(jìn)行識(shí)別。研究結(jié)果能滿足蘋果采摘機(jī)器人對(duì)不同光照下目標(biāo)識(shí)別速度和精度的要求,可為蘋果等類球形果實(shí)的快速識(shí)別提供參考。
圖像處理;分水嶺;蘋果識(shí)別;YCbCr;Otsu;梯度Hough圓變換
果實(shí)采摘機(jī)器人的發(fā)展日臻完善[1-2],視覺識(shí)別系統(tǒng)作為采摘機(jī)器人的一部分,其目標(biāo)檢測的速度、準(zhǔn)確率以及對(duì)周邊環(huán)境的適應(yīng)能力對(duì)機(jī)器人的工作效率和工作時(shí)長有較大影響[3-4]。近年來,基于深度學(xué)習(xí)的卷積神經(jīng)網(wǎng)絡(luò)(Convolutional Neural Networks, CNN)在目標(biāo)檢測方面性能卓越[5],已經(jīng)被廣泛應(yīng)用于復(fù)雜自然環(huán)境下水果檢測[6-8]。卷積神經(jīng)網(wǎng)絡(luò)可以通過對(duì)各種特征信息進(jìn)行學(xué)習(xí),增強(qiáng)特征的泛化能力,但該方法需要足夠大的訓(xùn)練集進(jìn)行模型訓(xùn)練,耗時(shí)長[9]。從目標(biāo)檢測流程來看,基于深度學(xué)習(xí)的卷積神經(jīng)網(wǎng)絡(luò)模型可以分為兩大類:一類是以YOLO網(wǎng)絡(luò)[10-12]為代表的單階段檢測算法,另一類是以Faster R-CNN[13-14]、FCN[15]和Mask R-CNN[16-17]為代表的兩階段檢測算法。兩類模型均具有檢測速度快、識(shí)別精度更高等特點(diǎn),檢測時(shí)長在幾十到幾百毫秒之間,識(shí)別準(zhǔn)確率可達(dá)95%以上。卷積神經(jīng)網(wǎng)絡(luò)由傳統(tǒng)的圖像處理技術(shù)作為支撐,傳統(tǒng)圖像處理根據(jù)果實(shí)顏色特征與形狀特征進(jìn)行識(shí)別,也是農(nóng)業(yè)采摘機(jī)器人視覺系統(tǒng)常用的技術(shù)方法[18-21]。其中,利用Otsu動(dòng)態(tài)閾值分割算法與Hough變換形狀檢測算法相結(jié)合的方法在果實(shí)識(shí)別領(lǐng)域受到了國內(nèi)外眾多學(xué)者的重視。周文靜等[22]選擇歸一化的紅綠色差作為分割特征,分別采用KNN(K-Nearest Neighbor)和Otsu兩種方法對(duì)紅提葡萄果穗圖像進(jìn)行背景分割,通過圓形Hough變換準(zhǔn)確識(shí)別果穗圖像中的葡萄果粒。Lv等[23]對(duì)蘋果顏色特征進(jìn)行Otsu分割,采用邊緣檢測和改進(jìn)的隨機(jī)Hough變換對(duì)蘋果進(jìn)行識(shí)別。對(duì)于未遮擋狀態(tài)下的蘋果果實(shí)識(shí)別率為100%,枝葉遮擋狀態(tài)下的蘋果果實(shí)識(shí)別率為86%。Gongal等[24]提出一種基于Otsu閾值分割與圓形Hough變換相結(jié)合的蘋果目標(biāo)識(shí)別方法,識(shí)別準(zhǔn)確率達(dá)79.8%。苗中華等[25]利用RGB空間的顏色作為目標(biāo)識(shí)別的主要特征,提出一種改進(jìn)算子的Otsu算法結(jié)合-means標(biāo)記作用區(qū)域的方法,并通過邊緣檢測、分水嶺算法,準(zhǔn)確提取了番茄目標(biāo)的邊界。蔡健榮等[26]利用Ostu自適應(yīng)閾值算法對(duì)2R-G-B色差分量圖像分割后,提出一種基于自動(dòng)設(shè)置成熟柑橘半徑搜索范圍的圓形Hough變換果實(shí)識(shí)別方法,對(duì)于完全裸露的果實(shí),識(shí)別準(zhǔn)確率可達(dá)95%。崔永杰等[27]利用Otsu法對(duì)獼猴桃果實(shí)的紅綠色差分量進(jìn)行分割,通過Canny算子提取邊界,對(duì)邊界圖像進(jìn)行橢圓形Hough變換,逐個(gè)識(shí)別出目標(biāo)果實(shí)。
綜上,在合適的顏色空間下先利用Otsu算法先進(jìn)行分割再利用Hough變換進(jìn)行目標(biāo)識(shí)別的方法實(shí)用性強(qiáng)、應(yīng)用范圍廣,可作為本研究的重要依據(jù)。但是自然環(huán)境中的光照變化和目標(biāo)粘連等對(duì)圖像質(zhì)量及圖像處理帶來了諸多不利,直接利用Otsu分割得到的圖像普遍質(zhì)量不高,直接使用Hough變換識(shí)別目標(biāo),運(yùn)算量大、耗時(shí)長,不利于機(jī)器人實(shí)時(shí)采摘。為此,本研究首先對(duì)采集到的光照不均勻圖像進(jìn)行校正,恢復(fù)圖像的細(xì)節(jié)信息,然后提取YCbCr顏色空間中的Cr分量并運(yùn)用改進(jìn)的Otsu算法對(duì)蘋果目標(biāo)進(jìn)行初步分割,得到目標(biāo)果實(shí)區(qū)域,在此基礎(chǔ)上,通過距離變換的分水嶺算法進(jìn)行二次分割,得到粘連果實(shí)及單個(gè)果實(shí)的邊緣輪廓,最后在輪廓外設(shè)置最小外接矩形作為有效區(qū)域進(jìn)行梯度Hough圓變換識(shí)別蘋果目標(biāo),以實(shí)現(xiàn)復(fù)雜環(huán)境下蘋果目標(biāo)準(zhǔn)確有效識(shí)別,保證算法的簡單高效。
圖像采集地點(diǎn)位于山東省淄博市沂源縣山東理工大學(xué)與山東中以現(xiàn)代智慧農(nóng)業(yè)有限公司聯(lián)合創(chuàng)建的生態(tài)無人農(nóng)場智慧果園,果樹為標(biāo)準(zhǔn)化種植,果樹品種為紅富士。在蘋果成熟期(9月中旬,此時(shí)果實(shí)已著色良好,與其他生長植被及樹葉、樹干等存在明顯差異)使用信息采集機(jī)器人前置攝像機(jī)(NikonD90)分別在順光和逆光條件下距果樹600~800 mm進(jìn)行多角度拍攝,圖像采集卡(AC-VS009)將采集到的圖像實(shí)時(shí)傳遞到計(jì)算機(jī)(Dell 筆記本)內(nèi)存,圖像格式為JPG,分辨率為1 000×666(像素),筆記本操作系統(tǒng)為Microsoft Windows10,在python3.8.8開發(fā)環(huán)境下,使用Visual Studio Code1.52.1軟件進(jìn)行編譯分析。試驗(yàn)共采集200幅蘋果圖像,包括順光、逆光、重疊、粘連、遮擋等多種情況。圖像采集系統(tǒng)如圖1所示。
圖1 圖像采集系統(tǒng)組成
1.2.1 光照不均勻圖像自適應(yīng)校正
由于受到果實(shí)、枝葉之間相互遮擋以及光照變化,導(dǎo)致蘋果部分信息缺失(圖2a、2b),對(duì)于果實(shí)目標(biāo)的提取造成一定難度,因此對(duì)光照不均勻的圖像進(jìn)行校正,以改善圖像局部亮度。首先,將輸入的圖像從RGB顏色空間轉(zhuǎn)換到HSV顏色空間,保持圖像的色調(diào)(H)和飽和度(S)不變,然后使用多尺度高斯函數(shù)對(duì)亮度(V)進(jìn)行卷積得到光照分量,采用二維伽馬函數(shù)作為校正函數(shù),根據(jù)光照分量分布特性自適應(yīng)調(diào)整二維伽馬函數(shù)的參數(shù),最后從HSV顏色空間轉(zhuǎn)回至RGB空間[28]。校正后的圖像如圖2c、2d所示,該方法可有效改善圖像光照不均的問題,提高圖像質(zhì)量,調(diào)整圖像的色彩平衡,更好地突出圖像的細(xì)節(jié)信息,為后續(xù)蘋果目標(biāo)的提取、分割、擬合奠定良好基礎(chǔ)。
a.順光a. Nature lightb. 逆光b. Backlight c. 順光校正c. Nature light correctiond. 逆光校正d. Backlight correction
1.2.2 基于YCbCr空間Cr分量差異性的目標(biāo)提取
合適的顏色空間是實(shí)現(xiàn)目標(biāo)分割的基礎(chǔ)。RGB顏色空間中3個(gè)分量相關(guān)性較高,不利于蘋果目標(biāo)的提取,而YCbCr顏色空間中的3個(gè)分量(亮度分量Y、藍(lán)色色度分量Cb和紅色色度分量Cr)彼此獨(dú)立,且與RGB顏色空間存在較為簡單的線性變換關(guān)系,故選擇YCbCr顏色空間進(jìn)行目標(biāo)提取。
圖3 YCbCr顏色空間各分量圖
選擇一張?zhí)O果與背景區(qū)別明顯的圖像,如圖3a所示,將其轉(zhuǎn)換至YCbCr顏色空間并提取各分量圖,如圖3b~圖3c所示,由于成熟的蘋果顏色主要以紅色為主,所以Cr分量圖中果實(shí)與背景之間有較為明顯的灰度差異。在圖3a中標(biāo)記一條白色水平線貫穿蘋果,作基于該線段水平方向上3個(gè)顏色分量灰度值剖面線圖,如圖4所示,Cr分量的灰度值剖面線中,蘋果所在區(qū)域(500~680像素之間)的灰度值范圍在150~200,而背景所在區(qū)域的灰度值范圍在130~150,蘋果區(qū)域的灰度值遠(yuǎn)高于背景區(qū)域的灰度值,說明Cr分量的蘋果與背景之間的灰度差值可以作為后續(xù)蘋果目標(biāo)分割處理的重要依據(jù)。
圖4 YCbCr顏色空間各分量灰度值剖面線圖
為獲取蘋果目標(biāo)區(qū)域,本研究采用改進(jìn)Otsu分割算法對(duì)蘋果目標(biāo)進(jìn)行初步分割,并結(jié)合基于距離變換的分水嶺算法對(duì)存在粘連的蘋果區(qū)域進(jìn)行二次分割,得到蘋果目標(biāo)輪廓。
Otsu算法利用圖像的灰度直方圖,以目標(biāo)和背景的方差最大來動(dòng)態(tài)確定圖像的分割閾值[29],但Otsu算法在圖像分割中仍存在一些缺點(diǎn),比如:1)抗噪性低,導(dǎo)致分割后圖像中夾雜噪聲;2)在最佳閾值選取時(shí),算法遍歷灰度范圍廣,計(jì)算量大,分割時(shí)間長。本研究針對(duì)以上缺點(diǎn)進(jìn)行改進(jìn),通過引入形態(tài)學(xué)開-閉重建濾波器壓制背景噪聲,通過縮減閾值選取區(qū)間加快分割速率。
2.1.1 形態(tài)學(xué)開-閉重建濾波去噪
形態(tài)學(xué)開-閉重建濾波器是一種基于數(shù)學(xué)形態(tài)學(xué)的非線性濾波,能夠有效消除局部噪聲,同時(shí)很好地保護(hù)目標(biāo)的邊緣輪廓。首先用腐蝕及開重建對(duì)Cr分量灰度圖像進(jìn)行處理,根據(jù)蘋果形狀特征,選用圓形結(jié)構(gòu)對(duì)灰度圖像做腐蝕運(yùn)算,將得到的腐蝕圖像作為標(biāo)記圖像,原灰度圖像作為掩模圖像,做形態(tài)學(xué)開重建運(yùn)算,得到開重建運(yùn)算后的圖像I。開重建運(yùn)算后的圖像中還存在一些非規(guī)則區(qū)域的干擾,因此,接著采用膨脹及閉重建對(duì)其進(jìn)行處理,先對(duì)開重建后的圖像I采用圓形結(jié)構(gòu)做膨脹運(yùn)算得到圖像I,再對(duì)I求補(bǔ)后作為標(biāo)記圖像,開重建運(yùn)算后的圖像I求補(bǔ)后作為掩模圖像,作形態(tài)學(xué)閉重建運(yùn)算,得到閉重建運(yùn)算后的圖像I。將I求補(bǔ)后得到形態(tài)學(xué)重建濾波后的圖像。
圖5為形態(tài)學(xué)開-閉重建濾波去噪前后的圖像對(duì)比,由圖5可知,該濾波器有效消除了圖像背景中灰度變化較大的枝干區(qū)域,使背景區(qū)域的灰度更加均勻。
圖5 圖像形態(tài)學(xué)開-閉重建濾波去噪前后對(duì)比
2.1.2 縮小閾值選取范圍
Otsu法基本思想如下:記原始圖像的灰度級(jí)為,灰度級(jí)取值范圍為[01],圖像中所有像素點(diǎn)的個(gè)數(shù)為,n為灰度值為的像素點(diǎn)數(shù)目,灰度值為的像素點(diǎn)出現(xiàn)的概率()=n/,假設(shè)存在閾值將圖像所有像素分為:背景區(qū)域0(0≤≤)和目標(biāo)區(qū)域1(<≤?1),則:
背景區(qū)域像素點(diǎn)占整圖比例0()為
目標(biāo)區(qū)域像素點(diǎn)占整圖比例1()為
背景區(qū)域灰度均值0(t)為
目標(biāo)區(qū)域灰度均值1()為
圖像全局灰度均值為
Otsu法最大類間判斷準(zhǔn)則下的最佳閾值為:
此種計(jì)算方式遍歷范圍廣,帶來了一定冗余計(jì)算,為降低運(yùn)算量,減少算法執(zhí)行時(shí)間。本文提出:當(dāng)蘋果目標(biāo)占圖像像素點(diǎn)比例≥1/2時(shí),其分割閾值位于[,]之間,當(dāng)蘋果占圖像像素點(diǎn)比例<1/2時(shí),其分割閾值位于[,]之間,其中為圖像全局灰度均值,為最小灰度值,為最大灰度值。證明如下:
定義全局灰度均值到背景灰度均值0和目標(biāo)灰度均值1的距離分別為0和1,則
對(duì)圖像的灰度級(jí)分布進(jìn)行統(tǒng)計(jì),得到灰度直方圖,如圖6所示。
由圖4中Cr分量的灰度值剖面線圖可知:蘋果的灰度值較高,背景的灰度值較低,從而可以確定圖6a和6b中的左側(cè)灰度值較低、面積較大的峰為背景,右側(cè)灰度值較高、面積較小的零散峰為蘋果,由此判斷蘋果目標(biāo)占圖像像素點(diǎn)比例1()<1/2,背景部分占圖像像素點(diǎn)比例0()>1/2,即20()?1>0,又因背景灰度值較低,所以背景灰度均值小于全局灰度均值,即0()<,將式(3)、(4)、(5)代入,整理可得0()(+)?>0。綜上所述,1?0>0,即1>0。故圖像全局灰度均值到目標(biāo)類灰度均值的距離大于到背景類灰度均值的距離,進(jìn)而可知均值更靠近背景峰,由于直方圖大致呈雙峰,最佳閾值位于谷底附近,則最佳閾值范圍在均值與最大值之間,即[,]之間。同理,若目標(biāo)占整幅圖的比例≥1/2,則最佳閾值位于[,]之間。
2.1.3 改進(jìn)Otsu法的實(shí)現(xiàn)
根據(jù)上述分析,改進(jìn)Otsu分割算法流程如圖7所示,基本步驟如下:
1)引入形態(tài)學(xué)開-閉重建濾波去除圖像中的噪聲。
2)統(tǒng)計(jì)圖像的灰度直方圖,得到最小灰度值、最大灰度值和全局灰度均值。
3)設(shè)定灰度閾值為135(135為試驗(yàn)測試所得),將大于該值的灰度值像素點(diǎn)初步判斷蘋果,統(tǒng)計(jì)蘋果像素點(diǎn)占整幅圖像的比例。
4)若蘋果目標(biāo)像素點(diǎn)占整圖比例≥1/2,則令的初始值為,根據(jù)最佳閾值選取公式(6),遍歷灰度級(jí),遍歷范圍為[,],得到關(guān)于的函數(shù);若<1/2,令的初始值為,根據(jù)公式(6),遍歷灰度級(jí),遍歷范圍為[,],得到關(guān)于的函數(shù)。
5)使函數(shù)取得最大值時(shí)的值即為最佳閾值,根據(jù)最佳閾值進(jìn)行蘋果目標(biāo)分割。
圖7 改進(jìn)Otsu算法流程圖
圖8為改進(jìn)前后Otsu算法的分割效果對(duì)比,對(duì)比可知,改進(jìn)后的Otsu算法能夠填充蘋果內(nèi)部產(chǎn)生的細(xì)小孔洞,抑制背景枝葉中的噪聲。
圖8 改進(jìn)前后Otsu分割效果對(duì)比
分割出的二值圖中蘋果外部輪廓會(huì)有一些毛刺凸起,受枝條遮擋嚴(yán)重的蘋果內(nèi)部會(huì)產(chǎn)生明顯的裂痕。采用腐蝕與膨脹算法對(duì)圖像進(jìn)行形態(tài)學(xué)處理,平滑目標(biāo)輪廓,填補(bǔ)蘋果內(nèi)部的缺口。結(jié)果如圖9a和9b所示。
對(duì)于相互粘連的蘋果,其內(nèi)部輪廓仍無法獲取。本文采用基于距離變換的分水嶺算法[30]對(duì)粘連蘋果進(jìn)行分離。距離變換計(jì)算過程如下:
圖9 形態(tài)學(xué)處理結(jié)果
根據(jù)距離變換公式得到距離圖,圖中每個(gè)非零點(diǎn)距離該點(diǎn)最近的零點(diǎn)的距離越遠(yuǎn),則在圖中顯示越亮,將這些高亮的點(diǎn)或點(diǎn)集作為蘋果的內(nèi)部標(biāo)記。通過膨脹二值圖將蘋果邊界增加到背景,確保背景中的任何區(qū)域都是背景。從背景區(qū)域中減去內(nèi)部標(biāo)記區(qū)域獲得未知區(qū)域,即為蘋果的邊界存在的區(qū)域,采用分水嶺算法對(duì)其分割,完成粘連蘋果目標(biāo)的分離。分水嶺算法分割過程及效果如圖10所示。
利用梯度Hough圓變換可以準(zhǔn)確識(shí)別出圖中的蘋果數(shù)目,但是該算法在整幅圖像上進(jìn)行遍歷,運(yùn)算量大,耗時(shí)長,且易產(chǎn)生誤識(shí)別,為此本文對(duì)算法進(jìn)行優(yōu)化,通過在輪廓外設(shè)置最小外接矩形作為有效區(qū)域,并在有效區(qū)域內(nèi)進(jìn)行梯度Hough圓變換,以提高識(shí)別速率與準(zhǔn)確率,優(yōu)化后算法的實(shí)現(xiàn)步驟如下:
1)使用Sobel算子計(jì)算輪廓的梯度方向,沿著指向圓心的梯度方向畫線段。
3)計(jì)算所有輪廓上的點(diǎn)到圓心的距離,即為可能的半徑值。
6)對(duì)于符合條件的圓心與半徑用加權(quán)平均法進(jìn)行合并,得到唯一確定的蘋果圓形。
圖11為優(yōu)化前后梯度Hough圓變換的識(shí)別結(jié)果,對(duì)比可知,優(yōu)化前易識(shí)別出錯(cuò)誤的圓,優(yōu)化后能夠減少誤識(shí)別。在順光和逆光圖像中分別識(shí)別出20個(gè)蘋果,其形心及半徑信息見表1。
圖11 優(yōu)化前后梯度Hough圓變換識(shí)別結(jié)果
表1 蘋果識(shí)別結(jié)果
為驗(yàn)證本文改進(jìn)后Otsu算法的分割性能,利用改進(jìn)前Otsu算法與-means算法作對(duì)比試驗(yàn),選擇4幅具有代表性的蘋果圖像(包括順光、逆光、果實(shí)重疊、遮擋等情況)進(jìn)行目標(biāo)分割,結(jié)果如圖12所示,分割時(shí)間見表2。
由圖12和表2可以看出:在分割效果上,改進(jìn)前Otsu算法與-means算法分割出的圖像中夾雜較多的背景噪聲,對(duì)蘋果目標(biāo)的提取造成一定干擾,而本文改進(jìn)后的Otsu算法有效剔除了大量背景噪聲,并對(duì)蘋果目標(biāo)原有的形狀進(jìn)行了保護(hù)。在分割時(shí)間上,3種方法的平均分割時(shí)間分別為2.101、2.518和1.458 s,相比于改進(jìn)前Otsu算法和means算法,本文算法分別縮短了0.643和1.060 s。
圖12 三種分割方法效果對(duì)比
表2 三種分割方法的耗時(shí)對(duì)比
為分離蘋果目標(biāo)相互粘連的區(qū)域,采用基于距離變換的分水嶺算法進(jìn)行二次分割,得到單個(gè)蘋果目標(biāo)的完整輪廓邊界。結(jié)果如圖13所示。圖中部分粘連的蘋果之間已經(jīng)建立起精確連續(xù)的分割線,形成獨(dú)立的閉合區(qū)域。
圖13 分水嶺算法提取蘋果目標(biāo)輪廓
為驗(yàn)證本文識(shí)別方法的有效性,利用傳統(tǒng)的梯度Hough圓變換、文獻(xiàn)[21]Hough圓變換作對(duì)比試驗(yàn),識(shí)別效果如圖14所示,識(shí)別準(zhǔn)確率與識(shí)別時(shí)間結(jié)果見表3、表4。
由圖14可知:傳統(tǒng)梯度Hough圓變換方法由于在整幅圖上進(jìn)行遍歷,存在一個(gè)蘋果果實(shí)被識(shí)別為多個(gè)圓形的誤識(shí)別現(xiàn)象。文獻(xiàn)[21]中的Hough圓變換方法對(duì)于重疊及枝葉遮擋較少的果實(shí)可以準(zhǔn)確識(shí)別,但是多果重疊、果實(shí)被枝葉遮擋較嚴(yán)重時(shí)會(huì)出現(xiàn)漏識(shí)別或定位不準(zhǔn)的現(xiàn)象。本文方法在蘋果區(qū)域外預(yù)先設(shè)置了最小外接矩形,約束其識(shí)別范圍,故能夠準(zhǔn)確檢測出圖像中蘋果目標(biāo)所在的位置,避免了誤識(shí)別、漏識(shí)別現(xiàn)象。
由表3和表4可知:傳統(tǒng)梯度Hough圓變換的平均識(shí)別準(zhǔn)確率為69.27%、平均識(shí)別時(shí)間為1.211 s,文獻(xiàn)[21]的平均識(shí)別準(zhǔn)確率為72.17%、平均識(shí)別時(shí)間為0.850 s,本文方法的平均識(shí)別準(zhǔn)確率為90.22%、平均識(shí)別時(shí)間為0.643 s,較前兩種方法,本文方法的平均識(shí)別準(zhǔn)確率分別提高了20.95和18.05%,平均識(shí)別時(shí)間分別縮短了0.568和0.207 s。整體來說,本文方法在識(shí)別準(zhǔn)確率與識(shí)別時(shí)間上均有一定優(yōu)勢。
圖14 不同識(shí)別方法效果對(duì)比
表3 不同識(shí)別方法的識(shí)別準(zhǔn)確率對(duì)比
表4 不同識(shí)別方法的識(shí)別時(shí)間對(duì)比
本文方法未準(zhǔn)確識(shí)別的原因主要有:1)2個(gè)蘋果重疊過于嚴(yán)重,識(shí)別為1個(gè)蘋果。2)果實(shí)生長畸形,或者受光照影響而使果實(shí)邊緣出現(xiàn)嚴(yán)重凹陷,導(dǎo)致無法準(zhǔn)確識(shí)別。
為進(jìn)一步驗(yàn)證3種方法在不同光照下的識(shí)別結(jié)果,對(duì)采集的200幅蘋果圖像進(jìn)行測試(前100幅為順光拍攝,后100幅逆光拍攝),方法1為傳統(tǒng)梯度Hough圓變換,方法2為文獻(xiàn)[21]Hough圓變換,方法3為本文梯度Hough圓變換。順、逆光下3種方法的對(duì)比結(jié)果如圖15所示,蘋果識(shí)別準(zhǔn)確率與識(shí)別時(shí)間見表5與表6。
由測試結(jié)果可知,相比于傳統(tǒng)梯度Hough圓變換法與文獻(xiàn)[21]Hough圓變換法,本文梯度Hough圓變換法針對(duì)不同光照下的蘋果識(shí)別準(zhǔn)確率較高、識(shí)別時(shí)間較短。由表5和表6可知,本文方法在順光與逆光下的平均識(shí)別準(zhǔn)確率為90.75%、89.79%,較前2種方法分別提高15.03、10.50個(gè)百分點(diǎn)和16.41、9.60個(gè)百分點(diǎn)。本文方法在順光與逆光下的平均識(shí)別時(shí)間為0.665、0.693 s,較前2種方法分別縮短0.664、0.267和0.643、0.262 s。
本文方法在順光下的識(shí)別準(zhǔn)確率略優(yōu)于逆光,原因在于光照校正算法作用于蘋果圖像上僅可以改善低照度圖像亮度問題,但是不能完全消除弱光帶來的影響,光照問題仍會(huì)對(duì)果實(shí)的分割與識(shí)別定位效果造成影響。綜合來看,本文方法較其他方法識(shí)別準(zhǔn)確率較高,時(shí)間較短,能夠滿足不同光照下機(jī)器采摘對(duì)蘋果目標(biāo)準(zhǔn)確識(shí)別與快速處理的需求。
圖15 順、逆光下三種方法識(shí)別準(zhǔn)確率與識(shí)別時(shí)間對(duì)比
表5 三種方法識(shí)別結(jié)果對(duì)比
表6 三種方法平均識(shí)別時(shí)間對(duì)比
1)為了實(shí)現(xiàn)復(fù)雜環(huán)境中蘋果目標(biāo)的快速準(zhǔn)確識(shí)別,本研究以不同光照下隨機(jī)生長的蘋果目標(biāo)為測試對(duì)象,對(duì)采集到圖像進(jìn)行光照自適應(yīng)校正,改善光照條件不佳引起的圖像質(zhì)量問題,結(jié)合蘋果顏色特征,選取Cr分量下的蘋果圖像進(jìn)行改進(jìn)后的Otsu算法分割,初步得到蘋果目標(biāo),該分割方法有效抑制了背景區(qū)域中噪聲,提升了分割速率。再通過分水嶺算法建立粘連蘋果內(nèi)部的分割線,提取出單個(gè)蘋果目標(biāo)的外部輪廓,最后在輪廓外設(shè)置最小外接矩形標(biāo)記蘋果區(qū)域,在標(biāo)記區(qū)域內(nèi)進(jìn)行梯度Hough圓變換,準(zhǔn)確識(shí)別蘋果目標(biāo)。
2)所提的改進(jìn)后的Otsu算法能夠準(zhǔn)確快速地分割出蘋果目標(biāo)區(qū)域,平均分割時(shí)間為1.458 s,較改進(jìn)前Otsu算法和-means算法分別縮短了0.643和1.060 s。針對(duì)區(qū)域中蘋果粘連的問題,再結(jié)合基于距離變換的分水嶺分割算法進(jìn)行二次分割,可實(shí)現(xiàn)粘連蘋果的有效分離。
3)在不同光照條件下,采用最小外接矩形作為有效區(qū)域進(jìn)行梯度Hough圓變換識(shí)別蘋果均達(dá)到了較高的識(shí)別率,在順光下的識(shí)別準(zhǔn)確率為90.75%,平均識(shí)別時(shí)間為0.665 s,在逆光識(shí)別下的準(zhǔn)確率為89.79%,平均識(shí)別時(shí)間為0.693 s。
本研究提出的蘋果分割與識(shí)別方法,對(duì)于在不同光照下、枝葉遮擋及果實(shí)重疊的蘋果目標(biāo)識(shí)別具有較強(qiáng)的實(shí)用性,能滿足采摘機(jī)器人對(duì)蘋果識(shí)別和響應(yīng)時(shí)間的要求。由于果園蘋果姿態(tài)萬千,對(duì)于遮擋嚴(yán)重的蘋果目標(biāo)采用基于距離變換的分水嶺分割還有一定的局限性,如何有效地分離出遮擋的蘋果,后期需要更深入的研究分析。此外,光照過強(qiáng)或過弱會(huì)對(duì)識(shí)別準(zhǔn)確率造成輕微影響,完全克服光照的影響本文還尚未解決,后期也需要進(jìn)行深度研究。
[1] 司永勝,喬軍,劉剛,等. 蘋果采摘機(jī)器人果實(shí)識(shí)別與定位方法[J]. 農(nóng)業(yè)機(jī)械學(xué)報(bào),2010,41(9):148-153.
Si Yongsheng, Qiao Jun, Liu Gang, et al. Recognition and location of fruits for apple harvesting robot[J], Transactions of the Chinese Society for Agricultural Machinery, 2010, 41(9): 148-153. (in Chinese with English abstract)
[2] 王丹丹,宋懷波,何東健. 蘋果采摘機(jī)器人視覺系統(tǒng)研究進(jìn)展[J]. 農(nóng)業(yè)工程學(xué)報(bào),2017,33(10):59-69.
Wang Dandan, Song Huaibo, He Dongjian. Research advance on vision system of apple picking robot[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2017, 33(10): 59-69. (in Chinese with English abstract)
[3] Wei X Q, Jia K, Lan J H, et al. Automatic method of fruit object extraction under complex agricultural background for vision system of fruit picking robot[J]. Optik, 2014, 125(19): 5684-5689.
[4] 麥春艷,鄭立華,肖昌一,等. 自然光照條件下蘋果識(shí)別方法對(duì)比研究[J]. 中國農(nóng)業(yè)大學(xué)學(xué)報(bào),2016,21(11):43-50.
Mai Chunyan, Zheng Lihua, Xiao Changyi, et al. Comparison of apple recognition methods under natural light[J]. Journal of China Agricultural University, 2016, 21(11): 43-50. (in Chinese with English abstract)
[5] Kamilaris A, Prenafeta-Boldu F X. Deep learning in agriculture: A survey[J]. Computers and Electronics in Agriculture, 2018, 147: 70-90.
[6] Ji J T, Zhu X, Ma H, et al. Apple fruit recognition based on a deep learning algorithm using an improved lightweight network[J]. Applied Engineering in Agriculture, 2021, 37(1): 123-134.
[7] Kang H W, Chen C. Fast implementation of real-time fruit detection in apple orchards using deep learning - ScienceDirect[J]. Computers and Electronics in Agriculture, 2020, 168: 105108
[8] Fu L H, Duan J L, Zou X J, et al. Fast and accurate detection of banana fruits in complex background orchards[J]. IEEE Access, 2020, 8: 196835-196846.
[9] 鄭太雄,江明哲,馮明馳. 基于視覺的采摘機(jī)器人目標(biāo)識(shí)別與定位方法研究綜述[J]. 儀器儀表學(xué)報(bào),2021,42(9):28-51.
Zheng Taixiong, Jiang Mingzhe, Feng Mingchi. Vision based target recognition and location for picking robot: A review[J]. Chinese Journal of Scientific Instrument, 2021, 42(9): 28-51. (in Chinese with English abstract)
[10] 趙德安,吳任迪,劉曉洋,等. 基于YOLO深度卷積神經(jīng)網(wǎng)絡(luò)的復(fù)雜背景下機(jī)器人采摘蘋果定位[J]. 農(nóng)業(yè)工程學(xué)報(bào),2019,35(3):164-173.
Zhao De’an, Wu Rendi, Liu Xiaoyang, et al. Apple positioning based on YOLO deep convolutional neural network for picking robot in complex background[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2019, 35(3): 164-173. (in Chinese with English abstract)
[11] 劉芳,劉玉坤,林森,等. 基于改進(jìn)型YOLO的復(fù)雜環(huán)境下番茄果實(shí)快速識(shí)別方法[J]. 農(nóng)業(yè)機(jī)械學(xué)報(bào),2020,51(6):229-237.
Liu Fang, Liu Yukun, Lin Sen, et al. Fast recognition method for tomatoes under complex environments based on improved YOLO[J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(6): 229-237. (in Chinese with English abstract)
[12] 趙輝,喬艷軍,王紅君,等. 基于改進(jìn)YOLOv3的果園復(fù)雜環(huán)境下蘋果果實(shí)識(shí)別[J]. 農(nóng)業(yè)工程學(xué)報(bào),2021,37(16):127-135.
Zhao Hui, Qiao Yanjun, Wang Hongjun, et al. Recognition of apple fruit in complex orchard environment based on improved YOLOv3[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2021, 37(16): 127-135. (in Chinese with English abstract)
[13] Fu L S, Feng Y L, Majeed Y, et al. Kiwifruit detection in field images using faster R-CNN with ZFNet[J]. IFAC-Papers OnLine, 2018, 51(17): 45-50.
[14] 熊俊濤,劉振,湯林越,等. 自然環(huán)境下綠色柑橘視覺檢測技術(shù)研究[J]. 農(nóng)業(yè)機(jī)械學(xué)報(bào),2018,49(4):45-52.
Xiong Juntao, Lui Zhen, Tang Linyue, et al. Visual detection technology of green citrus under natural environment[J]. Transactions of the Chinese Society for Agricultural Machinery, 2018, 49(4): 45-52. (in Chinese with English abstract)
[15] 王丹丹,何東健. 基于R-FCN深度卷積神經(jīng)網(wǎng)絡(luò)的機(jī)器人疏果前蘋果目標(biāo)的識(shí)別[J]. 農(nóng)業(yè)工程學(xué)報(bào),2019,35(3):156-163.
Wang Dandan, He Dongjian. Recognition of apple targets before fruits thinning by robot based on R-FCN deep convolution neural network[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2019, 35(3): 156-163. (in Chinese with English abstract)
[16] Yu Y, Zhang K L, Yang L, et al. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN[J]. Computers and Electronics in Agriculture, 2019, 163: 104846.
[17] Yang C H, Xiong L Y, Wang Z, et al. Integrated detection of citrus fruits and branches using a convolutional neural network[J]. Computers and Electronics in Agriculture, 2020, 174: 105469.
[18] 麥春艷,鄭立華,孫紅,等. 基于RGB-D相機(jī)的果樹三維重構(gòu)與果實(shí)識(shí)別定位[J]. 農(nóng)業(yè)機(jī)械學(xué)報(bào),2015,46(S1):35-40.
Mai Chunyan, Zheng Lihua, Sun Hong, et al. Research on 3D reconstruction of fruit tree and fruit recognition and location method based on RGB-D camera[J]. Transactions of the Chinese Society for Agricultural Machinery, 2015, 46(S1): 35-40. (in Chinese with English abstract)
[19] 錢建平,楊信廷,吳曉明,等. 自然場景下基于混合顏色空間的成熟期蘋果識(shí)別方法[J]. 農(nóng)業(yè)工程學(xué)報(bào),2012,28(17):137-142.
Qian Jianping, Yang Xinting, Wu Xiaoming, et al. Mature apple recognition based on hybrid color space in natural scene[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2012, 28(17): 137-142. (in Chinese with English abstract)
[20] 謝忠紅,姬長英,郭小清,等. 基于改進(jìn)Hough變換的類圓果實(shí)目標(biāo)檢測[J]. 農(nóng)業(yè)工程學(xué)報(bào),2010,26(7):157-162.
Xie Zhonghong, Ji Changying, Guo Xiaoqing, et al. An object detection method for quasi-circular fruits based on improved Hough transform[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2010, 26(7): 157-162. (in Chinese with English abstract)
[21] 馮俊惠,李志偉,戎有麗,等.基于改進(jìn)Hough圓變換算法的成熟番茄果實(shí)識(shí)別[J]. 中國農(nóng)機(jī)化學(xué)報(bào),2021,42(4):190-196.
Feng Junhui, Li Zhiwei, Rong Youli, et al. Identification of mature tomatos based on an algorithm of modified circular Hough transform[J]. Journal of Chinese Agricultural Mechanization, 2021, 42(4): 190-196. (in Chinese with English abstract)
[22] 周文靜,查志華,吳杰. 改進(jìn)圓形Hough變換的田間紅提葡萄果穗成熟度判別[J]. 農(nóng)業(yè)工程學(xué)報(bào),2020,36(9):205-213.
Zhou Wenjing, Zha Zhihua, Wu Jie. Maturity discrimination of “Red Globe” grape cluster in grapery by improved circle Hough transform[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2020, 36(9): 205-213. (in Chinese with English abstract)
[23] Lv J D, Zhao D A Z, Ji W, et al. Recognition of apple fruit in natural environment[J]. Optik, 2016, 127(3): 1354-1362.
[24] Gongal A, Silwal A, Amatya S, et al. Apple crop-load estimation with over-the-row machine vision system[J]. Computers & Electronics in Agriculture, 2015, 120: 26-35.
[25] 苗中華,沈一籌,王小華,等. 自然環(huán)境下重疊果實(shí)圖像識(shí)別算法與試驗(yàn)[J]. 農(nóng)業(yè)機(jī)械學(xué)報(bào),2016,47(6):21-26.
Miao Zhonghua, Shen Yichou, Wang Xiaohua, et al. Recognition of mature oranges in natural scene based on machine vision[J]. Transactions of the Chinese Society for Agricultural Machinery, 2016, 47(6): 21-26. (in Chinese with English abstract)
[26] 蔡健榮,周小軍,李玉良,等. 基于機(jī)器視覺自然場景下成熟柑橘識(shí)別[J]. 農(nóng)業(yè)工程學(xué)報(bào),2008,24(1):175-178.
Cai Jianrong, Zhou Xiaojun, Li Yuliang, et al. Recognition of ripe citrus in natural scene based on machine vision[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2008, 24(1): 175-178. (in Chinese with English abstract)
[27] 崔永杰,蘇帥,王霞霞,等. 基于機(jī)器視覺的自然環(huán)境中獼猴桃識(shí)別與特征提取[J]. 農(nóng)業(yè)機(jī)械學(xué)報(bào),2013,44(5):247-252.
Cui Yongjie, Su Shuai, Wang Xiaxia, et al. Recognition and feature extraction of kiwifruit in natural environment based on machine vision[J]. Transactions of the Chinese Society for Agricultural Machinery, 2013, 44(5): 247-252. (in Chinese with English abstract)
[28] 劉志成,王殿偉,劉穎,等. 基于二維伽馬函數(shù)的光照不均勻圖像自適應(yīng)校正算法[J]. 北京理工大學(xué)學(xué)報(bào),2016,36(2):191-196,214.
Liu Zhicheng, Wang Dianwei, Liu Ying, et al. Adaptive adjustment algorithm for non-uniform illumination image based on 2D gamma function[J]. Transactions of Beijing Institute of Technology, 2016, 36(2): 191-196, 214. (in Chinese with English abstract)
[29] 孫俊,宋佳,武小紅,等. 基于改進(jìn)Otsu算法的生菜葉片圖像分割方法[J]. 江蘇大學(xué)學(xué)報(bào)(自然科學(xué)版),2018,39(2):179-184.
Sun Jun, Song Jia, Wu Xiaohong, et al. Image segmentation method of lettuce leaf based on improved Otsu algorithm[J]. Journal of Jiangsu University(Natural Science Edition), 2018, 39(2): 179-184. (in Chinese with English abstract)
[30] 張文飛,韓建海,郭冰菁,等. 改進(jìn)的分水嶺算法在粘連圖像分割中的應(yīng)用[J]. 計(jì)算機(jī)應(yīng)用與軟件,2021,38(6):243-248.
Zhang Wenfei, Han Jianhai, Guo Bingjing, et al. Application of improved watershed algorithm in adhesion image segmentation[J]. Computer Applications and Software, 2021, 38(6): 243-248. (in Chinese with English abstract)
Apple recognition based on two-level segmentation and region-marked gradient Hough circle transform
Zhang Yanfei1, Liu Mingyang1, Gong Jinliang2※, Lan Yubin1
(1,,255049,;2.,,255049,)
Apples are produced in the large quantities each year, particularly for as the largest economic fruit in China. It is highly required for the rapid picking within the harvesting period. Therefore, the automatic apple picking is essential to the apple harvesting in intensive farming. An accurate and rapid identification of fruit can be fundamental for the automatic picking. However, some environmental factors surrounding the fruit can pose a great interference in the fruit identification under the natural, complex, and variable backgrounds, such as the light intensity, occlusion, and overlap of the fruit. In this study, an apple recognition was proposed using two-level segmentation and region-marked Hough transform. Experimental results show that the robust and practical performance was achieved for the apple recognition under different illumination, branch and leaf occlusion, as well as the fruit overlap. Specific steps were as follows. Firstly, the front camera (NikonD90) of the information acquisition robot was used to capture from 600-800 mm away from the fruit tree under the conditions of nature natural light and backlight, respectively. The brightness adaptive correction algorithm was then used to correct the brightness of apple images with the uneven distribution of surface brightness, in order to enhance the image details. The Cr component images of YCbCr color space were extracted as the preprocessing samples to combine with the feature of the color of the apple. Secondly, the improved Otsu algorithm was utilized to obtain the binary image of the apple target for the initial segmentation, in order to accurately extract the contour of the target fruit under different growth states (mainly including single and double fruits with the overlap and occlusion). A morphological open-close reconstruction filter was also introduced to the Otsu algorithm to remove the background noise. The traversal range of the gray level was reduced to shorten the complexity and running time of the algorithm for the high segmentation rate. Thirdly, the watershed algorithm was combined to perform the secondary segmentation of the segmented fruit region using distance transformation. The conglutinated and overlapping apples were separated to effectively extract the apple target contour. Finally, the gradient Hough circle transformation was selected to identify the number of apples. But the algorithm traveled through the whole image for the computational complexity, time time-consuming, and easy to produce the false identification. Therefore, the minimum circumscribed rectangle outside the contour was set as the effective area for the gradient Hough circle transform in the effective area, particularly for the recognition speed and accuracy. The experimental results show that: 1) The improved Otsu algorithm was achieved in the higher segmentation accuracy of fruit targets, especially with the less segmentation time. The improved algorithm was also filled the tiny holes in the apple to suppress the noise in the background branches and leaves, further to more clearly segment the target region more than before. The average segmentation time was 1.458 s, which was 0.643 and 1.060 s shorter than the Otsu and-means algorithm before the improvement. In the adhesion of partial apple regions in the binary images, a watershed segmentation with the distance transformation was used for the quadratic segmentation to effectively separate the sticky apples for the full apple target boundary. 2) The gradient Hough circle transform was used to recognize the 200 apple images under different lighting conditions, where the recognition accuracy was 90.75% in the nature natural light and 89.79% in the backlight, which were improved by 15.03 and 16.41 percentage points, respectively, compared with the traditional. The average recognition time was 0.665 and 0.693 s, which was 0.664 and 0.643 s shorter than before. Therefore, the proposed algorithm can meet the requirements of apple-picking robots, in terms of recognition speed and accuracy. The findings can provide a strong reference for the fast recognition of spherical fruits, such as apples.
image processing; watershed; apple recognition; YCbCr; Otsu; gradient Hough circle transform
10.11975/j.issn.1002-6819.2022.19.013
S225.93
A
1002-6819(2022)-19-0110-12
張彥斐,劉茗洋,宮金良,等. 基于兩級(jí)分割與區(qū)域標(biāo)記梯度Hough圓變換的蘋果識(shí)別[J]. 農(nóng)業(yè)工程學(xué)報(bào),2022,38(19):110-121.doi:10.11975/j.issn.1002-6819.2022.19.013 http://www.tcsae.org
Zhang Yanfei, Liu Mingyang, Gong Jinliang, et al. Apple recognition based on two-level segmentation and region-marked gradient Hough circle transform[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2022, 38(19): 110-121. (in Chinese with English abstract) doi:10.11975/j.issn.1002-6819.2022.19.013 http://www.tcsae.org
2022-06-06
2022-08-17
山東省引進(jìn)頂尖人才“一事一議”專項(xiàng)經(jīng)費(fèi)資助項(xiàng)目(魯政辦字[2018]27號(hào));山東省重點(diǎn)研發(fā)計(jì)劃(重大科技創(chuàng)新工程)項(xiàng)目(2020CXGC010804);山東省自然科學(xué)基金項(xiàng)目(ZR202102210303)
張彥斐,博士,教授,研究方向?yàn)闄C(jī)器人與智能農(nóng)機(jī)裝備。Email:1392076@sina.com
宮金良,博士,副教授,研究方向?yàn)闄C(jī)器人與智能農(nóng)機(jī)裝備。Email:gjlwing@qq.com