于景鑫,杜 森,吳 勇,鐘永紅,張鐘莉莉,鄭文剛,李文龍,3
基于云原生技術(shù)的土壤墑情監(jiān)測(cè)系統(tǒng)設(shè)計(jì)與應(yīng)用
于景鑫1,4,杜 森2※,吳 勇2,鐘永紅2,張鐘莉莉1,鄭文剛1,李文龍1,3
(1. 國(guó)家農(nóng)業(yè)信息化工程技術(shù)研究中心,北京 100097;2. 全國(guó)農(nóng)業(yè)技術(shù)推廣服務(wù)中心,北京 100125;3. 農(nóng)業(yè)農(nóng)村部農(nóng)業(yè)信息軟硬件產(chǎn)品質(zhì)量檢測(cè)重點(diǎn)實(shí)驗(yàn)室,北京 100097;4. 中國(guó)地質(zhì)大學(xué)(北京)土地科學(xué)與技術(shù)學(xué)院,北京 100083)
該研究針對(duì)全中國(guó)尺度的土壤墑情監(jiān)測(cè)需求,構(gòu)建基于自動(dòng)監(jiān)測(cè)站原位監(jiān)測(cè)與多源專題數(shù)據(jù)的土壤墑情數(shù)據(jù)獲取感知技術(shù)體系,提出數(shù)據(jù)質(zhì)量控制清洗策略并建立數(shù)據(jù)校正插補(bǔ)模型。系統(tǒng)基于云原生技術(shù)設(shè)計(jì),將模塊以微服務(wù)形式靈活開發(fā)部署,通過容器技術(shù)打包運(yùn)行獨(dú)立實(shí)例,布設(shè)了墑情數(shù)據(jù)上報(bào)采集、可視化分析和數(shù)據(jù)挖掘應(yīng)用等核心模塊。依托空間分析和WebGL技術(shù)開發(fā)3D WebGIS數(shù)據(jù)分析功能模塊,實(shí)現(xiàn)協(xié)同土壤墑情、土地利用、海拔高程等多源數(shù)據(jù)可視化分析與制圖,深入挖掘數(shù)據(jù)價(jià)值,實(shí)現(xiàn)墑情估算和基于水量平衡的灌溉決策應(yīng)用服務(wù)。系統(tǒng)已在中國(guó)21個(gè)省份得到應(yīng)用,建立自動(dòng)監(jiān)測(cè)站970個(gè),采集監(jiān)測(cè)數(shù)據(jù)6 000余萬(wàn)條,為用戶掌握土壤墑情現(xiàn)狀、指導(dǎo)農(nóng)業(yè)節(jié)水灌溉、獲取可靠科研數(shù)據(jù)等應(yīng)用提供數(shù)據(jù)與技術(shù)服務(wù)。
土壤墑情;監(jiān)測(cè);系統(tǒng)設(shè)計(jì);數(shù)據(jù)感知;WebGIS;深度學(xué)習(xí)
當(dāng)前,中國(guó)對(duì)于水資源高效利用的需求愈發(fā)迫切,而農(nóng)業(yè)用水總量占據(jù)經(jīng)濟(jì)社會(huì)用水總量高達(dá)60%左右[1],急需發(fā)展節(jié)水農(nóng)業(yè)。土壤含水率是精準(zhǔn)灌溉重要的參數(shù),在精準(zhǔn)灌溉中保證作物根區(qū)土壤含水率在適宜區(qū)間是實(shí)現(xiàn)作物水分高效利用的關(guān)鍵環(huán)節(jié)[2]。2020年農(nóng)業(yè)農(nóng)村部種植業(yè)重點(diǎn)工作提出將“測(cè)墑節(jié)灌”作為農(nóng)業(yè)節(jié)水工作的重點(diǎn)任務(wù),使得對(duì)土壤墑情監(jiān)測(cè)系統(tǒng)的研究具有重要意義。傳統(tǒng)的土壤水分測(cè)量方法一般是進(jìn)行烘干稱重法量測(cè),需要人工在田間利用取土鉆獲取土樣,隨后土樣在實(shí)驗(yàn)室稱重并放入烘干箱,土樣需在105~110 ℃高溫下烘干超過12 h形成干土,通過測(cè)量干土與原始土樣質(zhì)量差得出土壤含水率,烘干法測(cè)量方式不僅操作繁瑣、數(shù)據(jù)獲取滯后、采樣難度大,而且還會(huì)破壞原狀土體[3]。隨著傳感器技術(shù)的發(fā)展,利用土壤墑情監(jiān)測(cè)站可以快速準(zhǔn)確測(cè)定不同深度土壤含水率并通過移動(dòng)通訊網(wǎng)絡(luò)實(shí)時(shí)上傳至數(shù)據(jù)中心,使得土壤水分?jǐn)?shù)據(jù)高效采集成為可能[4]。
美國(guó)農(nóng)業(yè)部(United States Department of Agriculture,USDA)于1991年啟動(dòng)國(guó)家土壤氣候分析網(wǎng)絡(luò)(National Soil Climate Analysis Network,SCAN)項(xiàng)目,該系統(tǒng)可以監(jiān)控并報(bào)告全美200多個(gè)站點(diǎn)的土壤濕度、土壤溫度和其他氣候數(shù)據(jù)[5]。1994年,美國(guó)俄克拉荷馬大學(xué)開發(fā)了環(huán)境監(jiān)測(cè)系統(tǒng)(Mesonet),由覆蓋俄克拉荷馬州的120個(gè)自動(dòng)觀測(cè)站組成,自動(dòng)觀測(cè)站采集5 min間隔頻率的氣候和土壤水分?jǐn)?shù)據(jù),目前該系統(tǒng)也逐漸擴(kuò)展到林業(yè)、農(nóng)業(yè)生產(chǎn)服務(wù)領(lǐng)域[6]。2014年,美國(guó)地質(zhì)調(diào)查局(United States Geological Survey,USGS)牽頭的國(guó)家土壤墑情網(wǎng)絡(luò)(National Soil Moisture Network,NSMN)項(xiàng)目融合全美15個(gè)土壤墑情原位監(jiān)測(cè)網(wǎng)數(shù)據(jù),系統(tǒng)提供在線插值制圖、遙感數(shù)據(jù)下載和混合制圖等功能[7]。2015年國(guó)家氣象科學(xué)數(shù)據(jù)中心建立了中國(guó)氣象數(shù)據(jù)網(wǎng)平臺(tái),提供1991年至今中國(guó)653個(gè)農(nóng)業(yè)氣象站點(diǎn)所采集的逐旬土壤水分和氣象數(shù)據(jù)[8]。但就農(nóng)業(yè)生產(chǎn)全過程而言,土壤墑情作為其中的關(guān)鍵指標(biāo)直接決定作物水分、農(nóng)田旱澇情況,同時(shí)又受到如土壤、作物等多種因素的影響。目前針對(duì)農(nóng)業(yè)應(yīng)用的土壤墑情系統(tǒng)還存在以下問題:1)數(shù)據(jù)以土壤墑情為主,種類較為單一,還缺乏相關(guān)地理信息、作物、氣象、土壤數(shù)據(jù)等;2)系統(tǒng)以實(shí)時(shí)提供土壤墑情現(xiàn)狀數(shù)據(jù)為主,需要提供對(duì)數(shù)據(jù)的估算能力來(lái)把握未來(lái)趨勢(shì);3)系統(tǒng)以展示墑情分布為主,需要對(duì)土壤墑情數(shù)據(jù)的深入挖掘來(lái)提升指導(dǎo)農(nóng)業(yè)生產(chǎn)應(yīng)用的效率。
云原生(cloud native)技術(shù)是在云計(jì)算環(huán)境下構(gòu)建用于部署動(dòng)態(tài)微服務(wù)應(yīng)用的軟件堆棧,通過將各組件打包到容器(container)中,動(dòng)態(tài)調(diào)度容器以優(yōu)化云計(jì)算資源利用率,該技術(shù)具有敏捷開發(fā)、性能可靠、高彈性、易擴(kuò)展、故障隔離和持續(xù)更新等特性[9]。相比于傳統(tǒng)的Web架構(gòu),云原生技術(shù)能夠保證系統(tǒng)更加穩(wěn)定可靠運(yùn)行[10]。面向全國(guó)的土壤墑情監(jiān)測(cè)系統(tǒng)具有自動(dòng)站設(shè)備多、用戶訪問量大、數(shù)據(jù)運(yùn)算量大,具有高頻率、高并發(fā)、持續(xù)增長(zhǎng)的特點(diǎn),因此需要適配云計(jì)算特性的云原生技術(shù),利用微服務(wù)架構(gòu)和容器技術(shù)構(gòu)建靈活的開發(fā)模式并提升計(jì)算資源利用效率。
本研究針對(duì)此背景,結(jié)合中國(guó)土壤墑情監(jiān)測(cè)工作的實(shí)際需要,面向政府和各級(jí)農(nóng)業(yè)管理、技術(shù)推廣、科研人員等設(shè)計(jì)開發(fā)了基于云原生架構(gòu)的土壤墑情監(jiān)測(cè)系統(tǒng),旨在實(shí)現(xiàn)以下幾個(gè)方面功能:1)構(gòu)建土壤墑情數(shù)據(jù)感知技術(shù)方案,解決數(shù)據(jù)實(shí)時(shí)獲取和多源異構(gòu)數(shù)據(jù)融合問題;2)提出數(shù)據(jù)質(zhì)量控制策略,運(yùn)用深度學(xué)習(xí)技術(shù)實(shí)現(xiàn)缺失數(shù)據(jù)插補(bǔ);3)協(xié)同多源數(shù)據(jù)挖掘,實(shí)現(xiàn)土壤墑情預(yù)報(bào)和灌溉決策應(yīng)用。
土壤墑情數(shù)據(jù)感知的核心任務(wù)是農(nóng)田多層深度土壤水分的自動(dòng)采集與相關(guān)屬性及數(shù)據(jù)的在線化服務(wù),形成連續(xù)、準(zhǔn)確、可靠的土壤墑情大數(shù)據(jù)。本研究提出采用物聯(lián)網(wǎng)自動(dòng)設(shè)備監(jiān)測(cè)、深度學(xué)習(xí)校驗(yàn)插補(bǔ)建模和跨平臺(tái)數(shù)據(jù)協(xié)同獲取專題數(shù)據(jù)相結(jié)合的方式,構(gòu)建土壤墑情數(shù)據(jù)感知技術(shù),實(shí)現(xiàn)土壤墑情在線監(jiān)測(cè)與多源數(shù)據(jù)融合,主要技術(shù)流程如圖1所示。
注:DEM表示數(shù)字高程模型,LUCC表示土地利用與土地覆被變化,DBMS表示數(shù)據(jù)庫(kù)管理系統(tǒng)。下同。
土壤含水率傳感器主要采用時(shí)域反射(Time Domain Reflector,TDR)、頻域反射(Frequency Domain Reflectometry,F(xiàn)DR)技術(shù)方式測(cè)量土壤介電常數(shù),通過傳感器標(biāo)定模型轉(zhuǎn)換后得到土壤體積含水率,其優(yōu)勢(shì)是自動(dòng)化測(cè)量、人為干預(yù)少和采集頻率高[11]。
系統(tǒng)采用固定式遠(yuǎn)程土壤墑情監(jiān)測(cè)站實(shí)現(xiàn)土壤墑情和農(nóng)田氣象數(shù)據(jù)采集,設(shè)備具有自動(dòng)采集、存儲(chǔ)、遠(yuǎn)程傳輸?shù)裙δ堋M寥缐勄樾枰@取0~20、>20~40、>40~60和>60~80 cm 4個(gè)土層深度的土壤含水率和土壤溫度數(shù)據(jù),傳感器參數(shù)需滿足表1要求。農(nóng)田氣象數(shù)據(jù)包含空氣溫度、空氣濕度、降雨量、風(fēng)速、參考作物蒸散量(reference Evapotranspiration,ET0)等。監(jiān)測(cè)站點(diǎn)每小時(shí)自動(dòng)采集一次數(shù)據(jù),整合形成符合接收端口協(xié)議規(guī)范的報(bào)文,通過通用無(wú)線分組業(yè)務(wù)(General Packet Radio Service,GPRS)網(wǎng)絡(luò)將報(bào)文以TCP/IP協(xié)議上傳至云端系統(tǒng)數(shù)據(jù)接收后臺(tái),以實(shí)現(xiàn)土壤墑情原位監(jiān)測(cè)。
表1 土壤墑情傳感器技術(shù)指標(biāo)
系統(tǒng)利用多線程技術(shù)和TCP/IP數(shù)據(jù)傳輸協(xié)議構(gòu)建獨(dú)立的C/S(Client/Server)模式數(shù)據(jù)接收后臺(tái),實(shí)現(xiàn)地面自動(dòng)農(nóng)田氣象墑情監(jiān)測(cè)站回傳數(shù)據(jù)可靠傳輸。數(shù)據(jù)后臺(tái)在服務(wù)器端實(shí)現(xiàn)監(jiān)聽Socket、接受客戶端連接請(qǐng)求、維護(hù)Socket鏈表、數(shù)據(jù)解析、數(shù)據(jù)處理分析、數(shù)據(jù)存儲(chǔ)和日志記錄等功能。
除土壤墑情、農(nóng)田氣象數(shù)據(jù)外,需要整合非傳感器實(shí)時(shí)快速獲取的專題數(shù)據(jù),系統(tǒng)提出構(gòu)建多源異構(gòu)專題數(shù)據(jù)獲取機(jī)制。針對(duì)行政區(qū)邊界、數(shù)字高程模型(Digital Elevation Model,DEM)、土地利用類型、坡度等不同格式的地理信息系統(tǒng)(Geographic Information System,GIS)空間數(shù)據(jù),通過GIS數(shù)據(jù)共享網(wǎng)站獲取并統(tǒng)一存儲(chǔ)于ArcGIS Geodatabase數(shù)據(jù)庫(kù)中[12]。針對(duì)如農(nóng)業(yè)生產(chǎn)中作物名稱、生育期、土壤信息等需要用戶上報(bào)的文字類非結(jié)構(gòu)化的數(shù)據(jù),系統(tǒng)通過規(guī)范數(shù)據(jù)項(xiàng)名稱和統(tǒng)一數(shù)據(jù)選項(xiàng),讓用戶在系統(tǒng)界面中以選項(xiàng)的方式上報(bào)數(shù)據(jù),避免了人為錄入錯(cuò)誤和規(guī)則不同造成的混亂,以此將非結(jié)構(gòu)化語(yǔ)義數(shù)據(jù)轉(zhuǎn)化為結(jié)構(gòu)化數(shù)據(jù)并存儲(chǔ)于通用的關(guān)系型數(shù)據(jù)庫(kù)管理系統(tǒng)(Database Management System,DBMS)。系統(tǒng)通過構(gòu)建統(tǒng)一數(shù)據(jù)訪問層(data access layer)實(shí)現(xiàn)多源異構(gòu)數(shù)據(jù)融合管理,為后續(xù)進(jìn)行多源異構(gòu)大數(shù)據(jù)整合分析提供數(shù)據(jù)基礎(chǔ)。
自動(dòng)墑情監(jiān)測(cè)站一般安置于田間,周圍環(huán)境復(fù)雜,作物生長(zhǎng)、設(shè)備穩(wěn)定性、極端氣候等因素都有可能造成設(shè)備數(shù)據(jù)異常和缺失,降低數(shù)據(jù)可用性,為保證數(shù)據(jù)準(zhǔn)確、可靠和連續(xù),本研究提出數(shù)據(jù)質(zhì)量控制標(biāo)準(zhǔn)與數(shù)據(jù)插補(bǔ)方法,云端后臺(tái)收到符合TCP/IP協(xié)議的物聯(lián)網(wǎng)設(shè)備回傳的報(bào)文數(shù)據(jù)后進(jìn)行解析和質(zhì)量判定,對(duì)于異常或者缺失的數(shù)據(jù),通過數(shù)據(jù)校正插補(bǔ)模型進(jìn)行估算,避免數(shù)據(jù)中斷缺失造成的可用性喪失問題,保證數(shù)據(jù)的準(zhǔn)確性、完整性和可用性。
自動(dòng)墑情監(jiān)測(cè)站主要觀測(cè)指標(biāo)為土壤含水率以及農(nóng)田氣象信息,設(shè)備上傳報(bào)文采用十進(jìn)制字符串格式,本研究提出土壤墑情數(shù)據(jù)質(zhì)量控制技術(shù)流程(圖2),具體規(guī)則如下:
1)格式檢查:校驗(yàn)包括設(shè)備參數(shù)、報(bào)文編碼字節(jié)、發(fā)報(bào)時(shí)間等,報(bào)文正確解析且通過上述校驗(yàn)的數(shù)據(jù)為合格;
2)界限值檢查:通過設(shè)置土壤體積含水率觀測(cè)值的置信區(qū)間上、下界限實(shí)現(xiàn),土壤體積含水率(%)在(0,60)區(qū)間為合格;
3)內(nèi)部一致性檢查:若土壤體積含水率各層的觀測(cè)值完全相同則判定為數(shù)據(jù)錯(cuò)誤;
4)時(shí)間一致性檢查:若前后數(shù)據(jù)土壤相對(duì)含水率突降超20%或者當(dāng)降水量>10 mm/h而表層0~20 cm的土壤體積含水率2 h內(nèi)未增加則判定為數(shù)據(jù)錯(cuò)誤。
圖2 土壤墑情數(shù)據(jù)質(zhì)量控制技術(shù)流程
自動(dòng)墑情監(jiān)測(cè)站數(shù)據(jù)異常和缺失會(huì)造成土壤墑情適宜度判斷的錯(cuò)誤,尤其在關(guān)鍵農(nóng)時(shí)將會(huì)影響后續(xù)的農(nóng)事操作,因此需要對(duì)異常和缺失數(shù)據(jù)進(jìn)行校正和插補(bǔ)。土壤墑情數(shù)據(jù)呈現(xiàn)復(fù)雜的非線性關(guān)系,利用普通線性模型很難進(jìn)行模型擬合,面向海量、復(fù)雜、無(wú)明確關(guān)系的大數(shù)據(jù)擬合算法中,深度學(xué)習(xí)算法是目前最佳的選擇[13]。
系統(tǒng)的數(shù)據(jù)校正插補(bǔ)模塊定時(shí)掃描數(shù)據(jù)庫(kù),對(duì)土壤墑情數(shù)據(jù)進(jìn)行質(zhì)量評(píng)價(jià),針對(duì)數(shù)據(jù)質(zhì)量控制單元所判定的異常和缺失數(shù)據(jù)利用模型進(jìn)行校正和插補(bǔ),其中校正插補(bǔ)模型分別利用循環(huán)神經(jīng)網(wǎng)絡(luò)(Recurrent Neural Networks, RNN)對(duì)時(shí)間序列特征提取和卷積神經(jīng)網(wǎng)絡(luò)(Convolutional Neural Networks, CNN)對(duì)網(wǎng)格圖像高維特征提取的特性通過Stacking集成學(xué)習(xí)方式構(gòu)建[14],模型通過Python Flask發(fā)布為REST API接口以供外部調(diào)用[15],校正插補(bǔ)后的數(shù)據(jù)標(biāo)記相應(yīng)質(zhì)量代碼存入數(shù)據(jù)庫(kù)并記錄日志(圖3),確保數(shù)據(jù)的準(zhǔn)確和完整。
圖3 土壤墑情數(shù)據(jù)校正與插補(bǔ)技術(shù)流程
3.1.1 微服務(wù)架構(gòu)
微服務(wù)概念是一種新的架構(gòu)模式,將單一應(yīng)用程序劃分成一組小的服務(wù)重塑了面向服務(wù)架構(gòu)模式,通過服務(wù)之間相互協(xié)調(diào)、互相配合,為用戶提供最終價(jià)值[16]。微服務(wù)不僅圍繞著具體的業(yè)務(wù)進(jìn)行構(gòu)建,同時(shí)能夠獨(dú)立部署到生產(chǎn)環(huán)境、測(cè)試環(huán)境等,避免了統(tǒng)一、集中開發(fā)管理機(jī)制帶來(lái)的資源浪費(fèi)[17]。
土壤墑情監(jiān)測(cè)系統(tǒng)功能模塊多、業(yè)務(wù)功能復(fù)雜,需要不同技術(shù)和專業(yè)背景的人員共同參與開發(fā),系統(tǒng)采用微服務(wù)架構(gòu),避免傳統(tǒng)的開發(fā)模式需要統(tǒng)一開發(fā)環(huán)境、開發(fā)語(yǔ)言、部署環(huán)境等各類要素的要求,針對(duì)具體業(yè)務(wù)邏輯,選擇合適的語(yǔ)言、工具進(jìn)行開發(fā),從而提高開發(fā)效率。系統(tǒng)在服務(wù)器資源層面確保每個(gè)微服務(wù)實(shí)例運(yùn)行在其獨(dú)立的進(jìn)程中,各微服務(wù)之間采用基于HTTP的Restful API通信機(jī)制進(jìn)行輕量級(jí)的數(shù)據(jù)交互(圖4),構(gòu)建了靈活的架構(gòu)設(shè)計(jì)。
圖4 微服務(wù)技術(shù)架構(gòu)運(yùn)行機(jī)制
3.1.2 容器化技術(shù)
容器技術(shù)(container)是一種被廣泛認(rèn)可的服務(wù)器虛擬化資源共享方式,其可以按需構(gòu)建容器技術(shù)操作系統(tǒng)實(shí)例的特性,為系統(tǒng)管理員提供極大的靈活性,其主要特點(diǎn)為極其輕量、秒級(jí)部署、易于移植和彈性伸縮[18]。
系統(tǒng)采用容器技術(shù)來(lái)配合微服務(wù)架構(gòu)模式使得系統(tǒng)易于開發(fā)、維護(hù)和按需伸縮,針對(duì)獨(dú)立微服務(wù)利用容器把應(yīng)用和其運(yùn)行環(huán)境以高級(jí)多層統(tǒng)一文件系統(tǒng)(Advanced Multi-Layered Unification File System,AUFS)打包來(lái)保證應(yīng)用及其運(yùn)行環(huán)境的統(tǒng)一,并在裝有容器環(huán)境(Docker)的云計(jì)算基礎(chǔ)設(shè)施上以容器方式運(yùn)行,通過容器編排工具對(duì)容器服務(wù)的編排來(lái)實(shí)現(xiàn)容器啟動(dòng)、容器應(yīng)用部署、容器應(yīng)用在線升級(jí)等功能,利用容器集群將多臺(tái)物理機(jī)抽象為邏輯上單一調(diào)度實(shí)體的技術(shù),提供資源調(diào)度、服務(wù)發(fā)現(xiàn)、彈性伸縮、負(fù)載均衡等功能,充分利用云計(jì)算基礎(chǔ)設(shè)施資源。
通過以上土壤墑情數(shù)據(jù)感知技術(shù)獲取的數(shù)據(jù)資源和云原生技術(shù)架構(gòu)闡釋的系統(tǒng)開發(fā)方法理念,本研究選用主流開源軟件堆棧作為基礎(chǔ)軟件環(huán)境,在云計(jì)算框架下以微服務(wù)、容器技術(shù)為核心的云原生架構(gòu)進(jìn)行面向中國(guó)的土壤墑情監(jiān)測(cè)系統(tǒng)的設(shè)計(jì)與研發(fā),兼顧成熟開發(fā)方案配置和最新技術(shù)特性,保障系統(tǒng)的可靠性、先進(jìn)性和動(dòng)態(tài)擴(kuò)展性(圖5)。
注:HTTP是超文本傳輸協(xié)議,Websocket是一種全雙工通信的協(xié)議,API表示應(yīng)用程序接口,APP表示手機(jī)應(yīng)用程序,ET0表示參考作物蒸散量,mm/d。
土壤墑情監(jiān)測(cè)系統(tǒng)采用開源的Linux CentOS 7.2環(huán)境作為系統(tǒng)運(yùn)行環(huán)境,容器調(diào)度采用開源的容器編排調(diào)度引擎Kubernetes[19],容器技術(shù)采用Docker開源的應(yīng)用容器引擎[20],以業(yè)務(wù)需求和開發(fā)團(tuán)隊(duì)技術(shù)領(lǐng)域劃分微服務(wù)功能邊界并通過Nginx Web服務(wù)器配合Atlas+Keepalived中間件實(shí)現(xiàn)Web平臺(tái)與MySQL數(shù)據(jù)庫(kù)集群的反向代理和負(fù)載均衡[21]。通過在Kubernetes平臺(tái)上集成Gitlab代碼管理和Jenkins集成工具的敏捷迭代特性實(shí)現(xiàn)DevOps容器化敏捷開發(fā)運(yùn)維模式[22]。系統(tǒng)采用Html5前端技術(shù)開發(fā)Web用戶交互頁(yè)面(圖6),業(yè)務(wù)層布設(shè)了墑情數(shù)據(jù)分析、數(shù)據(jù)填報(bào)、GIS制圖分析和墑情數(shù)據(jù)挖掘應(yīng)用等核心模塊,為各級(jí)農(nóng)業(yè)節(jié)水管理人員、農(nóng)技人員、行業(yè)專家、企業(yè)用戶和科研機(jī)構(gòu)等提供可靠、穩(wěn)定、高性能的土壤墑情數(shù)據(jù)的獲取管理與挖掘分析服務(wù)。
圖6 土壤墑情監(jiān)測(cè)系統(tǒng)界面
3.3.1 3D WebGIS可視化
系統(tǒng)基于WebGL技術(shù)實(shí)現(xiàn)瀏覽器端3D WebGIS可視化[23],前端基于ArcGIS API for JavaScript 4.1通過場(chǎng)景視圖(scene view)實(shí)現(xiàn)瀏覽器端3D視圖瀏覽和基礎(chǔ)控件,GIS數(shù)據(jù)從空間數(shù)據(jù)庫(kù)(Geodatabase)中調(diào)取并以特征圖層(feature layer)形式加載。GIS后臺(tái)采用ArcGIS Server發(fā)布GIS數(shù)據(jù)和模型服務(wù)并通過地處理(Geoprocessor,GP)服務(wù)的形式調(diào)取,通過配置打印參數(shù)(print parameters)根據(jù)用戶圖層設(shè)置動(dòng)態(tài)調(diào)取打印服務(wù)(print task)實(shí)現(xiàn)地圖打印,其中地圖制圖模板(print template)通過服務(wù)器端配置的.mxd文件進(jìn)行管理。
3.3.2 協(xié)同空間分析制圖
土壤墑情數(shù)據(jù)空間插值制圖功能可以實(shí)現(xiàn)由點(diǎn)到面的空間數(shù)據(jù)拓展[24],其流程為獲取運(yùn)算后的空間點(diǎn)位數(shù)據(jù),空間插值分析,農(nóng)田區(qū)域掩膜裁剪,墑情等級(jí)重分類渲染,最終展示在前端實(shí)現(xiàn)分析與可視化制圖。本研究土壤墑情插值采用協(xié)同克里金插值法[25],選擇高程、坡度和土地利用分類為土壤水分“趨勢(shì)”擬合的協(xié)同考慮因子,如式(1)所示
以空間插值制圖為例,選取任意時(shí)間段范圍和制圖層次,系統(tǒng)調(diào)度相應(yīng)的微服務(wù)進(jìn)行數(shù)據(jù)獲取、點(diǎn)位運(yùn)算、插值運(yùn)算、成圖展示和制圖打印,分別取2019年6月18日和9月18日的土壤相對(duì)含水率數(shù)據(jù)為例,空間插值交叉驗(yàn)證結(jié)果(表2)顯示插值算法可以較好地進(jìn)行空間插值預(yù)測(cè)。
表2 空間插值模型交叉驗(yàn)證結(jié)果
土壤墑情估算模型采用深度學(xué)習(xí)集成策略將CNN與RNN相結(jié)合的網(wǎng)絡(luò)模型結(jié)構(gòu)[26],利用過去第-7次至第次的氣象和土壤墑情數(shù)據(jù)集合估算未來(lái)第+1次土壤墑情數(shù)據(jù)。模型結(jié)構(gòu)分別為基于門循環(huán)單元(Gate Recurrent Unit, GRU)的RNN和CNN,二者的輸出值拼接后輸入元學(xué)習(xí)器,最終得到估算結(jié)果,其中元學(xué)習(xí)器為全連接神經(jīng)網(wǎng)絡(luò)結(jié)構(gòu),其表達(dá)式如式(2)所示
式中′為輸入,′為輸出,為權(quán)重,為偏置,為神經(jīng)元數(shù)量。
選取2012-2018年山東省諸城市賈悅太古莊監(jiān)測(cè)站的46 944條土壤墑情數(shù)據(jù)對(duì)估算模型進(jìn)行實(shí)測(cè)驗(yàn)證。模型基于Keras框架搭建模型,后臺(tái)為TensorFlow 1.6,編程語(yǔ)言為Python3.6。模型的驗(yàn)證結(jié)果顯示,預(yù)報(bào)精度評(píng)價(jià)指標(biāo)均方誤差(Mean Square Error,MSE)、平均絕對(duì)誤差(Mean Absolute Error,MAE)、均方根誤差(Root Mean Square Error,RMSE)和決定系數(shù)(coefficient of determination,2)分別0.597 3、0.474 1、0.772 8和0.874 1。模型估算結(jié)果表明,所構(gòu)建模型能夠準(zhǔn)確的進(jìn)行土壤墑情估算(圖7)。
圖7 土壤墑情估算模型實(shí)測(cè)驗(yàn)證結(jié)果
土壤墑情直接決定作物的水分供需關(guān)系,在實(shí)際應(yīng)用中通過物聯(lián)網(wǎng)監(jiān)測(cè)設(shè)備所采集的原始數(shù)據(jù)進(jìn)行數(shù)據(jù)挖掘提供灌溉制度服務(wù)具有重要意義。系統(tǒng)灌溉決策基于水量平衡原理[27],計(jì)算如式(3)所示
式中ETc為作物實(shí)際需水量,mm;為灌溉量,mm;為降水量,mm;Δ為土體貯水量的變化,mm;為徑流量,mm;為土體下邊界凈通量,mm。
ETc的計(jì)算采用單作物系數(shù)法,其表達(dá)式如(4)所示
式中K為作物系數(shù),采用聯(lián)合國(guó)糧食及農(nóng)業(yè)組織(Food and Agriculture Organization of the United Nations,F(xiàn)AO)推薦值與用戶自定義[28];ET0為參考作物騰發(fā)量,選用FAO推薦的彭曼—蒙蒂斯(Penman-Monteith)模型[29]計(jì)算如式(5)所示
式中ET0為參考作物蒸散量,mm/d;Δ為溫度—飽和水汽壓關(guān)系曲線在溫度處的切線斜率,kPa/℃;R為凈輻射,MJ/(m2·d);為土壤熱通量,MJ/(m2·d);為平均溫度,℃;為干濕表常數(shù);2為2 m高處風(fēng)速,m/s;e為平均飽和水汽壓,kPa;e為實(shí)際水汽壓,kPa。
系統(tǒng)通過相應(yīng)的微服務(wù)模塊實(shí)現(xiàn)ET0計(jì)算與發(fā)布,以位于北京市昌平區(qū)小湯山的站點(diǎn)為例,該地塊于2019年10月5日播種冬小麥,通過系統(tǒng)可查詢相應(yīng)時(shí)段的ET0數(shù)據(jù)(圖8a),通過選取FAO推薦的作物系數(shù)與對(duì)應(yīng)種植作物的生育期階段計(jì)算作物的需水量,實(shí)現(xiàn)水量平衡分析并推薦參考灌溉水量(圖8b)。
圖8 灌溉決策服務(wù)功能界面
本研究設(shè)計(jì)和開發(fā)的基于云原生土壤墑情監(jiān)測(cè)系統(tǒng)已經(jīng)在中國(guó)21個(gè)省份得到應(yīng)用,已構(gòu)建自動(dòng)監(jiān)測(cè)站點(diǎn)970個(gè),累計(jì)采集土壤墑情與農(nóng)業(yè)氣象數(shù)據(jù)6 000余萬(wàn)條。近5年,年均用戶數(shù)增長(zhǎng)率14%,年均數(shù)據(jù)量增長(zhǎng)率95.2%。系統(tǒng)在促進(jìn)土壤墑情監(jiān)測(cè)技術(shù)、深度學(xué)習(xí)墑情估算模型構(gòu)建、多源數(shù)據(jù)協(xié)同空間分析及灌溉決策應(yīng)用方面具有一定的借鑒意義。
本研究基于上述系統(tǒng)設(shè)計(jì),以云原生技術(shù)為架構(gòu)基礎(chǔ),通過運(yùn)用深度學(xué)習(xí)、3D WebGIS等技術(shù)實(shí)現(xiàn)了土壤墑情多源大數(shù)據(jù)的數(shù)據(jù)感知、分析制圖與挖掘應(yīng)用,并取得以下結(jié)論:
1)提出了多維度土壤墑情數(shù)據(jù)感知獲取技術(shù)方案。綜合采用了設(shè)備上報(bào)、模型數(shù)據(jù)校正插補(bǔ)和多源異構(gòu)數(shù)據(jù)協(xié)同獲取3種方式,滿足對(duì)數(shù)據(jù)的采集頻率、屬性更新、連續(xù)完整和種類多樣的要求,構(gòu)建實(shí)時(shí)更新、智能模型和多源數(shù)據(jù)融合的數(shù)據(jù)獲取感知服務(wù)。
2)設(shè)計(jì)了以云原生技術(shù)為基礎(chǔ)的高可用云計(jì)算軟件平臺(tái)架構(gòu)。根據(jù)業(yè)務(wù)需求劃分微服務(wù)模塊以細(xì)化平臺(tái)服務(wù)粒度,通過容器技術(shù)打包微服務(wù)實(shí)例以消除環(huán)境制約,整合開發(fā)運(yùn)維工具鏈實(shí)現(xiàn)靈活、高效、一體化的敏捷開發(fā)與管理體系。
3)實(shí)現(xiàn)對(duì)土壤墑情數(shù)據(jù)可視化分析表達(dá)和深度挖掘應(yīng)用。運(yùn)用WebGL等技術(shù)實(shí)現(xiàn)前端三維空間可視化分析與制圖,提供直觀的決策支持依據(jù)。協(xié)同多源大數(shù)據(jù)分析與建模,實(shí)現(xiàn)土壤墑情估算功能。深入挖掘土壤墑情、氣象和作物數(shù)據(jù),提供基于水量平衡的灌溉決策服務(wù)。
[1] 操信春,任杰,吳夢(mèng)洋,等. 基于水足跡的中國(guó)農(nóng)業(yè)用水效果評(píng)價(jià)[J]. 農(nóng)業(yè)工程學(xué)報(bào),2018,34(5):1-8.
Cao Xinchun, Ren Jie, Wu Mengyang, et al. Assessing agricultural water use effect of China based on water footprint framework[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2018, 34(5): 1-8. (in Chinese with English abstract)
[2] 顧哲,袁壽其,齊志明,等. 基于ET和水量平衡的日光溫室實(shí)時(shí)精準(zhǔn)灌溉決策及控制系統(tǒng)[J]. 農(nóng)業(yè)工程學(xué)報(bào),2018,34(23):101-108.
Gu Zhe, Yuan Shouqi, Qi Zhiming, et al. Real-time precise irrigation scheduling and control system in solar greenhouse based on ET and water balance[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2018, 34(23): 101-108. (in Chinese with English abstract)
[3] 郁進(jìn)元,何巖,趙忠福,等. 農(nóng)田土壤水分各種測(cè)量方法的比較與分析[J]. 浙江水利科技,2007,15(6):1-2.
Yu Jinyuan, He Yan, Zhao Zhongfu, et al. Comparison and analysis of measuring methods for soil moisture of farmland[J]. Zhejiang Water Conservancy Technology, 2007, 15(6): 1-2. (in Chinese with English abstract)
[4] 孟德倫,孟繁佳,段曉菲,等. 基于頻域法的便攜式無(wú)線土壤水分測(cè)量裝置設(shè)計(jì)與試驗(yàn)[J]. 農(nóng)業(yè)工程學(xué)報(bào),2017,33(增刊1):114-119.
Meng Delun, Meng Fanjia, Duan Xiaofei, et al. Design and experiment of portable wireless soil moisture measuring device based on frequency-domain method[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2017, 33(Supp.1): 114-119. (in Chinese with English abstract)
[5] Schaefer G L, Cosh M H, Jackson T J. The USDA natural resources conservation service Soil Climate Analysis Network (SCAN)[J]. Journal of Atmospheric and Oceanic Technology, 2007, 24(12): 2073-2077.
[6] Brock F V, Crawford K C, Elliott R L, et al. The Oklahoma Mesonet: A technical overview[J]. Journal of Atmospheric and Oceanic Technology, 1995, 12(1): 5-19.
[7] Dorigo W A, Wagner W, Hohensinn R, et al. International soil moisture network: A data hosting facility for global in situ soil moisture measurements[J]. Hydrology and Earth System Sciences, 2011, 15(5): 1675-1698.
[8] 王國(guó)復(fù),李集明,鄧?yán)?,? 中國(guó)氣象科學(xué)數(shù)據(jù)共享服務(wù)網(wǎng)總體設(shè)計(jì)與建設(shè)[J]. 應(yīng)用氣象學(xué)報(bào),2004,15(S1):10-16.
Wang Guofu, Li Jiming, Deng Li, et al. Design and development of china meteorological data sharing service system[J]. Journal of Applied Meteorology, 2004, 15(S1): 10-16. (in Chinese with English abstract)
[9] Balalaie A, Heydarnoori A, Jamshidi P. Microservices architecture enables DevOps: Migration to a cloud-native architecture[J]. IEEE Software, 2016, 33(3): 42-52.
[10] 劉福鑫,李勁巍,王熠弘,等. 基于Kubernetes的云原生海量數(shù)據(jù)存儲(chǔ)系統(tǒng)設(shè)計(jì)與實(shí)現(xiàn)[J]. 計(jì)算機(jī)應(yīng)用,2020,40(2):547-552.
Liu Fuxin, Li Jinwei, Wang Yihong, et al. Design and implementation of cloud native massive data storage system based on Kubernetes[J]. Computer Application, 2020, 40(2): 547-552. (in Chinese with English abstract)
[11] 曹美. 土壤濕度傳感器影響因素及補(bǔ)償方法研究[D].天津:河北工業(yè)大學(xué),2015.
Cao Mei. Research on Influence Factors and Compensation Methods of Soil Moisture Sensor[D]. Tianjin: Hebei University of Technology, 2015. (in Chinese with English abstract)
[12] Mathiyalagan V, Grunwald S, Reddy K R, et al. A WebGIS and geodatabase for Florida's wetlands[J]. Computers and Electronics in Agriculture, 2005, 47(1): 69-75.
[13] Cai Yu, Zheng Wengang, Zhang Xin, et al. Research on soil moisture prediction model based on deep learning[J]. PloS One, 2019, 14(4): e0214508.
[14] 黃婕,張豐,杜震洪,等. 基于RNN-CNN集成深度學(xué)習(xí)模型的PM2.5小時(shí)濃度預(yù)測(cè)[J]. 浙江大學(xué)學(xué)報(bào):理學(xué)版,2019,46(3):370-379.
Huang Jie, Zhang Feng, Du Zhenhong, et al. Hourly concentration prediction of PM2.5 based on RNN-CNN ensemble deep learning model[J]. Journal of Zhejiang University: Science Edition, 2019, 46(3): 370-379. (in Chinese with English abstract)
[15] Taneja S, Gupta P R. Gupta. Python as a tool for web server application development[J]. JIMS8I-International Journal of Information Communication and Computing Technology, 2014, 2(1): 77-83.
[16] Balalaie A, Heydarnoori A, Jamshidi P. Microservices architecture enables DevOps: Migration to a cloud-native architecture[J]. IEEE Software, 2016, 33(3): 42-52.
[17] Fazio M, Celesti A, Ranjan R, et al. Open issues in scheduling microservices in the cloud[J]. IEEE Cloud Computing, 2016, 3(5): 81-88.
[18] Claus P. Containerization and the PaaS cloud[J]. IEEE Cloud Computing, 2015, 2(3): 24-31.
[19] Bernstein D. Containers and cloud: From LXC to docker to Kubernetes[J]. IEEE Cloud Computing, 2014, 1(3): 81-84.
[20] 翁湦元,單杏花,閻志遠(yuǎn),等. 基于Kubernetes的容器云平臺(tái)設(shè)計(jì)與實(shí)踐[J]. 鐵路計(jì)算機(jī)應(yīng)用,2019,28(12):49-53.
Weng Juyuan, Shan Xinghua, Yan Zhiyuan, et al. Design and practice of container cloud platform based on Kubernetes[J]. Railway Computer Application, 2019, 28(12): 49-53. (in Chinese with English abstract)
[21] 米向榮. 高可用MySQL同步集群的研究與設(shè)計(jì)[J]. 忻州師范學(xué)院學(xué)報(bào),2019,35(5):26-32.
Mi Xiangrong. Research and design of highly available MySQL synchronous cluster[J]. Journal of Xinzhou Normal University, 2019, 35(5): 26-32. (in Chinese with English abstract)
[22] 丁海斌,崔雋,陸凱. 基于Docker的DevOps系統(tǒng)設(shè)計(jì)與實(shí)現(xiàn)[J]. 指揮信息系統(tǒng)與技術(shù),2017,8(3):87-92.
Ding Haibin, Cui Jun, Lu Kai. Design and implementation of DevOps system based on docker[J]. Command Information System and Technology, 2017, 8(3): 87-92. (in Chinese with English abstract)
[23] Kim M S, Jang I S. Design and implementation of 3D geospatial open platform based on HTML5/WebGL technology[J]. Spatial Information Research, 2015, 23(6): 57-66.
[24] 徐馳,曾文治,黃介生,等. 基于高光譜與協(xié)同克里金的土壤耕作層含水率反演[J]. 農(nóng)業(yè)工程學(xué)報(bào),2014,30(13):94-103.
Xu Chi, Zeng Wenzhi, Huang Jiesheng, et al. Modelling soil water content in plow layer using co-kriging interpolation based on hyperspectral data[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2014, 30(13): 94-103. (in Chinese with English abstract)
[25] 劉姍姍,白美健,許迪,等. 畦田灌溉模擬中田面微地形空間分布插值方法改進(jìn)[J]. 農(nóng)業(yè)工程學(xué)報(bào),2015,31(17):108-114.
Liu Shanshan, Bai Meijian, Xu Di, et al. Improvement of interpolation methods for surface micro-topography spatial distribution in border irrigation simulation[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2015, 31(17): 108-114. (in Chinese with English abstract)
[26] Li Liang, Zhu Xinge, Hao Yiming, et al. A hierarchical CNN-RNN approach for visual Emotion classification[J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2019, 15(3): 1-17.
[27] 文冶強(qiáng),楊健,尚松浩. 基于雙作物系數(shù)法的干旱區(qū)覆膜農(nóng)田耗水及水量平衡分析[J]. 農(nóng)業(yè)工程學(xué)報(bào),2017,33(1):138-147.
Wen Yeqiang, Yang Jian, Shang Songhao. Analysis on evapotranspiration and water balance of cropland with plastic mulch in arid region using dual crop coefficient approach[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2017, 33(1): 138-147. (in Chinese with English abstract)
[28] Allen R G, Pereira L S, Raes D, et al. FAO irrigation and drainage paper No. 56[J]. Rome: Food and Agriculture Organization of the United Nations, 1998, 56(97): e156.
[29] Allen R G, Pereira L S, Raes D, et al. Crop evapotranspiration: Guidelines for computing crop water requirements-FAO irrigation and drainage paper 56[J]. FAO, 1998, 300(9): D05109.
Design and application of soil moisture content monitoring system based on cloud-native technology
Yu Jingxin1,4, Du Sen2※, Wu Yong2, Zhong Yonghong2, Zhangzhong Lili1, Zheng Wengang1, Li Wenlong1,3
(1.100097,; 2.100125,; 3.,100097,; 4.,,100083,)
To meet the demand of soil moisture content monitoring on a national scale, at the level of data acquisition, a soil moisture content data acquisition and perception technology system based on in-situ monitoring of automatic soil moisture content monitoring station and multi-source heterogeneous thematic data was constructed in this study, which realized the online monitoring of soil moisture content and multi-source data fusion. Further in terms of data quality control in the soil moisture data quality control strategy was proposed for data cleaning and established the soil moisture content data correction and interpolation model, in the cloud background received by the TCP/IP protocol of the Internet of things device came back after the packet data parsing and quality judgment. For abnormal or missing data, through the calibration data interpolation model to predict, avoided the interruption problem caused by the missing data, ensured data accuracy, integrity, and availability. Moreover, the soil moisture content data correction and interpolation model adopted the deep learning algorithm and the Stacking strategy to merge the Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) networks. The soil moisture content monitoring system facing the national scale had the characteristics of a large number of automatic station devices, massive user visits, and vast amounts of data computation, and had the characteristics of high frequency, high concurrency, and continuous growth. The ordinary web architecture could not ensure the stable and reliable operation of the system. Therefore, the system adopted the cloud-native technology system suitable for the cloud computing characteristics, used the micro-service architecture and the container technology to construct a flexible development model, and improved the efficiency of computing resource utilization. The system architecture design was based on the cloud-native technology, the module of the system was flexibly developed and deployed in the form of micro-services, the independent instance of packaging and running container technology was used to solve the problem of environmental configuration and resource utilization efficiency, and the container was dynamically scheduled to optimize the utilization of cloud computing resources. The core modules such as soil moisture content data reporting collection, soil moisture content data visualization analysis, and soil moisture content data mining application were arranged in the system. Based on GIS (Geographic Information System) spatial analysis and WebGL technology, the front-end 3D WebGIS data analysis function module was developed, and the collaborative Kriging interpolation method was used to realize the online analysis and visual mapping of collaborative soil moisture content, land use types, altitude, and other multi-source data. The system mined the data value deeply and utilized the deep learning algorithm to realize the soil moisture content prediction service which used the data of the past 8 days to predict the data of the next day. Based on the principle of water balance, the application service of irrigation decision was realized. By selecting the crop coefficient recommended by FAO and the growth stage of the corresponding planting crops, the water demand of crops was calculated, and the water balance analysis was realized and the reference irrigation water quantity was recommended. Since its application, the system had been deeply applied in more than 21 provinces, 970 automatic monitoring stations had been established, and more than 60 million automatic moisture monitoring stations had been collected. The system provided reliable data sources and technical support for decision-making departments, agricultural technicians, researchers, and other users to master the current situation of soil moisture content, guide agricultural water-saving irrigation, and obtain accurate and continuous soil moisture content scientific research data.
soil moisture content; monitoring; system design; data perception; WebGIS; deep learning
于景鑫,杜森,吳勇,等. 基于云原生技術(shù)的土壤墑情監(jiān)測(cè)系統(tǒng)設(shè)計(jì)與應(yīng)用[J]. 農(nóng)業(yè)工程學(xué)報(bào),2020,36(13):165-172.doi:10.11975/j.issn.1002-6819.2020.13.020 http://www.tcsae.org
Yu Jingxin, Du Sen, Wu Yong, et al. Design and application of soil moisture content monitoring system based on cloud-native technology[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2020, 36(13): 165-172. (in Chinese with English abstract) doi:10.11975/j.issn.1002-6819.2020.13.020 http://www.tcsae.org
2020-03-26
2020-05-24
國(guó)家重點(diǎn)研發(fā)計(jì)劃(2017YFD0301004);現(xiàn)代農(nóng)業(yè)產(chǎn)業(yè)技術(shù)體系建設(shè)項(xiàng)目-國(guó)家玉米產(chǎn)業(yè)技術(shù)體系(CARS-02-87);北京市農(nóng)林科學(xué)院院創(chuàng)新能力建設(shè)項(xiàng)目(KJCX20180706)
于景鑫,博士生,高級(jí)工程師,主要從事土壤墑情平臺(tái)開發(fā)與數(shù)據(jù)挖掘研究。Email:Jingx.Yu@outlook.com
杜森,研究員,主要從事土肥節(jié)水技術(shù)研究和推廣。Email:dusen@agri.gov.cn
10.11975/j.issn.1002-6819.2020.13.020
TP311.5
A
1002-6819(2020)-13-0165-08