Dynamic function allocation of agricultural robot vehicle controlled by man-machine cooperation*

Tingting Mao, Shuxian Dong, Jinlin Xue

(College of Engineering, Nanjing Agricultural University, Nanjing, 210031, China)

Abstract: It is necessary to distribute functions reasonably between a human operator and an automation system in a teleoperated agricultural robotic tractor to accomplish a task cooperatively. This paper proposes a strategy of dynamic function allocation on the basis of a BP neural network, genetic algorithm and adaptive genetic algorithm. Here, the operator’s state, workload, and task demand are chosen as trigger mechanism of dynamic function allocation. Then, a traditional BP neural network, genetic algorithm based BP neural network, and adaptive genetic algorithm based BP neural network are established by taking the operator’s state, workload, and task demand as inputs of the network and automation level as output. The three network are compared to obtain more effective dynamic function allocation. Simulation tests show that the adaptive genetic algorithm based BP neural network has minimum training time and has highest prediction accuracy.

Keywords: teleoperation; agricultural robot; function allocation; man-machine cooperation; genetic algorithm; adaptive genetic algorithm; BP neural network

0 Introduction

At present, teleoperation technology is mostly used in aerospace, deep sea exploration, and medicine industry and other related fields with robot operation, replacing human to complete tasks under dangerous and harsh environments[1-5]. The combination of robot system and network teleoperation technology, that is, the fusion of human intelligence and the intelligence of robot system refers to the situation when human intelligence is used to compensated the lack of autonomy of robot system, thus to improve practicability of the robot system in complex environments[8].

The combination of teleoperation technology and agricultural robot vehicle is the integration of human intelligence and machine intelligence through “human-in-the-loop” to bring about man-machine cooperation control, which means that operators need to understand and cooperate with automation system to complete the work task together. It can not only fully involve the operator but also greatly improve the work efficiency and intelligent level of the automation system[6-7]. The relationship between human and machine is cooperative in teleoperated agricultural robot vehicles. Therefore, the functions should be assigned between human and machine depending on the situation.

In 1951, Fitts put forward the concept of function allocation for the first time, which refers to the process of reasonably assigning functions/tasks in the system to human and machines. Function allocation is known as “one of the most important issues in the design process of man-machine intelligent system”[9]. The traditional method of function allocation is static allocation, that is, in the system design stage, the function/task is reasonably assigned to human or machine by comparing the advantages of abilities of human and machine, and does not change in the process of system operation. But in the whole process, the function state of the operator cannot be unchangeable. What’s more, if more tasks are assigned to the operator, the workload may exceed his/her capacity, resulting in the decrease of work efficiency, misoperation, and even accidents. On the other hand, if the machine always maintains high control authority and the operator is in the position of supervision for a long time, it will lead to the lack of human situation awareness and the absence of “human-in-the-loop”[10]. Therefore, it is necessary to allocate functions dynamically, that is, to allocate functions again according to the real-time environment, so that operator’s awareness of the situation can be maintained at a high level, and thus complex tasks can be completed efficiently[11]. In this way, we can not only give full play to the advantages of human judgment and decision-making but also ensure that the automation system has the ability of independent decision-making.

Scholars over the world have explored the function allocation methods of man-machine system in their respective research fields. Fits put forward a man-machine capability comparison method, which is widely used in the field of industrial automation[9]. Dearden et al. developed a scenario-based allocation method for naval ship system, which was later successfully applied to the functional allocation design of single-seat airplane[12]. Zhou’s team expounded the characteristics of man machine system, summarized the principles and methods of human machine function allocation in manned spaceflight system, and constructed a multi-objective fuzzy decision allocation model[13-14]. From the perspective of the overall effectiveness of the system, Zhang et al. explored the dynamic allocation of man-machine functions in UAV combat supervision and control system, and proposed the principles and methods of man-machine function allocation[15]. Based on the idea that a single operator controls multiple UAVs, Wang et al. designed a function allocation method according to the operator’s workload. The simulation results show that dynamic function allocation can improve the performance of the system[16]. Zhang et al. completed the allocation of fault detection function of civil aviation cockpit using uncertainty language multi-attribute decision-making, determined the automation level range using uncertainty extended weighted average operator, and combined it with uncertainty language mixed aggregation operator and finally determined the automation level[17]. Yang established a dynamic model that can predict the operator’s functional state and changed the current operator’s level of processing tasks according to the operator’s functional state and the level of processing tasks at the last moment[18]. In general, there are many researches and applications on function allocation in the fields of industrial automation, but there are few researches on the dynamic function allocation of man-machine system of remotely operated agricultural tractor.

1 Function allocation and levels of automation

1.1 Trigger mechanism of dynamic function allocation

According to the different control subjects, the trigger mechanism of dynamic function allocation can be divided into two types: human trigger and system trigger. Human trigger refers to the operator’s subjective decision on whether to switch and change the control authority according to his own workload and current task state. System trigger mainly has the following trigger mechanisms[19]: (1) emergency—the existing function allocation mode will be changed according to the emergency degree and number of events at any time; (2) operator’s working ability—the control authority is determined according to the change of human’s working ability at a certain time or period of time; (3) operator’s physiological state—the functions are allocated by monitoring the change of operator’s physiological index; (4) operator model—the function allocation is triggered by estimating and predicting the operator’s state. In the above trigger mechanisms, human trigger mechanism will increase the load of human. the trigger mechanism based on emergency needs to list all kinds of emergency situations and determine the degree of each situation in detail, which is too difficult for dynamic complex situations, and it is also difficult to build a reliable model with human awareness by the operator model based trigger mechanism. Therefore, in this paper, the state of the operator, the workload of the operator and the task demand are used as trigger mechanism to guide the function allocation.

On one hand, we should pay attention to the role of operators, make full use of human experience and knowledge, and reduce the complexity of automation level during dynamic function allocation. On the other hand, we should comprehensively consider the limitations of operators so that the workload and difficulty of allocation are within the scope of their ability. Function allocation is a typical multi-attribute decision-making problem, so experts can’t measure the relevant factors accurately and using only language value to evaluate them. Operator state are estimated according to the change of the physiological data of the operator. The experts evaluate the operator’s state parameters in three levels: poor, general, and good (represented by numbers 1, 2, and 3 respectively). Similarly, the task demand is evaluated and discussed by multiple experts according to actual experiences, corresponding to the three concepts of low, general and high (represented by numbers 1, 2, and 3 respectively). The operator’s workload refers to the number of tasks processed by the operator at a certain time (represented by numbers 1, 2, and 3 respectively).

1.2 Levels of automation and its authority

The determination of automation level is an important step in the allocation of human-machine functions. Shridan and Verplank divided the tasks/functions in the human-computer interaction system into 10 levels of automation (LOA)[20]. According to actual needs, this paper is divided into five level of automation, corresponding to different control permissions as shown in Table 1 and Table 2.

Tab. 1 Levels of Automation

Tab. 2 Control authority corresponding to levels of automation

Here, we obtained the corresponding relationship of levels of automation and three trigger factors (operator state, workload, and task demand) according to the expert knowledge as shown in Table 3.

Tab. 3 Corresponding relationship of LOA and trigger factors

2 Neural network models

Since BP neural network has good nonlinear mapping ability and generalization ability, it can be used to realize the human-machine function assignment of teleoperation agricultural robot system. However, there are some limitations such as slow convergence speed, poor network performance and ease to fall into local minimum. Therefore, this paper uses genetic algorithm and adaptive genetic algorithm to optimize the BP neural network.

2.1 BP neural network model

BP neural network is a kind of multi-layer feedback neural network adopting error back propagation learning algorithm[21]. It is one of the most widely used neural network models in artificial neural network (ANN). It has excellent nonlinear mapping ability, self-adaptive and self-learning characteristics. The topological structure of BP neural network model is divided into three parts: input layer, hidden layer and output layer. The external information is transmitted to the hidden layer through the input layer, and the learning rule uses the gradient descent method. Through back propagation, the weights and thresholds of each layer are adjusted gradually until the sum of squares of network errors is the minimum.

Suppose the output layer hasnneurons with the actual output value ofyand the expected output value ofy′, the total error function E, namely the optimal objective function, is as follows

(1)

And, the modified value of each weight is,

(2)

whereωi jis the weight from the input layer node to the hidden layer node,ηis the learning rate, andfjis the activation function of the hidden layer. Tansing type activation functionis used from input layer to hidden layer, and Purelin type activation functionis usedfrom hidden layer to output layer.

The main control factors of dynamic function allocation are operator state, operator workload, and task demand. The three factors above are set as input parameters of neural network model. According to relevant research, when the input node of BP neural network ismand the number of hidden layer nodes is set to 2m+1, the predicted value of BP network model is closer to the actual result[22]. Therefore, the number of hidden layer nodes is 7. The BP network model designed in this paper adopts a three-layer network, and the BP structure is 3-7-1, as shown in Figure 1.

Fig. 1 Topological graph of BP neural networks model

2.2 Genetic algorithm based BP neural network model

Genetic algorithm has the ability of global search, which is used to make up for the deficiency of BP neural network in randomly selecting connection weights and thresholds. The flow chart of genetic BP neural network is shown in Figure 2.

The specific steps are as follows:

1) First code and generate the initial population.

2) Set the fitness function value to determine the probability of individual selection.

Fig. 2 Flow chart of GABP neural networks

3) Set the operators to determine the probability of individual being selected.

4) Cross to obtain cross set ofNchromosomes, the new generation of individuals will carry the information of the previous generation.

5) Set the mutation probability to make some genes in the chromosome mutate to form a new population, thus to improve individual adaptability.

6) Calculate the fitness function value and judge whether the termination conditions are met. Otherwise, return to step 2).

2.3 Adaptive genetic algorithm based BP neural network model

Adaptive genetic algorithm has strong global optimization characteristics, and BP neural network is good at local search, therefore the combination of the two makes the network structure performance reach the optimal[23]. The improved algorithm flowchart is shown in Figure 3.

Fig. 3 Flow chart of BP neural network model based on adaptive genetic optimization

According to the flow chart, adaptive genetic algorithm BP neural network is mainly divided into three steps: initialization of network topology, search of optimal weight, and output of prediction results. Adaptive genetic algorithm is mainly used to adjust the crossover probability and mutation probability until the network weight is optimal to improve the prediction accuracy of BP neural network, enhance learning efficiency, and reduce training time.

The model parameters are listed in Table 4.

Tab. 4 Simulation parameters

3 Results and analysis

In this section, 50 sets of training data and 10 sets of test data are obtained by linear interpolation. The same data is used for LOA prediction of BP neural network based on genetic algorithm and BP neural network based on adaptive genetic algorithm, and the number of iterations and prediction accuracy are compared and analyzed[24-32].

3.1 Comparison and analysis of convergence rate

By training the same set of data, it can be seen from Figure 4 and Figure 5 that, under the same accuracy requirement the traditional genetic algorithm needs 50 iterations and the adaptive genetic algorithm needs 23 iterations. In terms of the number of iterations, the prediction model of BP neural network based on adaptive genetic algorithm is obviously better than that based on traditional genetic algorithm.

Fig. 4 Convergence curve of BP neural network based on genetic algorithm

3.2 Comparison and analysis of prediction model accuracy

The prediction results of BP neural network, genetic algorithm optimization and adaptive genetic optimization are compared with the expected values one by one.

Fig. 5 Convergence curve of BP neural network based on adaptive genetic algorithm

Fig. 6 Test results of BP neural network

Fig. 7 Test results of GABP neural network

Fig. 8 Test results of AGABP neural network

It can be seen from the above results that the prediction accuracy of BP neural network and genetic BP neural network are 40% and 45% respectively, which is quite different from the expected value. The BP neural network automatic grade prediction model based on the adaptive genetic algorithm not only has a faster convergence speed, but also has a better accuracy of 95%.

4 Conclusion

In order to solve the problem of dynamic allocation of functions in the remote operating agricultural tractor system, an adaptive genetic algorithm BP neural network is proposed in this paper. The dynamic function allocation takes into account the operator’s state, operator’s workload and task requirements. Genetic algorithm is used to optimize BP neural network. Although the prediction accuracy is only enhanced to some extent, the genetic algorithm easily falls into the local optimal solution. Therefore, this paper proposes a BP neural network automation grade model as a solution based on adaptive genetic algorithm. This model not only has a faster convergence speed but also has a higher prediction accuracy compared to BP neural network model and genetic algorithm optimization BP neural network model according to the results of the simulation.