Evaluation of a computer-assisted orthopedic training system for learning knee replacement surgery:a prospective randomized trial

Jie Xu, Deng Li, Bing Xu , Zhi-qing Cai, Ying-bin Zhang, Ruo-fan Ma,

1 Department of Joint Surgery, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, Guangdong Province, China

2 Department of Teaching and Learning, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, Guangdong Province, China

INTRODUCTION

Background

Halsted’s model of “see one, do one, teach one” is based on acquiring increasing amounts of responsibility that is culminated in near-independence. Halsted was not only interested in developing a system to train surgeons, but also in creating teachers and role models. A formal training program was the only way to ensure that surgical advancements would be passed on efficiently and effectively. Increased emphasis to obtain research funding is one cause of reduced time for mentoring.Furthermore, the emphasis on operating room efficiency to improve patient care and contain costs may also constrain the time that residents spend operating and therefore limits teaching opportunities. Therefore, the much sought after efficient model of “see one, do one, teach one” may be obsolete, but the general concept of observing, performing, and teaching can still be incorporated into a surgical residency program by adhering to certain principles of adult learning.

Since several years, the established teaching method “see one, do one, teach one” is being increasingly preceded by simulation-based training of surgical skills. Especially the advent of laparoscopy that introduced a whole new set of demanding technical requirements.

The use of surgical simulators as a training tool has increased rapidly over the past few years.1-5Seldom studies demonstrate that prior training of basic skills tying on simulators results in an improved resident performance in the operating room.Furthermore, consistent proof of benefit over conventional training is still lacking.6

Computer-assisted orthopedic training system (CAOS)

We use the CAOS, developed basing on Mimics, to be a simulator of surgery training. The computed tomography (CT) scan image is loaded into the system, and it produces 3-dimensional(3D) images. The CAOS allows real-time image-interactive navigation of the surgical tools and implant with respect to the two preacquired radiographic images processed from the CT scan. The computer also allows noncontact measurement of precise angles and depth of surgical tool and implant penetration of bone.

Use of CAOS in the training of total knee arthroplasty (TKA)

TKA is one of the most commonly operations in the joint surgery. Orthopedic trainees learn to do this procedure much later in their training career, and simulation with feedback for this procedure is hardly available.

CAOS uses CT scan-generated images to register the bone and landmarks. The system has an admin mode where the trainer can put the desired CT scans onto the system. The system identifies the bony cortices and generates two image(anteroposterior and lateral) views for the trainer to confirm the landmarks on the bone. The trainer in the admin mode of the system analyzes these images and decides the desired alignment, bone cuts and position of the implant. The CAOS guides the trainee to correct orientation and position in the training mode. The assessment mode assesses the performance of the trainee and gives the scores based on the scoring system.

Objective

In this study we investigated whether the skill acquisition on CAOS results in better operative performance. The effect of training was compared with the conventional training model on mannequin.

METHODS/DESIGN

Study participants

In total, 30 surgical residents in Department of Joint Surgery,Sun Yat-sen Memorial Hospital, Sun Yat-sen University, China were recruited based on the principle of voluntariness.

Inclusion criteria

? Junior orthopedic trainees from the local hospital

? Postgraduate years 1 to 5

? Informed consent was obtained from all the participants

Exclusion criteria

? Prior experience in TKA

? Previous or simultaneous clinical or laboratory-based practice in surgical techniques of TKA

This study population was chosen to avoid any interference with previous or simultaneous clinical or laboratory-based practice in surgical techniques of TKA. Before entering the study, all the subjects attended an intensive course in basic theory of TKA, including construction of the mechanical axis, alignment as well as ligament balance (Figure 1). This protocol was approved by the ethical committee of Sun Yatsen Memorial Hospital, Sun Yat-sen University, China. The study was reported in line with the Consolidated Standards of Reporting Trials (CONSORT) 2010 guidelines.7

Randomization and blinding

Through randomization by SPSS 15.0 (SPSS, Chicago, IL,USA), three groups of ten surgical residents were created.This is a sing-blinded investigation. The investigators responsible for outcome evaluation were kept blind to the grouping.

Interventions

The control group did not receive any additional training after the basic theoretical training program. The second group attended the conventional training on bone model, Halsted’s model of “see one, do one, teach one” was performed (conventional model training group). The third group attended a TKA training on the CAOS (CAOS training group).

Conventional model training program

The subjects randomized to model training were required to train approximately 1 hour daily for 10 days. Every training session, the TKA was performed on the bone model, the student focused on the reconstruction of the alignment, followed by the bone-sawing part, and sizing of the implants were performed. Halsted’s model of “see one, do one, teach one”was performed, all training sessions were supervised with continuous expert feedback.

Strengths and limitations

CAOS training program

Those subjects randomized to train on the CAOS attended a didactic hands-on session of the four procedural exercises,including a presentation of the parameters measured for assessment. The four tasks focused on reconstruction of alignment,bone cuts, selection of the size of implant, and position of the implant. Afterwards, they performed the tasks independently with available guidance concerning software or technical problems but without expert feedback concerning the procedure.Standardized feedback by the system through the assessment parameters was available for each trial. Training consisted of distributed, daily training modules of 30 minutes.

Outcome measures

At baseline, the skills were evaluated during a TKA on a bone model. Posttesting and retention testing during a TKA on a 3D-print bone model took place 1 week (= post) and 4 months(= retention) afterfinishing the training program. Before the procedure, they received appropriate cognitive information using a video instruction and detailed text. Their procedural knowledge and operative performance were evaluated not only on time needed to perform the procedure, but also the key points, which include the bone defects, the planned surgical cuts, the choice of the implant size, position and rotation with respect to the bone cortical surfaces, the distances between bones and components. Any of those parameters affects directly the implant position during the planning phase, and all those parameters are linked together.

The ability for CAOS to help in skill acquisition has never been used as a formal TKA training tool. The assessment of such skills demands a scoring system, which can be reproducible as well as validated. There is no scoring system that can accurately assess the ability to evaluating skills during joint replacement surgery. We devised our own scoring system based on task analysis, which included precision difference of alignment-recontruction, perform accurate tibial and femoral cuts,and perform efficient sizing of the implants tofinish the task.

The procedure was broadly divided into several tasks and the candidate was scored on these subtasks. The candidate was assessed using various parameters as follows:

Figure 1: Theflow chart of the study.

? Control the alignment of the implants with respect to the mechanical axis, this means to be able to control and plan any tibial and femoral cut in 3D (level and orientation).

? Perform efficient sizing of the implants.

? Perform accurate cuts according to instruments.

? Include quality control at each step.

All these parameters were given weightage depending upon the importance of the step. The candidate was scored according to these parameters with a maximum score of 100.Weightage of marks for each step (20% marks for each task):

? Alignment controlled with accuracy (–5 marks for 2° deviation from the mechanical axis) (20% marks).

? Sizing of the implants with accuracy for both femoral and tibial side (20% marks). Five marks are deducted for every oversize/undersize of implant.

? Bone-sawing according to instruments with accuracy (20%marks). Five marks are deducted for every 2 mm oversize/undersize of each cut.

? Number of procedure of tibial and femoral cuts (2 allowed)then 5 mark is deducted for every extra procedure (20%marks).

? Implant inserted with accuracy (20% marks); 5 marks for 2 mm deviation from center of the joint.

After completion of the study, residents completed a 5-point Likert scale8on how useful they estimated the training (1 =not useful at all, 5 = very useful). Personal comments were allowed.

Statistical analysis

The collected data were analyzed by the SPSS 15.0. Data are shown as the mean ± SD or median (interquartile range [IQR]).The Kruskall-Wallis and the Mann-Whitney U tests are used to compare groups. Interrater reliability was calculated using the Spearman correlation. A P = 0.05 was considered significant.

RESULTS

All the 30 surgical residents completed the whole study protocol and attended baseline, posttesting, and retention testing.The time needed to perform at the beginning of the study did not differ between the groups (Table 1). The scoring of the procedures is shown in Table 2. At baseline, there was no difference between the groups.

The quality of performance is shown in Table 3. There was significant difference between the control and the other groups. The CAOS training group performed significantly better than the model training group at post testing (P ＜ 0.05)and significantly better than the control group (P ＜ 0.05). The same trend, i.e., CAOS training group performing the procedure better than the model training group, was identified at retention testing but this did not reach statistical significance(Kruskall-Wallis test, P = 0.937).

Further analysis of the quality of performance of each task at post testing. For alignment controlled, there was an improvement of the scores for CAOS training group. In addition to the differences in axis deviation, a comparison of sizing-of-theimplants between these groups also demonstrated significant results. For accuracy of bone-sawing and implant inserted, results yielded significantly lower occurrence of outliers among the CAOS training group in our study (P ＜ 0.05; Figure 2).

The results of the 5-point Likert scale are shown in Figure 3.For the entire training, course residents in the CAOS training group assessed its usefulness with a median score of 5 (IQR= 0) and the residents in the model training group 3 (IQR= 1, P ＜ 0.05). For the alignment-controled exercise, this was 4 (IQR = 1.25) and 2 (IQR = 0.5) correspondingly (P ＜ 0.05).For sizing of the implants, this was 5 (IQR = 1) and 2 (IQR= 1) correspondingly (P ＜ 0.05). For bone-sawing according to instruments, this was 4 (IQR = 1) and 4 (IQR = 1) correspondingly (P = 0.403).

DISCUSSION

Until now, sometimes, in somefields, cadaver organ models have been used to teach these skills such as osteotomy, cutting,and coagulation.9-11However, significantlyfinancial and time resources are required for such endeavors not to forget the ethical concerns that come along with this kind of training.12This system helps residents practice complex procedure on the 3D virtual model without the costly setup required for thelaboratory-based simulations. This is of utmost importance where more and more junior residents are spending less time in the training and have less exposure to hands-on training by senior surgeons. The CAOS system provides results and scores with which the surgeon and trainee can readily identify and improve on subtasks within the particular surgery.

Table 1: Basic skill level at beginning of the study for control,model training, and CAOS training groups

Table 3: Quality of Performance for control, model training, and CAOS training groups

Table 2: Scoring base on task analysis at beginning of the study for control, model training, and CAOS training groups

Besides offinancial and ethical concerns, the aim of this study was to evaluate the transfer of procedural skills acquired during CAOS training to a real procedure. For the CAOS training group, significantly higher quality of performance was seen at posttesting compared with both the other groups.At that time, they also performed the procedure significantly faster than the model training group and the control group.

At retention testing, the difference that was measured initially seemed to be diminished but no statistical differences were seen between the groups. This indicates deterioration of performance, which stresses the importance of maintenance training. Another study which was focused on laparoscopic salpingectomy skill specifically addressed retention of procedural skills.13They found no deterioration of skill in novices after 6 months. In that study, surgical trainees were included who still had ongoing experience in the operating theater.

It is easy to see that CAOS can have many different applications in surgical education. CAOS may be used to instruct surgeons in the fundamentals of a procedure, assess their ability on objectively measured parameters, and allow skills required for an operation to be taught in a similar environment to that in which the skills will be used but without the pressures and distractions of an operating theatre. Predetermined criteria can also be used to assess a surgeon’s progress in mastering a particular skill, thus allowing the learner to advance at their own individual speed. Surgeons can also use the simulators to rehearse and master complex and challenging operations, and overcome steep learning curves before coming into contact with a patient, thus ensuring patient safety and optimizing the overall clinical experience of the surgeon at the same time. It will also allow practicing surgeons to learn robotic techniques that may not be easily taught in an operating theatre in the future.

Eventually, it is possible that the presence of training benefit for the virtual training group in this study indicates that virtual simulation is suitable and has the required features to teach the refined skills needed in procedural exercises. The students in our study indeed appreciated the cognitive imput of the virtual trainer to get to know the different steps of the procedure and the handling of the instruments but thought the realism of the haptic feedback was lack. Vapenstad et al.14similarly showed that surgeons assess haptic feedback as an important but currently insufficient feature. In China, with the continuous updating of medical education methods, we also plagued by the authenticity of the simulation. Fortunately, domestic research institutions are committed to improving this aspect.2,15

This study has some limitations. Firstly, the study was performed in junior residents without clinical or surgical experience in TKA. This group was chosen to prevent interference of previous or concurrent surgical practice. Compared with the senior residents, they might have had insufficient knowledge to fully profit from the training and training effect might have been underestimated. Secondly, as the sample size calculation was based on a study including general surgery residents, our current study could have been underpowered. We do think this might have concealed existing benefits for the model training group. However, the CAOS training group has shown a significant trend of improved performance compared with the other groups. We, therefore, do not believe increasing the number of participants would have changed this outcome.Furthermore, it is hard to verify equal amount of training in the model and CAOS training group was restricted by time.We tried to control for this by including a minimum amount of training (five hours) in both training groups. Apparently, in the coming work, modification of these rating scales is needed.

Figure 2: Performance quality of surgical residents after CAOS training or model training.

In summary, while there were significant differences in performance between the computer-assisted and conventional training groups in the anatomical identification task, the more positive perception of the simulator as an effective learning tool is encouraging. Surgical simulators are in the early stages of development and evolving rapidly. The development of such tools will set in motion the path to developing a comprehensive surgical curriculum to better train and prepare arthroplasty surgeons of the future.

Author contributions

JX was responsible for experiment design, DL, BX, ZQC were responsible for implementation and evaluation of the experiment, data collection and relevant analysis. JX and RFM were responsible for performing training. All the authors were responsible for reviewing.

Conflicts of interest

The authors declare that they have no competing interests.

Financial support

This study was supported by the Science and Technology Planning Projects of Guangdong Province, China (Grant No. 2014A020215009,2013B051000024, 2014A020212060).

Research ethics

This protocol was approved by the Ethical Committee of Sun Yat-sen Memorial Hospital, Sun Yat-sen University, China.

Declaration of patient consent

The authors certify that they have obtained all appropriate participant consent forms. In the form, the participants have given their consent for their images and other clinical information to be reported in the journal. The participants understand that their names and initials will not be published and due efforts will be made to conceal their identity,but anonymity cannot be guaranteed.

Data sharing statement

Datasets analyzed during the current study are available from the corresponding author on reasonable request.

Plagiarism check

Checked twice by iThenticate.

Peer review

Externally peer reviewed.

Open access statement

This is an open access article distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License,which allows others to remix, tweak, and build upon the work noncommercially, as long as the author is credited and the new creations are licensed under identical terms.

Figure 3: 5-point Likert scale concerning perceived usefulness of surgical residents after CAOS training or model training.

REFERENCES

1. Hariri S, Rawn C, Srivastava S, Youngblood P, Ladd A. Evaluation of a surgical simulator for learning clinical anatomy. Med Educ.2004;38:896-902.

2. Lin Y, Wang X, Wu F, Chen X, Wang C, Shen G. Development and validation of a surgical training simulator with haptic feedback for learning bone-sawing skill. J Biomed Inform. 2014;48:122-129.

3. Lopez G, Wright R, Martin D, Jung J, Bracey D, Gupta R. A costeffective junior resident training and assessment simulator for orthopaedic surgical skills via fundamentals of orthopaedic surgery:AAOS exhibit selection. J Bone Joint Surg Am. 2015;97:659-666.

4. Rambani R, Viant W, Ward J, Mohsen A. Computer-assisted orthopedic training system for fracturefixation. J Surg Educ.2013;70:304-308.

5. Rambani R, Ward J, Viant W. Desktop-based computer-assisted orthopedic training system for spinal surgery. J Surg Educ.2014;71:805-809.

6. Sonnadara R, McQueen S, Mironova P, et al. Reflections on current methods for evaluating skills during joint replacement surgery: a scoping review. Bone Joint J. 2013;95-b:1445-1449.

7. Schulz KF, Altman DG, Moher D, Group C. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Trials. 2010;11:32.

8. Azimzadeh R, Valizadeh L, Zamanzadeh V, Rahmani A. What are important for patient centered care? A quantitative study based on perception of patients' with cancer. J Caring Sci. 2013;2:321-327.

9. van Velthoven RF, Hoffmann P. Methods for laparoscopic training using animal models. Curr Urol Rep. 2006;7:114-119.

10. Scheeres DE, Mellinger JD, Brasser BA, Davis AT. Animate advanced laparoscopic courses improve resident operative performance. Am J Surg. 2004;188:157-160.

11. Olinger A, Pistorius G, Lindemann W, Vollmar B, Hildebrandt U, Menger MD. Effectiveness of a hands-on training course for laparoscopic spine surgery in a porcine model. Surg Endosc.1999;13:118-122.

12. Rusche B. The 3Rs and animal welfare - conflict or the way forward? Altex. 2003;20:63-76.

13. Maagaard M, Sorensen JL, Oestergaard J, et al. Retention of laparoscopic procedural skills acquired on a virtual-reality surgical trainer. Surg Endosc. 2011;25:722-727.

14. Vapenstad C, Hofstad EF, Lango T, Marvik R, Chmarra MK. Perceiving haptic feedback in virtual reality simulators. Surg Endosc.2013;27:2391-2397.

15. Yanping L, Dedong Y, Xiaojun C, Xudong W, Guofang S, Chengtao W. Simulation and evaluation of a bone sawing procedure for orthognathic surgery based on an experimental force model. J Biomech Eng. 2014;136:034501.